Using Spikes in SCRUM

I am writing about spike in this article because I feel it gets abused during sprints and SCRUM. I will try to address why we need a spike, when to use it and when not to use it. First, what is a spike? A spike is really a prototype, code to try something new – prove a concept. Sometimes spikes are used to de-risk technologies. As developers we may be asked to do work we have never done before and prototyping that work is usually the first approach we think of. But let me start by saying why spikes are not friends of SCRUM, the reason is that on their own they offer no business value to the product owner. They do however offer a lot of value to the developer – but at the end, user stories are about business value and not knowledge to the developer. So maybe the approach is to try to get both and have a win-win situation. Try not to do a spike! Considering we are dealing with something new, something we have not done before, before jumping into prototyping the work see if you can get some value by cutting the complexity into smaller stories. For example, say we need to communicate “Sales” to another service, however we never communicated to this service before – we don’t know exactly the security or protocol to use. So no matter what you need to try to communicate to this service and figure it out. Why not just do it as part of the real story and estimate the complexity of the communication as part of the story itself. As trade off the story should have a very simple UI to do this and focus more on the communication. A later story can focus more on the UI aspect and make it more user friendly, but in order to just focus on the communication part, keep the UI functional but minimal – still if everything works there is real value here. But – what if it doesn’t work… Well, if it doesn’t work you still gained the knowledge, the same knowledge you would have gained when doing a prototype. Except this time you can estimate better, know more and adjust. It is not the end of the world not finishing a story – it is the end of the world not finishing a story and learning nothing from the experience. So I decided to come up with tips for Spikes and how to handle them:

• Every time you think you need a spike see if you can do it without a spike by breaking stories down offer some value and still gain knowledge If your team wants to do spikes every sprint there is a problem. It is not normal. It means they are afraid to commit and they are afraid to fail.
• Get around the spike by using Mocks – although this is a little bit cheating you can still mock parts of the software that are either “not ready”, or “we don’t know yet” and still get value and feedback. For previous example of sending sales to a service, we can mock the receiving service by accepting sales without worrying about protocol, security and so on. We interface this part of the code out and the story is more about “exporting” sales. Who is consuming the exported data can be done in separate story when we know more.
• If you are going to do a prototype then:
  • The prototype should time boxed (normally should be only 1 sprint)
  • Try not to have more than one spike per sprint, focus on sprint giving business value over technical knowledge.
  • The prototype should have a goal (what are we trying to learn)
  • The prototype must provide knowledge – even if you fail that’s knowledge that can be used to estimate better.
  • Doing a prototype should be team decision and the team should be open to explore alternatives.
  • The product owner should gain visibility into the prototype, so if the prototype yields that something is complicated, the product owner can offer alternatives to simplify the work.
  • A spike can also be used as an experiment – trying out a feature but not sure we want to keep it. The feature might be done in a branch and only used for feedback purposes.
  • Spikes should be estimated and entered in the backlog as technical stories
  • If a story is estimated high because – “we just don’t know how to do this…” – don’t bother putting a large estimation; instead leave the story as not estimated. Do the spike and then go back and re-estimate (or adjust the large estimation after you learned the results of the spike).
  • When possible defer commitment, if the complexity can wait, let it wait a bit. The longer you wait the more knowledge you gain. That complexity might turn simpler over time because you have learned things along the way and got feedback.
  • Time your spike – sometimes during the holidays it might be quite, the whole team is not present, take the opportunity to do a spike if you needed one.
  • Keep the code; even if it is a prototype code it correctly. However the definition of done may not apply fully to a spike – so in that case mention in in the acceptance what is the definition of done for this spike.

I have seen the two extremes, a team that never does spikes and have crazy estimations, and a team that makes 2 stories out of each story, one story for the spike and one story for the real story. Overall, try to remember that we all about offering value, where spikes does not. However, spike offer knowledge to help us estimate and de-risk, this knowledge eventually turns into value. So you should use spikes in SCRUM, but don’t overdo it.

How do you do architecture with SCRUM?

This is a hard one let me tell you. Before becoming a full time scrum master, I was a software architect full time. So when we had to work with user stories and translate them into code, architecture seems to have taken a back sit. However this is not true, architecture does not take a back sit in SCRUM in fact it is always there non-stop. Its not that architecture is not important - its need to fulfill the needs of the business and on its own its useless. So the business comes first, defines what we want and architecture is built all the time to meet the business needs - its on going and it never stops. I call it “just in time architecture”.  This is how it works:

The business drives architecture

everything needs a reason, you don’t just code it because you think its important. These can be looked as constraints of your system. For example you might have a constraint that you need to handle 1000 concurrent connections to your website, now this might affect how we build the site. There is a reason. We are not going to create a design that supports concurrency for fun. We are doing it because it is defined in a user story or constraint of a user story. Product owners don’t always think of constraints, so its your job to ask the questions. Do we need security? do we need concurrency? What kind of error handling do we want? A product owner that doesn’t really understand that will look at you funny and not understand. So make him or her understand. Give examples and give possible solutions. Provide choice so the product owner can make a decision.

But it still take a long time to code, so how do you present it in the backlog?

True. Even if the business gives you a good reason to do a framework that handles the needs, it takes time. It can even take more than a sprint. The solution I like to do is build it behind the scenes and build it by supporting real user stories. This way I get to validate my framework against real features. For this to work the stories need to be very very thin across the layers. For example, “add a user to the system”, might take a larger cost due to the infrastructure work behind it. Still, I only build the framework to add a user, not to delete a user and not to get a list of users. I keep my functionality limited to what I need and not beyond that. I expand my framework only based on new features that are required. This way I don’t build code that might never get used.

OK, but my kind of framework takes a long time to build  just to meet one story.

I would challenge that. You should be able to make stories small enough so some framework elements can be coded, and then add more framework elements when doing another story. If you still can’t around this idea - then lets have a use story that is a technical user story covering a technical feature. However technical user stories are bad and I only used them as a last resort. The reason they are bad is because product owners normally don’t understand them and therefore can’t properly prioritize them. So before you go the easy way out and get your product owner all confused with technical terms, see if you build your framework on the backbone of business need (or a user story).

Architecture never ends

You build your framework and design every sprint, every story. You make it better and better all the time. Its not just the first 3 months of development - its continuous and iterative. Within the development of every user story you might have tasks to improve the framework, and add features to it. Avoid at all cost big design up front, you will probably get it wrong because you don’t know everything yet. You will most likely not use half the stuff you build when doing big design up front. Design as you go, improve as you go. Think of building framework elements as tasks items to every story but not stories on their own.

Use patterns

Lots of pattern improve code and architecture - use them. Here are some of them:

IOC, AOP, DI, TDD patterns, BDD patterns and DDD patterns, (if you don’t know what this means start googling).

Use framework that are already done and tested

don’t reinvent the wheel and focus on the business needs and not the technologies. Unless you are a company that builds development frameworks, you should use what’s there and not re-build it.

How to make the daily SCRUM meeting better

What did you do yesterday?

What will you do today?

Are there any impediments in your way?

We all know these 3 famous questions, and with these 3 question we are going to know where we are in the current sprint. We are going to know what everyone is doing and we are going to know if we have impediments. Well, in reality it doesn't really end up like this.

In reality Bob will tell you that yesterday he was sick and had to leave early. James was planning to work on our sprint tomorrow, but he had to take his kids to the doctor and Ted worked on our project by got distracted on another bug from another team (where was the scrum master to block that?)

This is normal, we ask these somewhat personal question so we are going to get personal answers. Do I know more about the user stories that are going to be done in the sprint? not so much, but I know more about my team schedule and personal problems. So how do we fix this? Simple! instead of focusing on the people in the team, focus on the stories to be done. Go over each story on the board and ask “what’s the status with this story? who worked on it and what did they do? what are they planning to do tomorrow? Then the person or persons that worked on the story speak. If you speak, its because you did something on the story. If a team member doesn’t   speak after going over all the stories it means he or she did not work on any stories, and you need to try to find out why. This way at least the product owner gets to hear about what he cares about, which is the stories and their progress during the sprint.

Still, after going over all  the stories, as a SCRUM master, I still ask if anyone has anything to add, and if there are any impediments I should know about. I also ask if the product owner has new information from the customer he might want to share with the team. Still, it's important to keep the meeting to 15 minutes or so.

There are other benefits to the daily stand up - everyone is together at the same room, this should be also be a fun time. Everyone should feel at ease and happy to be together. Lets crack some jokes, make it light and fun, so the daily stand up doesn't end up like a status meeting where people are worried their heads will be chopped off. If people are scared to speak or tell you the “bad” stuff, then the meeting is pointless. Spending 15 together everyday is “quality time” with your team - make it count.

SCRUM - are user stories ever done?

I have been doing SCRUM for over a year now and overall applied the framework of SCRUM with good success. Although a simple framework, it is hard to apply. I decided to write this article because I feel the terminology of SCRUM is misleading a little and maybe I can give my option on the definition of “done”. One of the core elements of SCRUM are “user stories” and writing them and defining them is not simple, but at the end we do it and we have nice stories everyone understands. During our grooming sessions and planning session we feel that we have a good set of stories ready to go into our next sprint. Using poker planning everyone votes on the complexity of the stories with story points and everyone in the team comes to an agreement on the final votes, and off we go and the sprint starts. For the sake of this article let’s write about one story which allows you to add a user to a web form application. The user story was simple enough, “As an administrator, I want to be able to add users to my system, so I can manage access to the application”. There was more to it, some acceptance tests that only valid users should be added, using some validation rules. Overall not much more details than that. A user in this example has only 3 fields, name, last name and e-mail. The development started and off we go into implementing this story. During our demo part of the sprint, issues came up. The product owner said, “how come the save button is enabled if there is invalid information on the screen?” the developer said “You never said anything about the save button and validation in your story…” Then another torpedo come right in “I made some changes to the screen, forgot to press save and got into another screen… how come it didn't worn me that I will lose my changes”. Again the developer said, “I didn't see anything about navigation in the story”. All of a sudden, another team member said: “The error messages are not too clear; maybe we should change the fronts”. We go over the definition of done, and see that the meet everything on the list. The story is unit tested, we have reviewed the code and we implemented support all the way from the UI to the DB. Yet, the product owner is not happy with it, and after some discussion the team decided the story is not done. We need to fix it up and move it to the next sprint.

Done = Good for now

In SCRUM we use the word “done”, and sometimes “done-done” and this gives us the idea of that story should be complete and that we should never go back to it. But this is false. We see this every day at work. People, developers and product owners alike always tend to change their mind and add feedback, not during planning but mostly during a demo. This is really just because of how we think and how our brains work. We love to “touch” it and “feel it” and then we have an opinion, “This is cool!”, or “I don’t like the way this works”. Sometimes you don’t always think about some requirements when you write a story, review a story, or when you plan or task a story, you only start thinking about them when you actually “see” or “use” the story in the software. Even if everyone loves the way we did it, our customers really have the final say, and may ask you to change the way a feature works, and again that feature or story is not done. So the word done is misleading, and when we ask ourselves “is this done?” we tend to think of “perfect complete user story or feature”. The truth is that we will never get perfection, there is always a room to improve and therefore we will never be done. The question we should ask ourselves is if the story is “good”, or “good for now”, or “are we are happy with this”.

The definition of done

But, wait a minute – we got the “definition of done”, so if we meet everything on that check-list we must be done. In SCRUM terms this is true and the definition of done is a good thing. It allows us to have quality, by forcing us to look at other things than just the development. For example, we can have documentation, code review, unit tests and deployment elements in there. Still, you may meet everything in that check-list and still find out that the story may not be complete due to feedback. So what do you do now? You say the story is not done, because we didn’t think of this and that? Or do we create new stories to fill in the gap? Well, how about avoid the problem all together with a simple solution. Demo the story as soon as possible. Don’t wait for the end of the sprint to demo the whole thing, demo and ask feedback as soon as you have something to show. Sometimes it can be a screen that doesn’t even do anything, and you ask the product owner, “is this what you looking for? It can be done on the development machine and it should be done without any formal process. Let’s say our product owner is James – you should be able say: “Hey James, when you have a second can I show this story about adding users, tell me if you like it”. Normally feedback will flow immediately – product owner may notice that fonts are not clear and ask to change them. This is a simple task the programmer can do on the spot. To make sure, this demo process is done – why not add it to the “definition of done”. This way we know this must happen before moving the story to “done”. A demo can also reveal new information and features no one thinks about when writing the story. In this case, the team should get together and decide what to do. Can we continue the story in current scope and offer some value, if yes that should be the best option. After all, stories are all about offering value, and the only time a story should not be set to done is if quality was compromised by not meeting the full definition of done, and if it doesn't offer value any more due to new information.

“It’s not in the user story syndrome”

People like to know of everything up front, expecting the product owner to hand them over a story with all the details from the screen, color and specs of every detail. When the feature doesn’t work as expected, developers tends to say: “This wasn’t in the user story”. Let’s face it this is not going to happen, product owners can’t think of every detail and if they do its probably waste. I gave this syndrome a name: “It’s not in the user story” syndrome. After being a scrum master for some time, I just kept hearing this over and over again. The solution is actually to seek this information often and soon. Demo the story, so you lean of new elements of the story. If possible implement them in the current sprint, if not possible discuss these elements with the team. Always, keep talking and talk often. Normally, things will work themselves out simply by common sense and communication. A good product owner will learn to settle when he feels this is “good for now”, but not “done”. A good product owner will look for value and not perfection. We should not ask if the story is done, but rather if we can improve the feature with other features.

Can we ship the product?

So SCRUM chose to use the word “done”, and that’s OK, but don’t think of done as a done thing that will never change because it will. Sometimes, when it comes to a problematic user story which actually met the definition of done check list, I like to ask product owners, “if you had to ship the product tomorrow, is this story good for now?” if they say yes, I believe this story should be set to done and any other improvements should come as new stories.