Using Spikes in SCRUM

I am writing about spike in this article because I feel it gets abused during sprints and SCRUM. I will try to address why we need a spike, when to use it and when not to use it. First, what is a spike? A spike is really a prototype, code to try something new – prove a concept. Sometimes spikes are used to de-risk technologies. As developers we may be asked to do work we have never done before and prototyping that work is usually the first approach we think of. But let me start by saying why spikes are not friends of SCRUM, the reason is that on their own they offer no business value to the product owner. They do however offer a lot of value to the developer – but at the end, user stories are about business value and not knowledge to the developer. So maybe the approach is to try to get both and have a win-win situation. Try not to do a spike! Considering we are dealing with something new, something we have not done before, before jumping into prototyping the work see if you can get some value by cutting the complexity into smaller stories. For example, say we need to communicate “Sales” to another service, however we never communicated to this service before – we don’t know exactly the security or protocol to use. So no matter what you need to try to communicate to this service and figure it out. Why not just do it as part of the real story and estimate the complexity of the communication as part of the story itself. As trade off the story should have a very simple UI to do this and focus more on the communication. A later story can focus more on the UI aspect and make it more user friendly, but in order to just focus on the communication part, keep the UI functional but minimal – still if everything works there is real value here. But – what if it doesn’t work… Well, if it doesn’t work you still gained the knowledge, the same knowledge you would have gained when doing a prototype. Except this time you can estimate better, know more and adjust. It is not the end of the world not finishing a story – it is the end of the world not finishing a story and learning nothing from the experience. So I decided to come up with tips for Spikes and how to handle them:

• Every time you think you need a spike see if you can do it without a spike by breaking stories down offer some value and still gain knowledge If your team wants to do spikes every sprint there is a problem. It is not normal. It means they are afraid to commit and they are afraid to fail.
• Get around the spike by using Mocks – although this is a little bit cheating you can still mock parts of the software that are either “not ready”, or “we don’t know yet” and still get value and feedback. For previous example of sending sales to a service, we can mock the receiving service by accepting sales without worrying about protocol, security and so on. We interface this part of the code out and the story is more about “exporting” sales. Who is consuming the exported data can be done in separate story when we know more.
• If you are going to do a prototype then:
  • The prototype should time boxed (normally should be only 1 sprint)
  • Try not to have more than one spike per sprint, focus on sprint giving business value over technical knowledge.
  • The prototype should have a goal (what are we trying to learn)
  • The prototype must provide knowledge – even if you fail that’s knowledge that can be used to estimate better.
  • Doing a prototype should be team decision and the team should be open to explore alternatives.
  • The product owner should gain visibility into the prototype, so if the prototype yields that something is complicated, the product owner can offer alternatives to simplify the work.
  • A spike can also be used as an experiment – trying out a feature but not sure we want to keep it. The feature might be done in a branch and only used for feedback purposes.
  • Spikes should be estimated and entered in the backlog as technical stories
  • If a story is estimated high because – “we just don’t know how to do this…” – don’t bother putting a large estimation; instead leave the story as not estimated. Do the spike and then go back and re-estimate (or adjust the large estimation after you learned the results of the spike).
  • When possible defer commitment, if the complexity can wait, let it wait a bit. The longer you wait the more knowledge you gain. That complexity might turn simpler over time because you have learned things along the way and got feedback.
  • Time your spike – sometimes during the holidays it might be quite, the whole team is not present, take the opportunity to do a spike if you needed one.
  • Keep the code; even if it is a prototype code it correctly. However the definition of done may not apply fully to a spike – so in that case mention in in the acceptance what is the definition of done for this spike.

I have seen the two extremes, a team that never does spikes and have crazy estimations, and a team that makes 2 stories out of each story, one story for the spike and one story for the real story. Overall, try to remember that we all about offering value, where spikes does not. However, spike offer knowledge to help us estimate and de-risk, this knowledge eventually turns into value. So you should use spikes in SCRUM, but don’t overdo it.

How do you do architecture with SCRUM?

This is a hard one let me tell you. Before becoming a full time scrum master, I was a software architect full time. So when we had to work with user stories and translate them into code, architecture seems to have taken a back sit. However this is not true, architecture does not take a back sit in SCRUM in fact it is always there non-stop. Its not that architecture is not important - its need to fulfill the needs of the business and on its own its useless. So the business comes first, defines what we want and architecture is built all the time to meet the business needs - its on going and it never stops. I call it “just in time architecture”.  This is how it works:

The business drives architecture

everything needs a reason, you don’t just code it because you think its important. These can be looked as constraints of your system. For example you might have a constraint that you need to handle 1000 concurrent connections to your website, now this might affect how we build the site. There is a reason. We are not going to create a design that supports concurrency for fun. We are doing it because it is defined in a user story or constraint of a user story. Product owners don’t always think of constraints, so its your job to ask the questions. Do we need security? do we need concurrency? What kind of error handling do we want? A product owner that doesn’t really understand that will look at you funny and not understand. So make him or her understand. Give examples and give possible solutions. Provide choice so the product owner can make a decision.

But it still take a long time to code, so how do you present it in the backlog?

True. Even if the business gives you a good reason to do a framework that handles the needs, it takes time. It can even take more than a sprint. The solution I like to do is build it behind the scenes and build it by supporting real user stories. This way I get to validate my framework against real features. For this to work the stories need to be very very thin across the layers. For example, “add a user to the system”, might take a larger cost due to the infrastructure work behind it. Still, I only build the framework to add a user, not to delete a user and not to get a list of users. I keep my functionality limited to what I need and not beyond that. I expand my framework only based on new features that are required. This way I don’t build code that might never get used.

OK, but my kind of framework takes a long time to build  just to meet one story.

I would challenge that. You should be able to make stories small enough so some framework elements can be coded, and then add more framework elements when doing another story. If you still can’t around this idea - then lets have a use story that is a technical user story covering a technical feature. However technical user stories are bad and I only used them as a last resort. The reason they are bad is because product owners normally don’t understand them and therefore can’t properly prioritize them. So before you go the easy way out and get your product owner all confused with technical terms, see if you build your framework on the backbone of business need (or a user story).

Architecture never ends

You build your framework and design every sprint, every story. You make it better and better all the time. Its not just the first 3 months of development - its continuous and iterative. Within the development of every user story you might have tasks to improve the framework, and add features to it. Avoid at all cost big design up front, you will probably get it wrong because you don’t know everything yet. You will most likely not use half the stuff you build when doing big design up front. Design as you go, improve as you go. Think of building framework elements as tasks items to every story but not stories on their own.

Use patterns

Lots of pattern improve code and architecture - use them. Here are some of them:

IOC, AOP, DI, TDD patterns, BDD patterns and DDD patterns, (if you don’t know what this means start googling).

Use framework that are already done and tested

don’t reinvent the wheel and focus on the business needs and not the technologies. Unless you are a company that builds development frameworks, you should use what’s there and not re-build it.