Making Continuous Delivery work with Scrum and Sprints

Scrum is promoting fixed-length sprints of 1, 2 or 4 weeks. We do 2. That means we plan for 2 weeks, the team works for 2 weeks, and then we’re ending the sprint with regression tests, release preparations, sprint review, a final sign-off, and the release. All engineering activities are set up around this. Now we want to release more often – continuously.

There’s a lot of good reasons to deliver continuously. Robert Johnson of  Facebook gave one of my favorite pep talks about this a while ago with “Facebook: Moving Fast At Scale”.

My requirement here is to stick to sprints of 2 weeks. I worked through a few alternatives, e.g. reducing the sprint length to one week instead. This would give us twice as many releases but that’s still nowhere near Facebook and others, and also wouldn’t solve the problem I wanted to solve since nothing would need to change. Another option that has been coming back regularly was Kanban, mostly because work flows continuously, without being boxed into sprints.

In the following I’m walking through what needs to change in order to make continuous delivery work with Scrum and fixed-length sprints.

Sprints vs. releases or: what are sprints for?

The main reasons for doing short sprints over long-term planning is the ability to respond to change and simply accepting the fact that long-term plans don’t work out anyways.

The reason why we ended up releasing after every sprint is that a) releases always create a certain overhead and it seems to make sense to batch up work and go through the overhead only once. And b) because it fits traditional project management thinking: once the planned work is done, it’s being signed-off and released.

But in the same way as we broke down work from a 12 months project into one sprint at a time during our transition to agile, it sounds reasonable to break things down further from batching up a release every 2 weeks into very small continuous releases.

Sprint review and final sign off

If releases are done every 2 weeks after a sprint is finished, it’s easy to combine sprint review and final sign-off. Again, a way of batching up things. But it doesn’t actually save a lot of time to batch up a sign-off, so we could as well do a quick sign-off after every story is completed. This has advantages anyways because a story is only then truly done after it’s signed-off and released. So the change that needs to happen is to de-couple sign-off of each story from the sprint review at the end of the sprint.

What remains is that the sprint review is an opportunity for the team to brag about the work they’ve done, to get stakeholders involved and updated and to make sure that the actual progress becomes visible and agreed upon. It has also advantages to show the current progress in a live environment because by the time of the sprint review, each story that is done is already running in production.

From a process point of view, the team would now be able to release continuously throughout the sprint. Now let’s get back to that “overhead” I talked about. The technical challenges need to be adressed, otherwise we’ll spend more time on the actual release than on development.

No junk in the trunk and code freeze

When releases are happening every 2 weeks, there’s always a bit of a touch down period at the end. Unfinished code is being finished, the last tickets are verified and time is spent to make sure that the Master is clean and ready to go. This often allows for a certain degree of sloppiness with the Master throughout the sprint. It’s not very critical if unfinished work ends up on the Master because there’s always enough time to fix it and clean it up. This needs to change if releases should happen “whenever we feel like it”.

First of all: No junk in the trunk! Master must always contain work that is finished and built to production quality. There is different branching strategies out there and documented. In this context they mostly come down adding two additional areas in the repository:

  1. Where work in progress happens. I recommend a separate feature branch for each story. Work remains solely on this branch until very high confidence is reached that a feature works.
  2. Where work gets integrated (but not yet pushed to Master). This is where finished stories are integrated with the latest Master – but outside of Master. Here, the remaining issues are caught and regression tests on related existing features are done. This should increase the confidence that a new story works and doesn’t break anything else to 100%. Then it goes into Master.

In an ideal world, this would allow us to get rid of good ol’ Code Freeze altogether. The different branches, the quality gates on each branch, and the integration down towards Master does exactly what a sprint-end code freeze does: make sure Master is clean and ready for a release.

Automate your testing

If each story should end up on a Master that is ready to release, then regression tests must be done for each ticket. Otherwise it’s hard to ensure that Master is really ready and recently added work doesn’t break anything. That’s where lack of test automation really does start to hurt.

There’s zillions of articles and books out there about this, so I’ll keep it short. The essence is: do it from the very beginning if you can. If it’s too late for that, invest some effort and get your regression tests automated, as close to 100% as you can.

This will decrease the overhead related to regression tests to as close to 0 as it gets.

Above I mentioned “quality gates”. These are all the different checks and tests a revision of your software must go through before it’s Done. Depending on your system they may consist of building your software, running unit tests, running static code analysis checks, running regression, UI and load tests and maybe some – hopefully not too many – manual steps. With many CI servers like Jenkins they can be automated and arranged in build pipelines. Such a pipeline runs them consecutively on a certain revision and only if it runs through until the end without failing in between, you’ve got a green light on this revision. I recommend using build pipelines.

Automate your deployments

Regression tests and deployments have a lot in common: if you only do them rarely, manual steps usually don’t hurt enough to automate it to near 100%. Now that we’re about to release very very often, this starts to hurt (= creates overhead). Release must be as lightweight, fast, and robust as it gets. If only 1 out of 10 releases makes even remotely trouble or fails, it’s hard to get the team confident enough to release frequently.

I recommend using a CI server, reducing deployments to a few clicks and add a suite of tests against your live servers into the script that tests the current deployment right after it’s done and fails or succeeds immediately.

I also recommend putting deployments as much as possible into the hands of the engineers who are writing the code or at least bring a member of the operations team into the Scrum team. This removes additional hand-overs and the team being blocked by others. Depending on the environment this is often difficult to achieve, but automating deployments down to a few clicks that never fail certain helps a lot.

Changing your definition of done

If all of the above works and releases indeed start to happen regularly and even after every story is done, it’s time to change the Definition of Done. Both the final sign-off and the release should be in there.

Up your skills!

A transformation to continuous delivery is a big step forward for a team. It requires a variety of skills to cope with all the technical and non-technical challenges. Being intentional about improving the team’s skills and being willing to spend time and money for this will definitely help. You can also learn along the way and learn from mistakes, but focus on better skills pro-actively will help making less mistakes, moving faster and gaining confidence within the team and outside.

Dealing with release problems

A live user environment much more likely breaks during a release, obviously. Hence a very common concern is that more frequent releases will as well introduce more frequent problems and actually increase the total amount of work necessary to deal with and fix all the problems.

But there’s also advantages: continuous releases consist of much smaller change sets. Hence it’s a whole lot easier to regression test them and to release them. And if something breaks, it’s also a whole lot faster to spot, understand and fix the problem and release a hotfix. Not to mention that continuous releases often happen within a day or so after the work is finished, so it’s likely engineers haven’t forgotten all about the details of a change.

There’s more ways of helping out on this, e.g. the ability to release only to a small sub-set of users, observe and then roll-out to all users after. Improvements like this should be evaluated.

Conclusion

Here’s a summary of what I’ve been working through on making continuous releases work with Scrum and fixed-length sprints:

  1. Decide to de-couple sprints from releases. Sprints are for planning, releases are just one more piece of getting work “done”.
  2. Move your final sign-off out of the sprint review (if that has been the case) and move it to the end of each story. Add this to your Definition of Done.
  3. Choose and implement the right branching strategy. Introduce feature branches and an integration branch and make sure work is being properly tested when it leaves a branch. Move only down to Master when things are really working and keep junk out of the trunk.
  4. Automate your testing as much as possible. Besides unit test coverage, implement as much automated tests as you need in order to gain the team’s and the product owner’s confidence, e.g. UI tests, load tests or even automated static code analysis. Use a CI server like Jenkins to tie all your automation together and make use of build pipelines.
  5. Automate your deployments down to a few clicks. Get the ability to execute the release into the Scrum team to avoid hand-overs and the team being blocked on the release.
  6. Release whenever you feel like it, be happy, and adding value to your live users quicker than ever.

What is your opinion about this? Are you doing the same? Did you face the same issues? Did you solve them in a similar way? How are you releasing your software when using Scrum and fixed-length sprints? I would be happy if you take a minute and leave a comment below.




  • http://moodyrain.com Rita Nguyen

    Love this and think it is an extremely viable way forward as we move into a technology world that is moving faster than anyone can “plan” for.

    The only thing I would be worried about is that developers & their product counterparts would need to be accountable for building the RIGHT feature/story, not just stable code. This can’t be caught in test automation and if it’s only caught in regression then you’re going to get a lot of collisions and churn. I believe this starts from the right company culture. Build a team that isn’t afraid to ask and challenge BEFORE they start building, especially critical for countries like Vietnam.

    Keep writing! Great insights.

    • hendrikbeck

      Thanks for your comment and for your encouragement, Rita! :)

      About your comment: yes, I agree. I agree that building the right product (vs. building something right) as well as challenging what is to be build are both important.

      But the advantage you gain with continuous delivery is that you can iterate a lot faster and actually find out quickly what works and what doesn’t. Robert Johnson put it well in the video saying “(…) unless you have an extremely good idea of what right is, slowing down to get something right just means you take longer until you find out that you’re wrong.”

      Continuous delivery puts a lot of focus on building stable code in extremely short cycle times that lets you then concentrate on finding out quickly whether your product assumptions are right. Add in a few more things like the right metrics, A/B testing, etc. so that you’re getting meaningful data and can improve in your next cycle.

  • Pingback: Building the Right Product with Hypothesis-Driven Development « Hendrik Beck

  • fangdi

    very interesting , in ideal case it will be great that we could have release in every 1 week sprint but in the real world ( or at least my company ) , it’s still far away from reality . anyway , this is great and I can take something home , keep writing and me looking forward to your next blogs :)

    • http://hendrikbeck.wordpress.com hendrikbeck

      Thanks for your interest and your encouragement, Huy, I appreciate it very much :)

      And I agree, the real world is often very different from blog posts. I would like to ask you the question why you think it’s not happening although you think it would be a good idea? What is the one biggest reason why it’s not happening?

      If it’s difficult to talk about it in public, I would be more than happy to chat with you in private and exchange experiences. Feel free to shoot me an email and maybe we can find ways for improvement together…

      Cheers
      Hendrik

      • fangdi

        Thank you for your answer , Hendrik , that’s so nice of you .

        You know , that’s not just only the technical problems we have to face when we want to go asymptotically to the ideal case , we also have to cope with different kinds of testing and integration and the most important one : changing the process of releasing . For that perhaps we have spent years to talk to so maybe a private discussion could be better .

        Thank you for your suggestion , I will send you an email and if you have time then it will be a pleasure to me to talk to you

        Cheers ,
        Huy

        • http://hendrikbeck.wordpress.com hendrikbeck

          Yeah, I believe it’s many different problems to solve. It would be a pleasure for me to discuss with you and see if I can give you some opinions and experiences. Would be more than happy to help if I can :)

          Cheers and looking forward to talk
          Hendrik

  • alansikora

    Great article, congratulations.

    • hendrikbeck

      Thank you, Alan, happy to hear you liked it :) Are you in the process of moving towards continuous deliveries yourself? I wish you good luck in getting it running smoothly…

      • alansikora

        Yes, we are starting to define how to make this whole process work well enough for us. Maybe later, when I have something better set up, I’ll write something and link you there!

        • hendrikbeck

          Awesome. Well, good luck setting up the whole process. And if you ever need someone to bounce ideas off of, you’re more than welcome to shoot me a message…

  • jstalp

    this is a great article. I am about to begin this process and found this extremely valuable. thank you!

    • hendrikbeck

      Thank you very much, happy to hear this. And hey, if you ever feel like picking my brain or sharing ideas and experiences, I’m very open to that. Always happy to meet likeminded people and learn and help… Good luck with your process!

  • http://www.amazon.com/Continuous-Delivery-Pipeline-Releasing-predictability-ebook/dp/B00VTZ2GSE/ref=sr_1_7?s=books&ie=UTF8&qid=1429554581&sr=1-7&keywords=continuous+delivery Juni Mukherjee

    I share your vision. My book on “Continuous Delivery Pipeline – where does it choke?” is releasing on June 1:

    http://www.amazon.com/Continuous-Delivery-Pipeline-Releasing-predictability-ebook/dp/B00VTZ2GSE/ref=sr_1_7?s=books&ie=UTF8&qid=1429554581&sr=1-7&keywords=continuous+delivery

    I sincerely hope we can embrace CD and Agile a no-brainer.

    Thanks,

    Juni.

  • Witko

    Great article.

    I’m a lover of being able to continuous delivery combined with the lean mentality of putting working code into the wild sooner rather than later. Point 1 in your conclusion is a must for any team who wants to minimise all sorts of problems.

    • hendrikbeck

      Thanks a lot! :-)

      And yeah, totally agreed on what you’re saying about #1. At the same time it feels like the hardest part to change people’s minds about this.

      In addition to the “working code” you’ve mentioned I would also add the ability to deploy code that is not working yet, aka. “dark-launching” (in any one of the different flavors). What’s your opinion about or experience with that?

      • Witko

        It really is the hardest thing to change! Quite a few people seem to be in the ‘shippable’ mindset not the ‘shipped mindset, as they see this as achievable within their sprint. Which restricts visibility of the whole and following things through to real completion.

        I’ve found the idea of dark-launching quite simple as there are a few patterns to achieve this. Either by constructing your systems in such a micro way, versioning efficiently, or by using features gates.

  • thisGuy

    sprint reviews are not supposed to be of items in production..

    • http://wendt.se/blog/ Fredrik Wendt

      I completely disagree! The point of the Sprint Review is to inspect a product increment – ie the product – _as it is right now, with only Done functionality (new and existing)_ in relationship to the market (which is why stakeholders are there), so that we can adapt the product backlog to maximize the (potential) value of future work. “Future work” may include “release to market” if we haven’t done so yet. It may be “roll out to everyone” if a feature is currently in production, but turned on only for a subset of users.

      • hendrikbeck

        A few additional words: I have come to value immediate or near-immediate releases of small features (hence reducing the amount of unreleased code) and immediate customer feedback *a lot*. The sprint review is more of a regular internal checkpoint of the team (and its stakeholders) to look at what’s been done and what that means going forward (e.g. re-prioritizing the backlog etc.). No need whatsoever in my eyes to wait for the sprint review before releasing features.

        • http://wendt.se/blog/ Fredrik Wendt

          +1!
          What’s the point of talking about what’s been done at the sprint review, in your mind? (I may be stuck on the words “do/been done”, as that’s not so interesting from a product point of view. What we’ve learned, and what attributes/features the product contains (and how they perform/impact users) should be more relevant than “this is work we have done”. But I’m guessing you’re also way passed that?)