To continue our story from part 1 and part 2…
The teams have determined their individual impediments to Continuous Integration. You, as the technical program manager, and the technical program team can take those impediments, with input from the teams can see the impediments to program-wide continuous integration. You have used a similar problem-solving approach, including the rule of three (which I learned from Jerry Weinberg) to generate some options for your problem solving. This is where it can get really expensive, because this is where you may need servers and build engineers. And, this is where you are going to start to see significant value, because this is where your program can start to come together every day, where you can get to releaseable product on a daily basis.
Once you do this, you may discover you have insufficient automated tests for your code. (You’ll uncover the next problem down the stack.) You may ask, “Is it still worth doing continuous integration?” I say, “Yes! Write more automated smoke tests as you write code.” But that’s just me. As with everything, your mileage will vary.
As a program manager, you can go back to your project pyramid and go back to the program charter, and go back to the tradeoffs you were supposed to discuss at the beginning of the program with the core program manager. Without continuous integration and the automated tests, you’ll be extending the time to release and possibly increasing the defects at release. With continuous integration, you may need to pay more to reduce the time to release and reducing the defects to get to release—or making it possible to release.
This is why I find the pyramid so helpful in having these discussions. You may need to show cumulative flow diagrams to explain your point. I especially like these cumulative flow diagrams that show features in integration. (You can do that if you use kanban.)
Now, let’s explore some paths I have seen programs use to move from staged integration to continuous integration.
Move From Staged Integration to Release Trains
If you are doing staged integration now, where you have a team of people integrating the software now, consider moving to a quarterly release train approach. Depending on how you implement release trains, you might be agile.
No matter how you implement release trains, every quarter, you integrate and have working product. That much is clear. So, at least four times a year, you have an entire product working in your program. The real question is how long does the product stay working? What is the cost of keeping the product working worth to you? (I don’t know, I’m asking.)
Move From Quarterly Release Trains to Monthly Release Trains
If you already have quarterly release trains, you know what it takes you to integrate the software every quarter. Are you stopping development every quarter to integrate for a few days to a week? If so, consider moving to a monthly train.
If it only costs you a few days, your cost of continuous integration is relatively low, and moving to monthly release trains is likely to be low. I would ask the teams if they can move to monthly release trains. This is where practicing to make the cost low and discovering the impediments to moving to real continuous integration is worth it.
This is still staged integration, but it’s reducing the staging time, and getting you ready for continuous integration. You might want to consider this if you are waiting for servers or build engineers, or other machines or people or training you have been promised, and “it’s coming.”
On the other hand, if you can barely get the product integrated for a few days every train, you have a high cost of integration. Moving to continuous integration inside trains is going to kill you. I have an even better plan. Move to continuous flow. This is the “just do it” philosophy.
Move to Kanban and Forget About Iterations
Some of you are thinking, “Ok, Johanna has lost it now. Staged integration was difficult, and release trains were difficult to integrate? Why move to kanban?” Because at some point, you will want to release the product. You do not want to wait for a quarter to end. If you start with continuous flow, and make your stories smaller, and each of your feature teams has to integrate as the feature team completes a feature, every single time they complete a feature, you will discover quickly what is going on.
If you visualize the entire technical program, and integrate every single sliver of a feature, you will be able to see the impediments and move to continuous integration more easily than if you struggle with more staged integration and more steps.
With iterations, you were like my college project, where you waited too long to integrate. You have too much work in progress across the entire program, and the costs of integration are killing your progress. Moving to kanban, continuous flow, will allow you integrate little bits (assuming your stories are small), and allow everyone to integrate a little bit at a time every day.
Grab a very large wall. You will need a large wall to visualize all the work in progress. Don’t bother with limits yet. First, visualize all the work in progress from every single team. Yes, for those of you with 47 teams, this is a huge pain in the tush. Well, if you don’t have continuous integration, that’s an even bigger pain in the tush, so get going with the stickies or the cards. Try a board that looks something like this:
Kanban for Continuous Integration
This kanban board has the backlog in the ready column, the development and the development-done columns before the system test on the local branch column. Then you can see the Waiting for Integration column. There’s a lot there. Finally, there’s the Done column.
When the teams realize how many local branches they are using, and how many features are waiting to be integrated, they might not even need the cumulative flow diagrams. But, I bet seeing the task boards even out and the cumulative flow look better will help them realize their hard work will pay off is worth it.
Use cumulative flow diagrams, make sure your feature teams are cross-functional, and I would ask each team to make sure they move a story across the board at least once every three days. Why three days? Because it’s less than once a week. I prefer once a day, and I realize that not every team can develop stories or slivers of features that small right away. But almost every team can learn to produce some sliver of a feature that they can deliver and show someone in about three days. If it takes a team about three days to deliver, they have the other two days to integrate and make sure nothing is broken. Now, in a week, they have delivered something of value, integrated it, and they can demo it. Ah, what a feeling. They have experienced what continuous integration is for themselves.
Multiply that by each and every team, and inside of no more than two weeks, each team has experienced continuous integration. Now, when the technical program team meets, they can intelligently discuss what they need for program-wide continuous integration. They will learn about collisions, where they need scripts, tests, tools.
What About the Waterfall Teams?
Some of you have waterfall teams in your programs, too. They need to use deliverable-based planning, rolling wave planning to get to the deliverables, and deliver something every quarter. This is called a staged delivery lifecycle. Or a design to schedule lifecycle. It’s an incremental lifecycle. Read more about these lifecycles in Manage It!.
Get Help if You Need It
Do you need a release or deployment engineer or team to help you with integration? I hope not. But as you start your program, if you are not accustomed to continuous integration, you might. As a program manager, I might have a release/deployment team to start to manage the risks, and see if I could help those people into feature teams and out of a release/deployment job.
Only you, the program manager, your product owner, and your program architect as the triad to assess the technical debt and business value of the backlog and technical risk can know the value of continuous integration on your program. Maybe you have well-behaved code, and the cost of continuous integration is not worth the aggravation.
But, if you are like the programs I see, the value of continuous integration is quite high. Consider the options for your program. The more teams you have, in general, the more value you will get from continuous integration. You will be able to see the state of the program more often, because the program will be demoable and releaseable more often.
My guidelines are these: small programs of up to three teams: let the teams decide. Programs of 4-8 teams: nudge the teams into continuous integration, using cumulative flow and show them the value of not having lots of work in progress. Programs of more than 8 teams: continuous integration is not negotiable. You either pay for it now, or you pay a lot more and have a disaster later.
If you see more options than I’ve outline here, please do comment.
