Water in the Boat, Leaks in the Boat

How do you get to zero defects?

Divide and conquer.

We took the following approach. We established a run team. The run team handled customer support issues and fixed existing production defects. Members from other teams would rotate through the run team every iteration. Thus, no one was stuck supporting production perpetually and everyone had an opportunity to learn what our production support issues were.

By having a few developers focused on production issues per iteration, the run team’s productivity would increase as they did not have to switch between new feature development and defect fixes. The run team also provided a firewall to the teams focused on new feature development. By protecting the feature teams from interruptions, the feature teams’ productivity also increased thus protecting our product roadmap.

We created the following metaphor.  We would call defects in production “water in the boat”. The run team would fix defects in priority order. By taking this approach the run team was able to empty the “water in the boat” i.e., fix all production defects over a sequence of iterations. We developed a zero tolerance policy regarding defects and we did not cut corners. We would have entry an exist meetings for each defect, just like we did for features and we would test drive our fixes. This helped ensure that we fixed defects right the first time and with test wrapped around the fixes they would squeak should some future change break what had been fixed.

Meanwhile, the feature teams were busy adding new features to our product. Initially, we did not have our regression tests automated. Thus, after an iteration, we would find that we introduced new defects into our product. The introduction of new defects was called “leaks in the boat”. These leaks existed in spite of pairing, TDD and ATDD. All of these practices did contribute significantly to the quality of our product but did not get us to zero defects. Only when we put in place an automated regression suite that ran daily against all customer configurations were we able to plug the “leaks in the boat”.

The metaphor of “water in the boat” and “leaks in the boat” was a powerful one. It helped everyone visualize an important goal for the run team and the feature teams. Developing a zero tolerance policy for defects did much to reduce our cost and increase our velocity. By having practices in place that ensured what was built today would not need to be fixed tomorrow, it enabled the teams to remain focused on new features and not be distracted by fixes.

We still have a run team, and it does handle the occasional production support issues. However, it now spends the majority of its time working on small enhancements that improve our product and customer experience but by themselves would never make it to the product backlog, which highlights another point. We have learned there is a benefit to providing some limited capacity for those small features which by themselves never make it to the backlog but in aggregate improves our product and brings benefit to our customers. It has made the difference between a good product and a great product.