Saturday, February 26, 2011

Release Retrospective

It deployed, then scaled and the feedback was positive. The product owner was delighted and the users loved it. This was the first major application to deploy nationally and enterprise wide in several decades. Our business partners held celebrations. The team was proud of what they have accomplished.

Many things were done right. Most importantly, we had the product owner in the room and the team was seeded with embedded coaches. Although many of the team members were employees, the iteration manager, analyst lead, development lead and test lead were skilled experienced agilist from a consulting firm known for its agile practices. The team used story cards, big visible charts, two week iterations, show and tells and iteration planning meetings, Pairing and continuous integration were used along with primitive automation of unit and functional tests. We could run a full regression test every three days.

However, the project had a near death experience. The agile team, living in a waterfall world was disconnected. Although it delivered functionality, it did not scale. It would barely run in the production environment for longer than 5 minutes.  How would it ever support the score of thousand of users planning to use it.

Mind you, the production environment was new. It was not understood and much had to be learned to make it scale. However, that did not matter to our business partners. If we could not figure it out, our users would never benefit from the team's efforts. We were on the verge of being yet another failed project for scaling reasons.

The team learned that for large applications, another kind of testing was required. Performance testing (PT). PT should not happen at the end as planned. By then it was too late.  This uncovered a dysfunctional attitude inside the team room. The team believed they delivered functionality and the infrastructure team delivered scale. Of course the infrastructure team believe it was the application that would not scale as their infrastructure was fine. We had the classic corporate finger pointing stand off.

The embedded coaches, the agile experts, did not prepare the team for this blocker. The team was very internally focused. It had no connection to the enterprise. It was clear that the team succeeded using agile ... but also nearly failed.

Nonetheless, we did succeed. We experienced bliss in our own ignorance. Yes, we practiced agile, but just barely. Yes we were able to build and fully deploy an enterprise release, but only through heroics and a swarm of engineers busting through a little understood infrastructure and round the clock performance testing and tuning staying just ahead of the deployment schedule.

We had much to learn, yet we knew we were on the right path. These lessons did not go unheeded. Our product owners had big plans. There was so much more they wanted to deploy. It was round two for the team.

Meanwhile, a second new enterprise application was in the works. It was having problems. We were asked to start a second agile team to help. We were pumped. We were ready. So we thought.

No comments:

Post a Comment