Dec 28 2011
Dec 28

The next few days are like a timer counting down. Sometimes the conversations feel a bit like a NASA control room. We review and re-review our plans because this deployment isn't going to be a small one. It is likely in the top 3 in scope since I began over two years ago.

Day 12 - Testing

We started with Scrum, Scrum, Scrum, and Scrum of Scrums.

Testing continued on Day 12 - and all the while we continued to work through Product Manager user stories. Supporting artifacts were shared as well. This segued into the Project Managers reviewing points and starting to assign teams for the next timebox. For this cycle, we found that our story points were roughly double to the number of the points available for the next timebox. This requires a feedback loop with the Executive committee to prioritize completed set of user stories.

The Executive Technical Leadership Committee met where we discussed a variety of issues, successes, and follow ups from the prior week. This committee fulfills the role of a CTO.

Finally, I also reviewed a patent application.

Day 13 - Testing, Stories, and Horse Trading

We continued to work through user stories and test. We had a 90 minute meeting with the Executive Committee to discuss the priorities where some items were shifted lower on the priority list and, for intents, got knocked out of the upcoming timebox and into the next.

Testing is coming to an end at this point. Day 14 will be release day and so only the most critical items are prioritised to be addressed before we release. This means several status meetings throughout the day. In this timebox, we were still engaged with status meetings until later in the evening. This was partially due to a timebox that was 25% smaller than our normal 20 day period. It was also in part because we launched our new mobile experience transitioning from a third party, Verve, to an entirely Drupal based mobile site. This shift greatly enhances the user's experience on a mobile device by leveraging block logic from the desktop site and providing access to all Examiner.com content. The Verve experience was limited to the previous 30 days.

User stories and the supporting artifacts are being heavily examined at this point. We have to lock down our next timebox on Day 14 - and that is now less that a day away. We also anticipated that the morning release would be long based on the new features that have been made ready during the current timebox and previous ones. The good news is that the work of creating stories is largely done - there are a few holes that need to be filled - but there is also bad news. Even with the prioritization of scope, the points that will be needed for Timebox 6 exceed the points we have to expend. This will mean and tertiary review prioritisation prior to the planning days 1 and 2 for timebox 6. Fortunately the holidays have afforded us a week in which the team will be largely working on defects giving enough breathing room before the next timebox starting to do that evaluation.

Work went late into the evening. There was a 6 am MT release start time.

Day 14 - Deployment Day

We start at 6 am MT - a time which is a compromise between low traffic and a time when our entire distributed team can be available. Most of us work on deployment from our homes rather than in the office. We have several communication lines available during our deployments.

  • A logged "Live" IRC channel
  • A logged "Post Deployment Validation" IRC channel
  • This timebox we added a Voice line using Skype

We want everything logged during deployment. If we need to go back and see what someone was doing at any given time, the logged channels help in those forensics. Anything that is spoken in Skype is to be logged as well. The goal here is to allow for extremely rapid communication in an emergency that doesn't rely on the typing speed of any one person. One of the lessons learned in a previous timebox was just that - fingers, in a pinch, cannot trump the speed of someone saying out loud, "STOP".

Normally deployments, from start of code push to end of post deployment validation, take 2-3 hours. We were launching a new mobile site making this deployment a longer process. It was about 10 hours total if you include hot fixes that occurred after validation to mitigate defects that manifested after push. After the code push is complete and post deployment validation is done, team leaders huddle and discuss which issues that have arisen need to be addressed immediately. A rapid triage occurs were we identify what order the needed changes will be pushed. They are then fixed, pushed to staging, validated, pushed to production, and validated one at a time.

After deployment is complete, it is time to do the final lockdown of stories for the next timebox. Folks have one last opportunity to lobby for switching priorities. Stacey is busy preparing our demo plan for the next day and informing folks who will be presenting what order and for how long.

Day 15 - Demo and Retrospective

Demo Day - when we arrive we will start to prepare the Examiner lounge - it has a large screen that we hook laptops up to, a video camera so our virtual team can peek in and take part. We set up a Skype call and a video feed for folks to access. The demo lasts for about an hour with each Product Manager going over the work we've completed during the timebox. Generally several of the Development Team will present something as well. This time around quite a bit of time was spent on the new mobile experience and America Inspired which will, on the 9th of January, will boast a brand new flexible polling module with different states for roles and voting status.

The retrospective was held as a round table. It is designed to be a safe place where we can honestly discuss what worked and what didn't work in the timebox. It allows us to make adjustments to our process. We have changed things like timing for artifacts, length of our testing period, adding a environment synch and freeze between prod and staging for a few days after deployment, and adding voice to our deployments.

Rinse and Repeat

The process repeats itself. Currently we are moving into development for our 6th timebox, user stories and artifacts for timebox 7, and ideation for timeboxes 8 through infinity. We track our burn rates pretty closely and are able do a good job of estimating the time our stories will take. Every timebox we complete improves our team's efficiency. We learn new things about how to do our jobs better every 20 days. I am incredibly proud of our team and how our process has grown.

Image Credit from Flicker. "Stopwatch" by William Warby
sed under a Creative Commons License.

Bookmark/Search this post with:

About Drupal Sun

Drupal Sun is an Evolving Web project. It allows you to:

  • Do full-text search on all the articles in Drupal Planet (thanks to Apache Solr)
  • Facet based on tags, author, or feed
  • Flip through articles quickly (with j/k or arrow keys) to find what you're interested in
  • View the entire article text inline, or in the context of the site where it was created

See the blog post at Evolving Web

Evolving Web