Development to production pipeline

As I mentioned in an earlier post on test automation strategy we want code to flow smoothly into production in small increments in an automated fashion.

To be honest I don’t know whether we’ll do continuous deployment all the way, once hundreds of thousands of users depend on our site. I sure hope so, but as one of Bestofmedia managers says “By aiming for it we’re sure to end up with a very healthy and automated process. Developers will take on a very qualitative attitude to their work”. What’s more, those who do it IMVU, Digg4 and Flickr for instance are very satisfied with the results.

We’re not a big company that can afford to have dozens of engineers work for months on a CI and deployment infrastructure, but we’re a small dynamic and fit company and that’s all we need because we look for simplicity. Here’s the autonomy of our current system.

A walking skeleton

So we start of on a new product and instead of jumping into development right away we start by creating the walking skeleton as described in Freeman and Pryce’s excellent book GOOS. It’s a skeleton because it does the simplest possible thing, a “Hello world” for instance. But it is walking because it is under version control, it runs its unit tests, some sort of functional tests, the code is continuously inspected by Sonar and it gets deployed to some sort of production like environment. All of this is scripted ofcourse. By the way that was our first story.

Code review by pair programming

We pair-program for most development work, we don’t force ourselves to do it 100%, but most of the team members feel that pair-programming is beneficial for

  • ensuring code is always reviewed
  • reduce the number of bugs getting into the system
    • Edsker Dijkstra – The humble programmer (Turing Award 1972)
      If you want more effective programmers, you will discover that they should not waste their time debugging, they should not introduce the bugs to start with.
  • not getting blocked
  • remove need for technical documentation
  • avoid SPOFs on the team (every part of the application is known by at least two people)

We also try to analyze our pair programming to get better and better at it. For instance, we’ve found it very useful to have a second laptop per pair so that the pair can split and join again without moving. Sure pair programming is sometimes inefficient, but in order to be a good judge of where it serves, we have to practice it in the first place.

The pipeline

This is what we thought we needed

As it turns out we didn’t need all that, well at least not yet. It turns out that we don’t create versioned binaries (the release stage above). The risk is that we can’t rollback to a previous binary and we can’t be certain that the combination of binaries tested in the QA stage is in fact the one we deploy to production. Sounds dangerous! Well it isn’t, at least today it isn’t. It didn’t happen yet and we’re more concerned to get features out to our alpha users (and it won’t hamper our speed later on).

Anyway the workflow is as goes : the pair commits some code to revision control. A Hudson job runs unit and framework tests before building snapshot binaries. These binaries get published to a mix of Artifactory, a Debian repository and a ftp server.

The QA Sanity job deploys the application to our QA server and runs our sanity tests (see previous post).

The preprod sanity job deploys to servers that are not in production but in a very production like environment. That is no easy root access. It runs the same sanity tests and if all is successful proceeds to production where we run the sanity tests again.

Latest we looked this cycle ran 48 times a day.

What if something goes wrong in the pipeline?

The hudson job goes red and the team is notified by mail. Tasks further down the pipeline won’t run until the previous ones are green. In the future perhaps we’ll need to make an automatic rollback if the deployment phase fails. But lets wait and see.

This entry was posted in Agile, Continuous Delivery, General, TDD, XP and tagged , , , . Bookmark the permalink.

Comments are closed.