That was one of the big question in a previous life. On a medium size project (20people) what’s the most valuable (and/or less risky)  for the team when we need to do emergency stuff.  I’ll try to give concrete example of benefit and risk of both approaches.

Before starting a bit of vocabulary :

  • Component : a part of a system, can be seen as one Database(DB), one web service(WS), one Backoffice, …
  • Library : can be seen as a DLL,jar, .so,… a peace of code. Several library made a component. A library can be shared among several components

So, when you need the fix a component, most of the time the change requires only an update of a small set of libraries, or sometime that can be solved by adding a new library.

Taking the assumption that such stuff are made most of the time during a rush, the first stuff to ensure is the quality of the change. There’s a bunch of techniques to help you to secure the change :

  • Pair programming
  • Code reviews
  • Risk analysis.

Once the change is done, the big question is what are you going to update on your production site. Even if you use Continuous Integration you may not want to deploy the current version of the application. On large system, some of the test are not easily done (performance,  GUI, …) and requires some human time, that you may not have (security issues, …).

So, you have mainly  two possibles approaches : ( I’ll try to describe some alternatives)

  1. Rebuild the component and deliver it totally. So, depending of your delivery process, you consider your Patch as a standard release and deploy it as usual. That requires first a build totally under control (no automatic update of third party library, no automatic update of resources file (translation, configuration),  no automatic refresh of your build environment from CVS/SVN (so you are using  tag and/or branches..).
  • The benefits of this approach is : if you need several patches, you take fewer risks, as all your patches are included in the release process. If you need to install a new box, your application is delivered as a block, and not as a main version and several patches to apply after the main install.  Your next release is more predictable as you may not forget some patches to include in it. You avoid the risk of the patch store only on a developer box at lost/forget after few days.
  • The risk of the approach, depend mainly of the quality of the build. If it’s totally under control, nearly no risk, if their is lot of fancy stuff (automatic numbering of some stuff, automatic update of library, automatic update of translation…) the patch can be very difficult to control.2.

2. The second approach is to do a patch, that is to say deploy only one library on the box where the component is installed. This seems less risky as you (think to) control totally what you deploy.

  • The main benefit is the list of impacted file is under control
  • The main risk is the lost of the patch. For example, if you create manually the patch, the files in the patch may not be stored in CVS/SVN. As the library may be build outside the standard release process you may expect some manual error (if your library can be build independently you avoid this risk). As the deployment may be different than the standard release process a lot of human factor can interact with your patch.

So, In fact, the magic solution to have both a secure and quick patch cycle, is to have a build process that is able to do incremental release (build only what’s needed), and controlable by a human (the human decided what’s needed).

If you don’t have such sofisticated build process you have to take care of :

  • Ensure the quality of the patch, ….
  • Ensure all the files modified for the patch are stored under CVS/SVN and they will be automatically take into account at next release
  • A risk analysis of the change : for example to fix a bug, you update a library L1, used by L2,and L3. By fixing an issue on L2, you can create a bigger issue on L3.
  • A quick sanity check to ensure the bug is fix, and doesn’t introduce new issue
  • Try to use the same deployment process than for a full release. Most of the time installation time is more link to time to stop/start the application than copying some megabyte of the box …
  • Ensure to track this patch for the next release.
  • Roll back plan : define it … as the cure can be worst than the disease …

I’ve tried to summarize the keys issues of doing a patch or a release, but this is some other alternative.

For example, if your component is install on two boxes, you can deploy the patch/release on the second one, have both boxes live for some time. As soon as you are confident enough with your patch, you can deploy it on the first boxes.

Advertisements