One question I’ve often got, is : Does CI is applicable to my small (or big) project ?

The answer is Yes ! The complexity of the setup, and the benefit of the CI are dependent of course,  but not to the size of you project, it’s dependent of the nature of what you integrate : library / application , and the deployment constraint you have on what you integrate : one OS, several OS, several version, ….

In this post, I’ll try to give some thought on the topic.

First of all, let start simple : The goal of the CI is to build high quality, reliable packages from a set of sources files, possibly edit by several person. This is shown in Figure 1.

Figure 1 : CI Goal

gif_2

The full process of CI can be summarized as (illustrated in Figure 2):

  • Edition of the source file (in an IDE or not, but … I’ll show in a future post than an IDE usage can enhance/speedup development process)
  • Validation local of the quality of the package : compilation / test / analysis with Static Code Analysis tools, packaging …
  • Share the file through a Revision Control tool (CVS, SVN, P4,…)
  • Remote Validation of the package (nightly build, black box build) : remote compilation / test / analysis / packaging …
  • History of change and notification to the team if there is issues (CI … can be integrated with previous point)
  • Distribution of the package to be use by the team / external team / …

Figure 2 : CI Process

gif_1

Now, Imagine we are working on two blocks or libraries, two strategies can be put in place. The first one consider the two libraries independent (event if B use A) from a life cycle point of view (B use a stable version of A for example), the second one introduce a life cycle coupling among the two libraries (this is illustrated in Figure 3) .  The choice of solution 1 or 2 depends mainly of the usage of both lib, and their release cycle.

Figure 3 : Integration of several blocks or libraries

gif_5

Let describe the situation where we consider A and B independent. In this case we simply duplicate the previous framework without any difficulties.

Figure 4 : A and B independent

gif_3

The second case is more powerful, that means we can do more fancy things with CI. Of course, we need to validate A and B independently (B can use previous A build version), AND, (the power of CI come from this AND 🙂 ) you can validate integration of A and B.  You can imagine complex and fancy mechanism to validate this integration, but a simple approach is to rebuild B with the new A, and to run B tests again.  So, in a first step you validate your new version of B and your new version of A, and after you validate their integration.  But, warning, by doing that you create a release constraint among A and B. Indeed, if you deliver B, with an old version of A, your integration is useless. You need to validate your library with libraries you deliver with.

Figure 5 : Dependency among A and B

gif_4

The big question, now is does it scale to a large project or not ? and a similar question, is do we apply the same strategy to the whole project.

Their is no magic solution, that’s depends and it will mainly depends of your block organization.  The complexity of having several block does not took to much time as you compile each block with previous version of dependents block (and you can do it in parallel), and you need to build the whole thing again (or just redo the link step for C/C++ lib for example).  You need also to validate this big block. Unit test is not enough, and you should run also integration test, or smock tests (subset of integration test, that can be run quickly to guarantee rapide CI feedback) (note : you can run the whole Integration Test suite during the night, or Saturday/Sunday).  This is shown in Figure 6.

But in order to reduce release cycle constraint I will encourage to have this scenario for all applications, and to having linking applications together.

For example, if you have X libraries, a 5 applications  (Front End, WebService1, WebService2, Database, and BackOffice) most of the time you’ll have :

  • Core components for you project (perl module, RPM, …. )
  • Front End
  • WebService Core , by isolating this block, you make your core block living without any constraint on WS1 and WS2, and you made de release of WS1 independent form the one of WS2.
  • WebService1
  • WebService2
  • Database
  • BackOffice

Figure 6 : Large Project

gif_6

So, if the CI scale without any problems (you just need to have enough servers to compile/test/validate packages in parallel to be fast enough :-D)  (For example Hudson provide a remote compilation module), where the difficulties come from ?

They come mainly from two origins :

  • your deployment constraints
  • your application

Let explain, if you need to deploy your application on 4 operating system (in this case all Linux based, in both 32 / 64 bits) you need to compile (or at lease emulate/simulate) this compilation for the 4 OS, and you need to Test your application on the 4 OS to avoid OS specific issues.

(You’ll have the same issues, if you need to deployed on several Apache server version, or Mysql, Tomcat, …)

Figure 7 :

gif_7

The second source of complexity, comes from your application. For example, if you provide an Internet application , you’ll need to validate (test) you application with several client configuration (browser / OS / Version ….).  That’s feasible, several framework allow to do that (selenium) for example, they are scalable (selenium Grid), and some techniques like virtualisation allow you to do it efficiently/easily.

Figure 8 : Several application Client

gif_8

So, don’t be afraid, think before implementing your CI, and GO !

In a future post I’ll give you team practices to follow to do even better !

Good luck !

Advertisements