[Game, Jeu] 3ème jour

Leave a comment

Le week-end a été riche en événements.  Déjà, première surprise en ouvrant ma boite à lettre.

IMG_20160109_113636

Mon bienfaiteur fait preuve d’une belle organisation, et je note l’effort sur le timbre. En tout cas, cela fait très plaisir de voir l’imagination déployée. Du coup, ce matin, je me dis, faut que je trouve quelque chose de chouette aussi à faire, et chemin faisant, je me demande aussi qu’elle idée mon bienfaiteur va avoir aujourd’hui …

Le temps de faire le tour du bâtiment, de regarder par la fenêtre  et bien ….. mon bienfaiteur à déjà œuvré.

Un petit paquet m’attends sur mon clavier :  IMG_20160111_082153.

La journée commence très bien.

Je vois que d’autre bienfaiteurs ont agis dans le bâtiment : Dorine a visiblement un fan IMG_20160108_082649, et une personne aura une agréable surprise en récupérant sa voiture à midi : IMG_20160106_171848.

 

A mon avis, les discussions au café et à midi sont déjà toutes trouvées. Notons, l’ambiance “Bisounours” comme dirait certain qui règne dans les bureaux depuis quelques jours. En tout cas, j’entends plus parler de problèmes de build :-D.

 

 

Agile Grenoble 2015 : un projet Live

2 Comments

L’idée

Un jour, une idée un peu folle m’est venue. Plutôt que de parler d’agilité, montrons la en pratique et invitons les gens à venir la goûter.

Du coup, j’ai lancé un petit appel à contribution pour être une petite tribu de fou.

Capture_d_e_cran_2015_11_22_a_16.28.50

Xavier a été le premier à répondre, suivit de prés par Matthieu, Jean-Charles, Stéphane, Franck, ….

Quelques tweets alléchants et un coup de téléphone on suffit pour lancer le projet : le rendez-vous est pris, le jour J sera Agile Grenoble 2015. Les orgas (vivent les hommes en rouge) qui nous suivent, encore une fois merci! , et nous voila lancé.

Nous avons avec Xavier et Franck vu dans d’autre conférences des tentatives pour faire un tel projet. Les organisateurs se sont souvent confrontés à la difficulté d’inclure dans le projet des personnes de passages (personnes qui n’ont pas l’habitude de travailler avec 500 personnes dans le dos, de travailler en pair, sur un projet inconnu ….).

Pour remédier à cela, notre idée, a été de combiner à la fois une équipe technique (2 à 3 personnes) à 1 product owner et 1 scrum Master.

L’équipe technique

L’équipe technique a été présente tout le temps, pour assurer le passage de relai, la connaissance technique, et le déploiement de l’application.  Ce groupe à ainsi pu accueillir entre 3 et 6 personnes chaque itération (une itération durée environs une heure). Chaque itération a permis de discuter du fonctionnement de l’équipe, du rôle du développeur (conseils/échange avec le PO, excellence technique bien dosée, …).

Le(s) Product Owner(s) (POs)

Le PO avait un rôle de gestion du produit, il devait tenir compte des participants pour adapter le produit. Le PO avait aussi un rôle de gestion des questions, étant “en théorie” plus disponible que les développeurs.
Lors des 2 premières itérations beaucoup de personnes (débutantes et expertes) sont venues jouer aux POs en définissant des user stories et en gérant les priorités.

Nous avons ainsi, changé notre idée initiale de produit dès la fin de la première itération, ou notre vingtaine de POs a eu de très bonnes idées suite à notre première Démo.

Le Scrum Master

Le Scrum master, avait un rôle de gestion des impédiments (protection de l’équipe de dev:)) , inclusion des nouveaux en leur présentant le projet, animation des points de synchronisations.

Le rôle de SM est avec le rôle de PO, je pense un des points clés du succès de la journée, ils ont permis de faire l’explication en temps réels de ce qui se passe, tout en donnant à l’équipe de dev les orientations attendus par le public.

Le détail de la journée a été parfaitement raconté par Xavier, ici. Je vais donc jouer au fainéant et me concentrer sur un ou deux points techniques que je voulais partager.

Un peu de technique

Le repo GIT est disponible ici sur github.

Les technos utilisées sont, pour le Front end :

  • un peu de Php/Css pour la version 1 ultra simple
  • une lib javascript pour les V2 et V3 de l’affichage.

Un exemple de résultat ici (les tweets de la dernière heure à Agile Grenoble):

AgileGrenoble2015-tagDerniereHeure.png

Pour le backend :  Flink en Java/Scala .

Flink est un framework de processing Big Data, certains me diront n’est ce pas un peu beaucoup pour 75 tweets par heure ? Peut-être, mais un de mes buts était de me faire plaisir avec un framework que j’adore, de le faire découvrir ainsi que de permettre facilement de manipuler des flux de tweets.

Ci dessous, tout notre code produit (après quelques refactoring).

TwitterFilterSource twitterSource = new TwitterFilterSource();
twitterSource.trackTerm(#AG15);
DataStream<String> dynamicjson = env.addSource(twitterSource);

DataStream streamTwits = streamSource
.map(…)
.flatMap(new TokenizeFlatMap())
.filter(new RemoveLink())
.filter(new RemoveStopWord())
.keyBy(0).sum(1)
.map(… });

De ce point de vu, succès.

J’inclus ici, un des documents initiaux qui ont permis d’avoir une idée du logiciel que nous aimerions faire durant cette journée :

FlinkTwitter.png

La géolocalisation des tweets a disparu, en fait, très peu de tweet sont géolocalisés (~2% pour agile Grenoble).  Nos PO ont voulu un affichage qui alternait entre nuage de tweets et nuage de twittos (pour stimuler leurs activités), et un compteur de tweet est apparu.

 

Conclusion

En conclusion, que retenir d’une journée si riche ?

Un peu de frustration de ne pas avoir été un PO et un SM sur le projet.

Plein d’échanges, et du partage. Des idées d’amélioration. Une envie de recommencer vite.

Un grand merci à tous les participants et surtout à Xavier.

 

 

Maven 3 : avoiding duplication of information, make stuff easy to maintain

Leave a comment

This post in the 3rd one of a series on maven3. You can find the information related to the demo project used in example here.

One of the painful thing with all build management system, is the maintenance.

Setting it up at the beginning is ok, but maintaining it over time take time. Ensuring that all tools run on all modules, with same version, that the experiment you made on one module, or on a subset of module do not brake the whole things ….

So, in this post will see some tips to avoid information duplication in order to ease maintenance.

1: use pomParent mechanism to share most of the configuration

The pom parent mechanism is an “inclusion like” mechanism. It’s well defined here.
In the demo project, you have it located in the PomParent directory, and it’s used in all the others pom files.
In the following pomParent I am :

  • defining common properties (java version, encoding, plugin version) use in all pom files
  • defining plugin management element for all plugin used in one of the pom, in order to have plugin version centralized in one location
  • defining developer, license, organisation in one location

PomParent content :

   <project>
<modelVersion>4.0.0</modelVersion>

<!– The Basics –>
<groupId>com.persistentsas.mavenTraining.PomParent</groupId>
<artifactId>parent</artifactId>
<version>1.9-SNAPSHOT</version>

<packaging>pom</packaging>
<properties>
<!– build mechanism property –>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<!– Plugin having a custom setting –>
<maven.pmd.plugin>2.7.1</maven.pmd.plugin>
<pmd.java.version>1.7</pmd.java.version>
….
</properties>
<dependencyManagement>
…..
</dependencyManagement>
<modules></modules>
<!– Build Settings –>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-resources-plugin</artifactId>
<configuration>
<encoding>${maven.resources.plugin.encoding}</encoding>
</configuration>
</plugin>
……
</plugins>
<pluginManagement>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-release-plugin</artifactId>
<version>${maven.release.plugin}</version>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-site-plugin</artifactId>
<version>${maven.site.plugin}</version>
</plugin>
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>findbugs-maven-plugin</artifactId>
<version>${maven.findbugs.plugin}</version>
</plugin>
…………….
</plugins>
</pluginManagement>
</build>
<reporting></reporting>

<!– More Project Information –>
<name>parent</name>
<description></description>
<url></url>
<inceptionYear></inceptionYear>
<licenses>
<license>
<name>Apache License, Version 2.0</name>
<url>http://www.apache.org/licenses/LICENSE-2.0.txt</url&gt;
<distribution>repo</distribution>
<comments>A business-friendly OSS license</comments>
</license>
</licenses>
<organization>
<name>PersistentSAS</name>
<url>http://www.persistentsas.fr</url&gt;
</organization>
<developers>
<developer>
<id>LaurentT</id>
<name>Laurent Tardif</name>
<email>Laurent.Tardif at persistentsas.com</email>
<url>http://www.persistentsas.fr</url&gt;
<organization>PersistentSAS</organization>
<organizationUrl>http://www.persistentsas.fr</organizationUrl&gt;
<roles>
<role>architect</role>
<role>developer</role>
</roles>
<timezone>+1</timezone>
<properties>
<picUrl>http://www.persistentsas.fr</picUrl&gt;
</properties>
</developer>
</developers>
<contributors></contributors>

<!– Environment Settings –>
<issueManagement></issueManagement>
<ciManagement></ciManagement>
<mailingLists></mailingLists>
<scm></scm>
<prerequisites></prerequisites>
<repositories></repositories>
<pluginRepositories></pluginRepositories>
<!– not inherited –>
<distributionManagement>
<site>
<id>${project.artifactId}-site</id>
<url>file:///c:/temp/site/${project.artifactId}/${project.artifactId}.${project.packaging}</url>
</site>
<repository>
<id>GrenobleWrite</id>
<name>artifatoryWrite</name>
<url>http://MyBinaryRepo:8081/artifactory/libs-release-local/</url&gt;
</repository>
<snapshotRepository>
<id>GrenobleWrite</id>
<url>http://MyBinaryRepo:8081/artifactory/libs-snapshot-local/</url&gt;
</snapshotRepository>
</distributionManagement>
<profiles></profiles>
</project>

Inclusion in other pom :

<parent>
        <groupId>com.persistentsas.mavenTraining.PomParent</groupId>
        <artifactId>parent</artifactId>
        <version>1.9-SNAPSHOT</version>
<!– hint path, not mandatory –>
        <relativePath>.\PomParent\parent.pom</relativePath>
    </parent>

2: use properties to centralize in a common place important informations.

This is the first advise I’ll give. Use properties to put in a common place, informations your need to share.

you can define as many properties you want, and reuse them in any part of the pom file. If you define them in a PomParent, you can reuse them in all pom files. That’s help having easy to maintain project.

Among the practices i liked (but not every one agreed :-), I’m agree also to say that in the example, it’s a bit too much :-), but it’s an exercise … ), is to duplicate variable definition, in order to know where it’s used. That make also my life easier when the value need to be changed locally.

<properties>
<!– build mechanism property –>

        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
        <project.reporting.outputEncoding>${project.build.sourceEncoding}</project.reporting.outputEncoding>
        <maven.resources.plugin.encoding>${project.build.sourceEncoding}</maven.resources.plugin.encoding>
        <maven.compiler.plugin.encoding>${project.build.sourceEncoding}</maven.compiler.plugin.encoding>

        <maven.deploy.plugin>2.8.2</maven.deploy.plugin>
        <maven.checkstyle.plugin>2.7</maven.checkstyle.plugin>
….
</properties>

 3) define common build step in the pom Parent

In the pom parent you can define common build section in order to centralized that.

Now Imaging you have 4 modules, let’s say two standard jar libraries, and two webservices with code generation, and so on….

The build steps are not common to the 4 modules, but in this case you can make 3 pomParent :

1) first one common to every one (with dependencies version, common variables, … )

2) Second one, including first one, and defining build step for jar

3) 3rd one , including first one, and defining , source code generation, and so on for webservices.

Maven 3: some useful tips

Leave a comment

Maven is a build tool, but not only. It’s able to manage to whole lifecycle of your library/application.

I’ll use the project describe in a previous post and the source is available on github.

 

 1 : Work online / offline

One of the common usage of maven, is to work most of the time connected. Before going deeper, let explain quickly how maven work. when you run the following command to display the project hierarchy :

mvn dependency:tree

It will check on you build repository for new snapshots, or dependencies update to display it. There’s lot of command that will lead to an update of either your project dependencies, or the plug-in used by maven to build your project.

In order to control when you check update or not, you have the –offline to disable all remote access, and the -U as example, to force verification of updates. The maven –help command list all the options possible at Maven level, it’s independent of the content of your pom, or of any setting you defined.

mvn –offline dependency:tree

 

2: Use several thread to work

By default, maven will use One thread to work. Since maven3, you can use several thread to do the job.

You have 2 options to define the number of threads to use

  • The first one is by explicitly specifying the number of threads to use (2 in the example)

mvn -T2 compile

  • The second one is by specifying the number of thread relatively to the number of core you have on the PC (2 thread per core in the example)

mvn -T2C compile

The second option is very interesting on a heterogeneous build farm or in script, where you can adapt you number of thread regarding the number of cores available at run time.

 

3: Build partially your project

You can also optimise your build time , when running  :

mvn compile

Maven will build the list of project/module to build, and will determine a build order.  For each module, it will check if building it make sense (if no sources, tests or ressources change, there’s no point to rebuild the project).

Even if the verification is efficient, doing it on large project make no sense when, as a developper, you know where you change stuff, and want to do only a partial build.

Let’s see some running examples :

mvn compile

will generate a full build

[INFO] Scanning for projects…
[INFO] ————————————————————————
[INFO] Reactor Build Order:
[INFO]
[INFO] module1
[INFO] module2
[INFO] project1
[INFO] P2module1
[INFO] project2
[INFO] Root
[INFO]
[INFO] ————————————————————————
[INFO] Building module1 1.9-SNAPSHOT
[INFO] ————————————————————————
[INFO] Compiling 1 source file to C:\Dev\GITHUB\mavenTraining\Project1\module1\target\classes
[INFO]
[INFO] ————————————————————————
[INFO] Building module2 1.9-SNAPSHOT
[INFO] ————————————————————————
[INFO] — maven-compiler-plugin:2.5.1:compile (default-compile) @ module2 —
[INFO] Nothing to compile – all classes are up to date
[INFO]
[INFO] ————————————————————————
[INFO] Building project1 1.9-SNAPSHOT
[INFO] ————————————————————————
[INFO]
[INFO] ————————————————————————
[INFO] Building P2module1 1.9-SNAPSHOT
[INFO] ————————————————————————
[INFO]
[INFO] — maven-compiler-plugin:2.5.1:compile (default-compile) @ module1 —
[INFO] Nothing to compile – all classes are up to date
[INFO]
[INFO] ————————————————————————
[INFO] Building project2 1.9-SNAPSHOT
[INFO] ————————————————————————
[INFO] ————————————————————————
[INFO] Building Root 1.9-SNAPSHOT
[INFO] ————————————————————————
[INFO] ————————————————————————
[INFO] Reactor Summary:
[INFO]
[INFO] module1 ……………………………………. SUCCESS [  1.020 s]
[INFO] module2 ……………………………………. SUCCESS [  0.017 s]
[INFO] project1 …………………………………… SUCCESS [  0.001 s]
[INFO] P2module1 ………………………………….. SUCCESS [  0.013 s]
[INFO] project2 …………………………………… SUCCESS [  0.001 s]
[INFO] Root ………………………………………. SUCCESS [  0.001 s]
[INFO] ————————————————————————
[INFO] BUILD SUCCESS
[INFO] ————————————————————————
[INFO] Total time: 1.240 s
[INFO] Finished at: 2014-10-07T10:00:38+01:00
[INFO] Final Memory: 11M/214M
[INFO] ————————————————————————

With such small project, optimizing build time in a intellectual game, but can save lot of time on large project 🙂 So, let’s optimize it.

You can launch the build from a module level (ex: module2). We are going to use the maven “-rf”  parameter to do so (resume from).

[INFO] Scanning for projects…
[INFO] ————————————————————————
[INFO] Reactor Build Order:
[INFO]
[INFO] module2
[INFO] project1
[INFO] P2module1
[INFO] project2
[INFO] Root
[INFO]
[INFO] Using the builder org.apache.maven.lifecycle.internal.builder.singlethreaded.
count of 1
[INFO]
[INFO] ————————————————————————
[INFO] Building module2 1.9-SNAPSHOT
[INFO] ————————————————————————
[INFO]
[INFO] — maven-compiler-plugin:2.5.1:compile (default-compile) @ module2 —
[INFO] Nothing to compile – all classes are up to date
[INFO]
[INFO] ————————————————————————
[INFO] Building project1 1.9-SNAPSHOT
[INFO] ————————————————————————
[INFO]
[INFO] ————————————————————————
[INFO] Building P2module1 1.9-SNAPSHOT
[INFO] ————————————————————————
[INFO][INFO]
[INFO] — maven-compiler-plugin:2.5.1:compile (default-compile) @ module1 —
[INFO] Nothing to compile – all classes are up to date
[INFO]
[INFO] ————————————————————————
[INFO] Building project2 1.9-SNAPSHOT
[INFO] ————————————————————————
[INFO]
[INFO] ————————————————————————
[INFO] Building Root 1.9-SNAPSHOT
[INFO] ————————————————————————
[INFO] ————————————————————————
[INFO] Reactor Summary:
[INFO]
[INFO] module2 ……………………………………. SUCCESS [  0.689 s]
[INFO] project1 …………………………………… SUCCESS [  0.002 s]
[INFO] P2module1 ………………………………….. SUCCESS [  0.016 s]
[INFO] project2 …………………………………… SUCCESS [  0.002 s]
[INFO] Root ………………………………………. SUCCESS [  0.003 s]
[INFO] ————————————————————————
[INFO] BUILD SUCCESS
[INFO] ————————————————————————
[INFO] Total time: 0.900 s
[INFO] Finished at: 2014-10-07T10:04:05+01:00
[INFO] Final Memory: 7M/214M
[INFO] ————————————————————————

 

or even directly from a project :

[INFO] Scanning for projects…
[INFO] ————————————————————————
[INFO] Reactor Build Order:
[INFO]
[INFO] project2
[INFO] Root
[INFO]
[INFO] ————————————————————————
[INFO] Building project2 1.9-SNAPSHOT
[INFO] ————————————————————————
[INFO]
[INFO] ————————————————————————
[INFO] Building Root 1.9-SNAPSHOT
[INFO] ————————————————————————
[INFO] ————————————————————————
[INFO] Reactor Summary:
[INFO]
[INFO] project2 …………………………………… SUCCESS [  0.003 s]
[INFO] Root ………………………………………. SUCCESS [  0.068 s]
[INFO] ————————————————————————
[INFO] BUILD SUCCESS
[INFO] ————————————————————————
[INFO] Total time: 0.252 s
[INFO] Finished at: 2014-10-07T10:04:53+01:00
[INFO] Final Memory: 6M/214M
[INFO] ————————————————————————

 

Not so bad, we reduce build time by 80% 🙂

As you see, maven will scan for project,  construct the ordered list of projects, and will start build at the module or project.

You have several option to control more precisely the resume, let see some :

Options:
-am,–also-make                         If project list is specified, also build projects required by the  list
-amd,–also-make-dependents            If project list is specified, also build projects that depend on  projects on the list
-B,–batch-mode                         Run in non-interactive (batch)  mode
-fae,–fail-at-end                         Only fail the build afterwards;  allow all non-impacted builds to  continue
-ff,–fail-fast                                   Stop at first failure in reactorized builds
-N,–non-recursive                     Do not recurse into sub-projects
-pl,–projects <arg>                   Comma-delimited list of specified  reactor projects to build instead  of all projects.
A project can be  specified by [groupId]:artifactId  or by its relative path.
-rf,–resume-from <arg>          Resume reactor from specified project

Maven 3 standard layout

2 Comments

I have help a team recently to move to maven 3, and ask for “what’s the standard way to organise a project ?”.

Using Maven, you are encourage to follow convention to organise your library sources, but working at a project level you have more freedom.

When you use Maven since years, yo know there’s some good / bad practices that will make your project easy to maintain or not. Recently, some plug-in maintainer have advise a structure, and I’m pretty align with it.

So let’s describe it, and see how to benefit from it. For those interested, you can clone the repository on github, to have the files locally on your PC ( https://github.com/LaurentTardif/mavenTraining )

The project is made of :

  • 3 java libraries (2 in project 1 : module1 and module2, and 1 in project2)
  • one build-tools project
  • one pom parent project

We’ll saw later the benefit of this structure.

+—build-tools
|   \—src
|       \—main
|           \—resources
|               \—build
+—PomParent
+—Project1
|   +—module1
|   |   \—src
|   |       +—main
|   |       |   \—java
|   |       |       \—org
|   |       |           \—persistentsas
|   |       |               \—mavenTraining
|   |       \—test
|   |           \—java
|   |               \—org
|   |                   \—persistentsas
|   |                       \—mavenTraining
|   \—module2
|       \—src
|           +—main
|           |   \—java
|           |       \—org
|           |           \—persistentsas
|           |               \—mavenTraining
|           \—test
|               \—java
|                   \—org
|                       \—persistentsas
|                           \—mavenTraining
+—Project2
|   \—module1
|       \—src
|           +—main
|           |   \—java
|           |       \—org
|           |           \—persistentsas
|           |               \—mavenTraining
|           \—test
|               \—java
|                   \—org
|                       \—persistentsas
|                           \—mavenTraining
\—src
\—site
\—markdown
\—training

Now, let add some files inside this structure ;

  • Some java files in sources and tests. The current setup is minimalist, and contains one file per project.
  • A pom files per library, and some others to make life easier
  • Some files in the build-tools repository, these files will be use to share some common configuration or settings among  libraries.
  • A package-info.java to demonstrate package documentation in Java
  • a Site repository with some markdown file (.md), to demonstrate how to provide documentation up to date with your project.

C:.
|   .gitignore
|   LICENSE
|   pom.xml
|   README.md
|
+—build-tools
|   |   pom.xml
|   |
|   \—src
|       \—main
|           \—resources
|               \—build
|                       checkstyle.xml
|                       lib-filter.xml
|                       LICENSE.txt
|
+—PomParent
|       parent.pom
|
+—Project1
|   |   pom.xml
|   |
|   +—module1
|   |   |   pom.xml
|   |   |
|   |   \—src
|   |       +—main
|   |       |   \—java
|   |       |       \—org
|   |       |           \—persistentsas
|   |       |               \—mavenTraining
|   |       |                       HelloWorld.java
|   |       |                       package-info.java
|   |       |
|   |       \—test
|   |           \—java
|   |               \—org
|   |                   \—persistentsas
|   |                       \—mavenTraining
|   |                               HelloWorldTest.java
|   |
|   \—module2
|       |   pom.xml
|       |
|       \—src
|           +—main
|           |   \—java
|           |       \—org
|           |           \—persistentsas
|           |               \—mavenTraining
|           |                       HelloWorld2.java
|           |
|           \—test
|               \—java
|                   \—org
|                       \—persistentsas
|                           \—mavenTraining
|                                   HelloWorld2Test.java
|
+—Project2
|   |   pom.xml
|   |
|   \—module1
|       |   pom.xml
|       |
|       \—src
|           +—main
|           |   \—java
|           |       \—org
|           |           \—persistentsas
|           |               \—mavenTraining
|           |                       HelloWorld3.java
|           |
|           \—test
|               \—java
|                   \—org
|                       \—persistentsas
|                           \—mavenTraining
|                                   HelloWorld3Test.java
|
\—src
\—site
|   site.xml
|
\—markdown
|   index.md
|
\—training
Training.md

Now, let describe what are the benefit of such layout.

  1. You have all items related to a project, inside a project (pom, source code, test code, resources)
  2. You are able to share common build related configuration file using the build-tools repository
  3. you have several pom to make a hierarchical build, let’s call them aggregator pom for the moment
  4. you have “master” pom at the root of the project, that allow you to easily build, tests all your projects, but all to run command on a dedicated module.
  5. you do not duplicated informations inside build (versionning, behavior, …) using pomParent mechanism.

We’ll see in the next posts, how these functionalities works.

[Hudson] Checker le nombre de jobs

Leave a comment

Si vous voulez avoir un job qui joue le rôle de sentinelle :


$url="http://10.190.200.87:8080/hudson/api/xml"
$path="c:\Temp\jobs.xml"
$expectedJobsNb=41
$client = new-object System.Net.WebClient
 #Download the hudson home page in xml, through the web service
$client.DownloadFile( $url, $path )
[xml] $content=get-Content $path
#Get the jobs list
$jobs=$content.SelectNodes("/hudson/job")

if ( $jobs.count -ne $expectedJobsNb ) {
    Write-Error "We do not have the good number of jobs $($jobs.count)"
    exit 1;
} else {
    Write-Output "We have the good number of jobs : $($jobs.count)"
}

Distributed CI is the present

Leave a comment

I’ve seen recently, lot of posts, articles, saying distributed CI is the future. Sorry guys, it’s the present.

When we look at evolution of tools, techniques, requirement, we saw the distributed tools having lot of success ? So, does build-master over design their build framework ? or do they already work distributed ?

In this article I’ll summarize why I think the CI is already distributed.

What’s distributed today ?

  • Teams … yep,  collocated team, became more and more uncommon. Industry requirements, specific knowledge make collocation of brains more and more difficult, even for Giant ….
  • Source Repository, as soon as your team is distributed, you need to distribute your source repository.  Of course, network speed, huge servers may give you the feeling you don’t need distributed source repository. But do you have really distributed administration ? does the team really use the power of your source repository ? Do you use pre/post build event ? how often they commit ?
  • Dependencies … of course if the team are not co-located, your build process should be able to generate a partially your application, and download up-to-date (or the version you want)  of required libraries. So shared directory, and this kind of stupid stuff, are difficult to sync across the world.  Java community have understand that with artifacts repository that you can setup in your enterprise. Indeed, some language, provide similar functionality (perl,php,ruby, python, …) but how difficult it is to setup a proxy/mirror locally (to have the process under control).
  • The build process, should allow dynamic download of dependencies. Does European, US, Indian, will download dependencies at the same location of course not ! Imagine in Visual Studio the ability to define a binaries repository for all your projects ? able to handle binaries not generated by visual ? or artefacts no generated by VS ….
  • The programming languages are more and more different. Once upon a time, project were C or ++,  Borland even Java. Now, some part are in Java, some in PHP, some in Ruby, …..  Even if you have only one programming language (C#as example), you may have to handle different version in your application ( .Net 1.0 up to .Net 4.0 as example). On large project, it’s may not make sens to keep all your libraries to the last level of your programming language. So it’s common today to have at least 3 programming languages, that you want to test on a least 5 major OS ( and if you want to test 32/64b, you double this number)
  • You clients became more and more heterogeneous, and so, you test platform is more and more complex.  If you write an application, you have to test it on at least 5 , 5 major OS, 5 major language, 5 major Web Browser, …. more than 125 combinations. Of course, if you minimized and combined some tests, you have at least  5 tests  platforms.
  • The two previous point, encourage the continuous integration framework to scale easily, and to distribute build on a farm of servers, and that, at the job level. Indeed, all your libraries, applications, doesn’t have the same constraints.
  • Static code analysis, syntax checker. As your team are distributed, don’t have the same knowledge, all your programmers can not follow the same rules at the same time. So, every thing should be configurable by libraries or project.
  • Reporting tools, as all the stuff is distributed, asynchronous, reporting tools should take that into account. Does it make sens to compare code coverage of a new project under ruby, and a legacy one in C ? Does it make sens to compare an experimented team in Java, and a team of beginners ?

So, if your build master take all these elements into account, instead of using centralized tools, you may end with a distrusted platform, allowing team to work at their speed, with their constraints, their knowledge, ….

Older Entries