[Devops / Docker] how to manage files with docker

When starting using docker, lot of people store file either in the container instance itself either on the file system of the server. in this post, will see the different option to deals with files, and we’ll discuss in which case it’s a good approach.

With docker, like in general, there’s several way to manage files :
– directly in the container
– via a mount point on your server
– data container
– using a docker volume
– using a network file system (amazon S3, Hadoop, …) .

Files stored directly in the container :
The main advantage of this solution is it’s simplicity.
But there are several drawbacks:
– your container is not persisted, so, when your container crash, or is removed, you may loose your files. In some case it’s not important : temporary files, demo, files generated by a script, ….
– Performance, as docker use union fs, when reading and writing files it may not be as fast as expected.
– file sharing : if you want to share the file among several process, either you have several process in your container (not very good), either you may link strongly two containers (or have them in the same pod if you use K8S), that’s may not be very good too.

Files mount on the host :
One of the advantage of this solution is also simplicity, and it solves partially the problem of sharing a file among container (as long as thy run on the same server).
This solution have some drawback too:
– your container need to run on this server,
– the container need to have access in write (or read) to your host, it may not be secure.
– that’s to be avoid in production, because you create a hard link between the container and the host, and you loose some of the advantage of the isolation concept.

Files store in a data container :
Some people do that, …, I still not understand why doing that when you have volumes….

File in a cloud provider storage :
It may looks like a perfect solution if your containers run in the cloud, but you create a direct link between the container and the cloud provider. In some case you may not want that. So, use it carefully, being cautious of what your are doing. Using a volume may offer your the abstraction needed.

Files in a docker volume :
This solution may be a bit complex for beginner, as it introduce a new concept, but in fact it’s pretty efficient. The full documentation is available here. You can see it , as creating a new logical disk dedicated for your need, where you can configure the size, the name, the filesystem used.
The main advantages of using a volume are :
– it’s ease file sharing among containers
– it doesn’t make any link to the server where the container and the volumes are stored.
– it’s allow to scale, as you can define the filesystem you want inside it, you may even imagine to have a distributed file system like GlusterFS. (you may have a look to all driver available here) . You can even use directly cloud provider solution. So, it’s my favorite solution, as it’s offer also an abstraction level between your container and the file system.

Hope it give you some hint on how to manage files, i will try to give some performance metrics soon.

Why to we need to share a file in the docker world ? I will give some common use case see on several projects :

• As a container should run only one process, when you start an application, and you need to generate a configuration file at the start up (configuration file that may change over the time because you are using consul, or etcd, or vault ..). So you have a daemon process that scan change , and that update the configuration file, and then that will notify the application to read it. Having two processes in your document will be a mess, as if one of the two process die, you may be in a strange situation. A way to manage it, is too have have two side car container sharing a volume.
• If you want to push your logs in a central place (ELK, ….), you may use the same side container concept.
• Database files, you may store your files in a dedicated volume, that will allow you to start another psql process, using the volume in read-only mode, this will give you access to the data, without taking any risk with the process running your DB.

[Devops / Docker] How to write a good dockerfile

Reading or writing a docker file, as soon as you get a bit familiar with the syntax is quiet simple. But how many time you spent some time wondering why your image is so big, or why the image you download was so big, how many time lost trying to find the port, the volume defined, have you loose. So, writing a good docker file is not so easy. I’ll try to explain the process that we have defined with Jules.

First of all, what’s a good docker file ? is it a secure one ? , a readable one ?, one that generate a small docker image ? all that ? that’s a challenge.

In this article, we’ll not focus on security aspect, but we’ll explain how to write a readable docker file, easy to maintain and that will generate a small docker image.

Let’s start with a simple example :

FROM node
WORKDIR /usr/src/app
COPY babel.config.js ./
COPY package.json ./
COPY yarn.lock ./
COPY public ./public
COPY src ./src
RUN yarn
RUN yarn lint
RUN yarn build
RUN yarn serve

if we docker build this image, we can notice several point :

1. it may download a new version of the referenced image “node”
2. it create several layers. A layer is an immutable set of files that can be shared among several docker images for performances and disk usage optimization. If you use several images having the same root image, you will have the files of this images present only once on your disk.

if you rebuild this image, without any modification, you will notice that docker is using a cache mechanism. To give some hints, if nothing change on your disk (for example, you do not change the package.json) it will not redo the step (and if all the previous steps have not changed). In summary, docker will do the minimal stuff needed.

So, let’s take this 2 points in account.

FROM node:10.15.0-Alpine
WORKDIR /usr/src/app
COPY “babel.config.js” “package.json” “yarn.lock” ./
COPY public ./public
COPY src ./src
RUN yarn
RUN yarn lint
RUN yarn build
RUN yarn serve

So, to avoid a random node image being picked, i defined a strict version for it. And to avoid useless layer, i defined a layer with the 3 configurations files.

Now that we have reduced the two first “issues”, let’s take advantage of the cache mechanism.

FROM node:10.15.0-Alpine
WORKDIR /usr/src/app
COPY “babel.config.js” “package.json” “yarn.lock” ./
RUN yarn
COPY public ./public
COPY src ./src
RUN yarn lint
RUN yarn build
RUN yarn serve

So, I move up the “RUN yarn” just after the copy of the configuration file. Like that, as long as the configuration files do not change, my “RUN yarn” will not be executed, and I’ll benefit of the docker cache mechanism.

Now, I’ll not run directly use yarn to run my application. Let’s set up quickly an nginx server in front of it. So, I’ll use the multi stage feature of docker. It’s allow me to reuse the result of a docker image in an another one, define in the same dockerfile. Let’s do it.

FROM node:10.15.0-Alpine AS builder
WORKDIR /usr/src/app
COPY “babel.config.js” “package.json” “yarn.lock” ./
RUN yarn
COPY public ./public
COPY src ./src
RUN yarn lint
RUN yarn build
RUN yarn serve

# Volume inherited from nginx image
# VOLUME /usr/share/nginx/html
# VOLUME /etc/nginx

FROM nginx:1.12.1-alpine
EXPOSE 80 
HEALTHCHECK –interval=5m –timeout=3s \
CMD curl -f http://localhost/ || exit 1
COPY –from=builder /usr/src/app/dist/ /usr/share/nginx/html/

In order to do it, I’ve defined a logical name to my first image (I named it “builder”, it will allow me to refer to it in the second image).

Then I defined a new image, from nginx, and I copy the website content from the first image. So like that, my nginx image is minimal and contains only mandatory stuff (from a security stuff, the less you have, the better it is).

You can notice I’ve define the expose and healtheck in the top of my docker image definition. Because it a kind of API of my image, i make it quickly visible to the reader. You can access my image on port 80, there’s a check that insure the nginx server respond, the run and the copy element are a kind of implementation, and only few people want to read it.

Of course, we can improve this image, but if you reach this point, it’s a very good starting point for people using your image.

How to promote a change in a team or a group

Since several years, I’m helping team to transform, sometime in a Agile’s way, sometimes in a technical way (CI / CD / DevOps / Craft /…) .
All the times, I’m facing challenges. That’s normal, it’s a change, and you can not change persons.

Last week, we did an agile meeting with some friends, and we try to figure out / summarize the good practices to give to a tech lead, a scrum master, or simply to any change actors.

The first element to take into account, when you want to promote a change is to define a GOAL or an objective. This is important, because everyone will take a different path, but everyone should go in the same direction. The goal should be CLEAR, Achievable, and be time-boxed.

As example : everyone should have at least one project, with some valuable automated test, by the end of the month. Of course valuable, and some may lead to discussion. But it can be a team agreement to review others test to ensure value.

The second element is SHARING, that seems obvious, if you want to have everyone in the same direction, you should share this goal and objective. Often it’s implicit or not clear.  This will also give the necessary support to early adopter or leader to encourage everyone in the team to try the new method.

The third element is to value the success, the effort, and even some failures if they help person to learn.

Do not forget, everyone is different, the needs, the motivations, and the path will be different for everyone. So ONE solution or path will not satisfy everybody.

Either you have a good experience, and you may try to propose several path, adapted to everyone, elsewhere you may help everyone to find his path.

This post is just some notes for me, to remind me how important are this 3 steps in all transformation.

 

MixIT 16

MixIT arrive, l’équipe infernale qui avait sévit à Grenoble, se relance dans l’aventure de la journée Agile en Live.

Venez participer a une journée intense d’agilité, venez (re)découvrir les pièges classique d’un projet: techno push, story mal spécifiée, …. et venez parler avec des devs, POs et SM avec plus de 10ans d’agilité derrière eux….

Des erreurs on en a fait pleins, venez profiter de notre expérience.

 

Petit souvenir

[Game, Jeu] La fin ……..

La fin est la … le mystère est levé …..

Retrouvons tous les joueurs (~30) autours d’une bonne galette pour discuter et lever le voile sur les mystérieuses actions qui ont eu lieues.

Commençons par rappeler brièvement la chronologie des faits.

Tout le monde (du Grand Chef, aux employés, en passant par la RH et les managers, …. ) ont été invité individuellement et discrètement à participer à un jeu, sans savoir lequel. Le but en tant que maître du jeu (et oui, c’est moi…. roh… ), était de construire un cadre mystérieux et d’avoir des participants volontaires.

Dans un deuxième temps, j’ai informé les joueurs des règles du jeu, en précisant qu’il n’y avait aucun risque physique ou psychologique. En effet, certaines étape peuvent nécessiter un peu d’organisation ou de réflexion de la part des participants. Je précise aussi que le jeu ne prendra d’ailleurs que quelques minutes par jour aux participants. J’en profite pour annoncer le jour de démarrage du jeu, et la date de fin. Une semaine … nous allons pouvoir nous amuser une semaine …..
Sans vous dévoiler le jeu, je vous donne 2 clés essentielles du jeu :  mystère et générosité.

Le jour J arrivant, j’ouvre le jeu, et je précise les derniers détails du jeu.

Et la … une semaine de bonheur. Des surprises, des débordements de générosité, ….. En résumé, le jeu et l’ambiance dépassent mes plus folles espérances.

Tous les commentaires des participants donnent envies de recommencer dans un an … avec encore plus de monde.

J’ai même reçue ma petite part de bonheur en tant que maître du jeu.

La prochaine étape, dérouler ce jeu dans 2 autres sociétés, j’ai déjà trouver 2 complices…. affaire à suivre.

Et a la question “Pourquoi je n’en dis pas plus”, la réponse est simple,  pour vous laisser la chance d’y participer :-p

Si vous avez des questions, n’hésitez pas.