This is a series that we will dedicate to the CI/CD pipeline and how you can use tools from ClusterHQ to help gain fungibility, efficiency and mobility to your CI/CD infrastructure and build pipelines.
The series will start with a focus on running Jenkins and how to provide flexibility and mobility running your master and its associated configuration. Then, the series will slowly move into other topics such as limiting non-deterministic failures, fungibility for slaves, greater efficiency for popular plug-ins and overall a more deterministic and repeatable Jenkins infrastructure.
What is Flocker
Flocker is a container data volume manager that allows stateful services to run inside containers in production. When running a database in production, it’s important to think about things like recovering from host failure and availability of data. Flocker provides tools for managing data volumes across a cluster of machines like those in a production environment.
For example, as a MySQL container is scheduled between hosts in response to server failure, Flocker can automatically move its associated data volume between hosts at the same time. This operation can be initiated using the Flocker API, CLI, or a container orchestration framework such as Docker Swarm, Kubernetes and Mesos.
Why use Flocker with your Jenkins Master
Running a Jenkins master deployment means downloading and installing various plugins to use for different build purposes and configuring those plugins according to your particular infrastructure environment. On top of that you will have logs, job output, job configuration, update data, artifacts, libraries and more associated with your Jenkins master for your build pipeline.
All of this data and configuration is stored within a directory
/var/jenkins_home. If you run your Jenkins master in a container you then need to consider how you make sure this data is safe and mobile in the cases where the container restarts, gets rescheduled or the host running Jenkins fails.
You can follow the suggestions on the Jenkins guide in DockerHub but even using the suggestions there, your data is not always safe because it’s not able to recover from more than host-level failure.
Three main ways to run Jenkins according to DockerHub
In the above case, all of your data resides inside the container’s own file-system storage. This means that your data is most vulnerable to loss because when the container goes away, so does the data.
In this case, your data is placed into a Docker volume. This means that the data is within
/var/lib/docker/volumes/ but won’t be safe if you lose your docker host. Also, if you start your Jenkins master on another node it will not move to the new node so it will be like booting a brand new Jenkins master.
In this last case, you are doing much the same as the previous case but instead of it being a docker volume you’re mounting in a directory that exists on the docker host that is outside of the docker file-system. This approach still leaves data vulnerable to node failure and container movement issues.
In the next portion of this post, we will explore how to have a more portable and fungible Jenkins master.
Fungible Jenkins Master
We will enable our Jenkins master to use an attached storage volume so that not only is your Jenkins home directory always safe, it will always follow your Jenkins master no matter which host it is on in a cluster of VMs.
In this example I have five EC2 nodes running Docker, Flocker and Swarm. I will use this cluster throughout this demo which will use EBS volumes for storage.
To start a Jenkins master with a Flocker volume, you can use the below example of a compose file that does this.
To deploy this setup you can run
docker-compose up like so.
What happens now is Flocker will provision any volumes that are not created yet and bring up your Jenkins master. After it completes you will have a running Jenkins Master.
Note: I am using a fake IP in the ouput, yours would be different.
You will have to find your Jenkins admin password by running a command on your Jenkins container which will give you the initial password
Once you login, you will be prompted to install plugins. You can pick and choose plugins or install the recommended plugins.
All plugins will be stored in
/var/jenkins_home/plugins within your Flocker volume that’s mounted at
Once that is complete you have now added some data to your Jenkins home directory.
What we haven’t shown yet is configuring Jenkins, but after you configure Jenkins, all of this data is stored here in
Needless to say, after you put in the effort to install and configure Jenkins, you want it to be fairly robust in the case of failure because you do not want to lose all of the configuration.
We can also verify that our Flocker volume is mounted so all this data is safely persisted to our EBS Flocker volume.
In this case, we also want our Jenkins master to easily be able to recover on the existing cluster we have for our containers.
You may have noticed earlier that our Jenkins master was deployed to
10.0.0.2 so now were are going to show you that by using Flocker and Jenkins together, the Jenkins master data is safe and can handle failure.
If you had scheduled the Jenkins master with
-e reschedule:on-node-failure and are using Swarm, you could halt the node running Jenkins to cause a movement. If you are not, you can instruct docker-compose to
rm -f the Jenkins build and let it start on a new host.
Now, you should not be able to access Jenkins.
Now let’s instruct our Jenkins master to come back up, but first let’s add a scheduling filter to tell it to make sure and not deploy to the server it was on so we can show moving around the Jenkins master won’t matter when we orchestrate the data underneath it.
You can simply add this snippet to your service configuration.
Now start your Jenkins master back up.
We can also see it’s on a new node.
You should be able to login and pick up right where you left off!
You can also do a few other things with Flocker to help you with your Jenkins master deployment.
Create separate volumes for different parts of your Jenkins configuration, this allows you to re-use portions of your Jenkins configuration or store data separately on different tiers of storage.
In the case above, all my plugins will live on their own volume. And I could provision a special volume that’s only 5G but at a
GOLD tier by first creating it before bringing up my cluster.
Note: you would also have to declare the Flocker volume as
externalif created before hand.
I could then potentially create a snapshot, or re-use such volumes in new Jenkins masters and this would allow me to import existing data into a new master. We won’t go into detail on this but stay tuned for more posts in the future.
Having the plugins “pre-installed” in this directory actually increases plugin installation times. In the below example I ran plugin installations for all 56 plugins when you first boot a Jenkins master. Using a Flocker volume that had plugins pre-loaded, I could decrease plugin installation times 4X. While this isn’t the most powerful example, the idea of re-using or keeping configuration state can be in big deployments.
Another example is to use Flocker to help with Jenkins Backups Plugin. In this case, we attach a seperate “backup” volume that we can use to store backup of our Jenkins home.
Here is a compose file with a separate backup volume at
After you run this Jenkins master, you can configure the backup manager with your Flocker volume.
Then, if you want to you can restore your Jenkins masters by attaching the flocker volume to other Jenkins masters, or recover your Jenkins master by expanding a backup and using it at your Jenkins home.
Here is an example of recovering your Jenkins master by expanding a backup in a Flocker volume and using it at your Jenkins home.
After you take a backup, you may have a tar file that looks like this in your Flocker volume.
Then, you can use the
jenkins_backups folder as your
$JENKINS_HOME, see below.
Then bring your Jenkins server up and you can login like normal from a point in time backup from a Flocker backup volume.
In conclusion, Jenkins stores a lot of data, from jobs, system configuration, to build output and artifacts. It all can be persisted and saved so you can manage a robust, fungible master.
Not only can the above persistence techniques by applied to Jenkins masters, but also to Jenkins slaves and other CI/CD tools too.
Stay tuned for more on running your CI/CD infrastructure with container-based data management tools from ClusterHQ.