Running your Jenkins infrastructure with ClusterHQ: Part 1

Jenkins_Part1Img

This is a series that we will dedicate to the CI/CD pipeline and how you can use tools from ClusterHQ to help gain fungibility, efficiency and mobility to your CI/CD infrastructure and build pipelines.

The series will start with a focus on running Jenkins and how to provide flexibility and mobility running your master and its associated configuration. Then, the series will slowly move into other topics such as limiting non-deterministic failures, fungibility for slaves, greater efficiency for popular plug-ins and overall a more deterministic and repeatable Jenkins infrastructure.

    Introduction

    What is Flocker

    Flocker is a container data volume manager that allows stateful services to run inside containers in production. When running a database in production, it’s important to think about things like recovering from host failure and availability of data. Flocker provides tools for managing data volumes across a cluster of machines like those in a production environment.

    For example, as a MySQL container is scheduled between hosts in response to server failure, Flocker can automatically move its associated data volume between hosts at the same time. This operation can be initiated using the Flocker API, CLI, or a container orchestration framework such as Docker Swarm, Kubernetes and Mesos.

    Why use Flocker with your Jenkins Master

    Running a Jenkins master deployment means downloading and installing various plugins to use for different build purposes and configuring those plugins according to your particular infrastructure environment. On top of that you will have logs, job output, job configuration, update data, artifacts, libraries and more associated with your Jenkins master for your build pipeline.

    All of this data and configuration is stored within a directory /var/jenkins_home. If you run your Jenkins master in a container you then need to consider how you make sure this data is safe and mobile in the cases where the container restarts, gets rescheduled or the host running Jenkins fails.

    You can follow the suggestions on the Jenkins guide in DockerHub but even using the suggestions there, your data is not always safe because it’s not able to recover from more than host-level failure.

    Three main ways to run Jenkins according to DockerHub

    $ docker run -p 8080:8080 -p 50000:50000 jenkins

    In the above case, all of your data resides inside the container’s own file-system storage. This means that your data is most vulnerable to loss because when the container goes away, so does the data.

    $ docker run --name myjenkins -p 8080:8080 -p 50000:50000 -v /var/jenkins_home jenkins

    In this case, your data is placed into a Docker volume. This means that the data is within /var/lib/docker/volumes/ but won’t be safe if you lose your docker host. Also, if you start your Jenkins master on another node it will not move to the new node so it will be like booting a brand new Jenkins master.

    $ docker run -p 8080:8080 -p 50000:50000 -v /your/home:/var/jenkins_home jenkins

    In this last case, you are doing much the same as the previous case but instead of it being a docker volume you’re mounting in a directory that exists on the docker host that is outside of the docker file-system. This approach still leaves data vulnerable to node failure and container movement issues.

    In the next portion of this post, we will explore how to have a more portable and fungible Jenkins master.

    Fungible Jenkins Master

    We will enable our Jenkins master to use an attached storage volume so that not only is your Jenkins home directory always safe, it will always follow your Jenkins master no matter which host it is on in a cluster of VMs.

    We will build on this post from Yoanis Gil Delgado about how to do this and then dig into how to enable some more flexible options for a master deployment.

    In this example I have five EC2 nodes running Docker, Flocker and Swarm. I will use this cluster throughout this demo which will use EBS volumes for storage.

    To start a Jenkins master with a Flocker volume, you can use the below example of a compose file that does this.

    version: "2"
    
    services:
        jenkins-all:
            image: jenkins:2.0
            networks:
                - jenkins
            ports:
                - "8080:8080/tcp"
            volumes:
                 - jenkins:${JENKINS_HOME}
    
    networks:
        jenkins:
           external:
             name: overlay1
    volumes:
        jenkins:
            driver: flocker

    To deploy this setup you can run docker-compose up like so.

    $ export JENKINS_HOME=/var/jenkins_home
    $ docker-compose -f compose-jenkins-master.yml up -d

    What happens now is Flocker will provision any volumes that are not created yet and bring up your Jenkins master. After it completes you will have a running Jenkins Master.

    Note: I am using a fake IP in the ouput, yours would be different.

    $ docker-compose -f compose-jenkins-master.yml ps
     Name   Command      State       Ports
    -----------------------------------------------------------------------
    jenkins_jenkins-master   /bin/tini -- /usr/local/bi ...   Up      50000/tcp, 10.0.0.2:8080->8080/tcp

    You will have to find your Jenkins admin password by running a command on your Jenkins container which will give you the initial password

    $ docker exec <YOUR_JENKINS_CONTAINER_NAME> cat /var/jenkins_home/secrets/initialAdminPassword
    4b081a6756014cbda504256e9fe239e5

    Once you login, you will be prompted to install plugins. You can pick and choose plugins or install the recommended plugins.

    All plugins will be stored in /var/jenkins_home/plugins within your Flocker volume that’s mounted at /var/jenkins_home/.

    Jenkins_Installplugins

    Once that is complete you have now added some data to your Jenkins home directory.

    $ [email protected]:/$ ls /var/jenkins_home/
    
    com.dabsquared.gitlabjenkins.GitLabPushTrigger.xml                  
    jenkins.install.InstallUtil.lastExecVersion  secret.key.not-so-secret
    com.dabsquared.gitlabjenkins.connection.GitLabConnectionConfig.xml  
    jenkins.install.UpgradeWizard.state          
    secrets
    config.xml                                                          
    jobs/                                         
    updates/
    copy_reference_file.log                                             
    logs/                                         
    userContent/
    hudson.model.UpdateCenter.xml                                       
    nodeMonitors.xml                             
    users/
    hudson.plugins.git.GitTool.xml                                      
    nodes/                                        
    war/
    identity.key.enc                                                    
    plugins/                                      
    workflow-libs/
    init.groovy.d                                                       
    secret.key

    What we haven’t shown yet is configuring Jenkins, but after you configure Jenkins, all of this data is stored here in /var/jenkins_home.

    Jenkins_Complete

    Needless to say, after you put in the effort to install and configure Jenkins, you want it to be fairly robust in the case of failure because you do not want to lose all of the configuration.

    We can also verify that our Flocker volume is mounted so all this data is safely persisted to our EBS Flocker volume.

    $ docker inspect jenkins-master
    .
    .
    "Mounts": [
                {
                    "Name": "jenkin_jenkins_home",
                    "Source": "/flocker/40de74ba-3a48-4876-a9e3-a1586476191c",
                    "Destination": "/var/jenkins_home",
                    "Driver": "flocker",
                    "Mode": "rw",
                    "RW": true,
                    "Propagation": "rprivate"
                },
    .
    .

    In this case, we also want our Jenkins master to easily be able to recover on the existing cluster we have for our containers.

    You may have noticed earlier that our Jenkins master was deployed to 10.0.0.2 so now were are going to show you that by using Flocker and Jenkins together, the Jenkins master data is safe and can handle failure.

    If you had scheduled the Jenkins master with -e reschedule:on-node-failure and are using Swarm, you could halt the node running Jenkins to cause a movement. If you are not, you can instruct docker-compose to stop and rm -f the Jenkins build and let it start on a new host.

    $ docker-compose -f compose-jenkins-master.yml stop
    Stopping jenkins_jenkins-master ... done
    
    $ docker-compose -f compose-jenkins-master.yml rm -f
    WARNING: Not including one-off containers created by `docker-compose run`.
    To include them, use `docker-compose rm --all`.
    This will be the default behavior in the next version of Compose.
    
    Going to remove jenkins_jenkins-master
    Removing jenkins_jenkins-master ... done

    Now, you should not be able to access Jenkins.

    Now let’s instruct our Jenkins master to come back up, but first let’s add a scheduling filter to tell it to make sure and not deploy to the server it was on so we can show moving around the Jenkins master won’t matter when we orchestrate the data underneath it.

    You can simply add this snippet to your service configuration.

    environment:
              - constraint:node!=<node jenkins was running on>

    Now start your Jenkins master back up.

    $ docker-compose -f compose-jenkins-master.yml up -d
    Creating jenkins_jenkins-master

    We can also see it’s on a new node.

    $ docker-compose -f compose-jenkins-master.yml ps
     Name   Command      State       Ports
    -----------------------------------------------------------------------
    jenkins_jenkins-master   /bin/tini -- /usr/local/bi ...   Up      50000/tcp, 10.0.0.5:8080->8080/tcp

    You should be able to login and pick up right where you left off!

    Jenkins_Moved

    More Options

    You can also do a few other things with Flocker to help you with your Jenkins master deployment.

    Separate Volumes

    Create separate volumes for different parts of your Jenkins configuration, this allows you to re-use portions of your Jenkins configuration or store data separately on different tiers of storage.

    version: "2"
    
    services:
        jenkins-seperates:
            image: jenkins:2.0
            networks:
                - jenkins
            ports:
                - "8080:8080/tcp"
            volumes:
                 - jenkins_home:${JENKINS_HOME}
                 - jenkins_plugins:${JENKINS_HOME}/plugins/
    
    networks:
        jenkins:
           external:
             name: overlay1
    volumes:
        jenkins_home:
            driver: flocker
        jenkins_plugins:
            driver: flocker

    In the case above, all my plugins will live on their own volume. And I could provision a special volume that’s only 5G but at a GOLD tier by first creating it before bringing up my cluster.

    $ docker volume create -d flocker --name jenkins_jenkins_plugins -o size=5G -o profile=gold`

    Note: you would also have to declare the Flocker volume as external if created before hand.

    I could then potentially create a snapshot, or re-use such volumes in new Jenkins masters and this would allow me to import existing data into a new master. We won’t go into detail on this but stay tuned for more posts in the future.

    Having the plugins “pre-installed” in this directory actually increases plugin installation times. In the below example I ran plugin installations for all 56 plugins when you first boot a Jenkins master. Using a Flocker volume that had plugins pre-loaded, I could decrease plugin installation times 4X. While this isn’t the most powerful example, the idea of re-using or keeping configuration state can be in big deployments.

    Jenkins_ImproveInstalls

    Backups

    Another example is to use Flocker to help with Jenkins Backups Plugin. In this case, we attach a seperate “backup” volume that we can use to store backup of our Jenkins home.

    Here is a compose file with a separate backup volume at /jenkins_backups

    version: "2"
    
    services:
        jenkins-seperates:
            image: jenkins:2.0
            networks:
                - jenkins
            ports:
                - "8080:8080/tcp"
            volumes:
                 - jenkins_home:${JENKINS_HOME}
                 - jenkins_plugins:${JENKINS_HOME}/plugins/
                 - jenkins_backups:/jenkins_backups/
    
    networks:
        jenkins:
           external:
             name: overlay1
    volumes:
        jenkins_home:
            driver: flocker
        jenkins_plugins:
            driver: flocker
        jenkins_backups:
            driver: flocker

    After you run this Jenkins master, you can configure the backup manager with your Flocker volume.

    Jenkins_BackupManager

    Then, if you want to you can restore your Jenkins masters by attaching the flocker volume to other Jenkins masters, or recover your Jenkins master by expanding a backup and using it at your Jenkins home.

    Here is an example of recovering your Jenkins master by expanding a backup in a Flocker volume and using it at your Jenkins home.

    After you take a backup, you may have a tar file that looks like this in your Flocker volume.

    $ docker exec jenkins_jenkins-seperates ls /jenkins_backups/
    backup_20160519_1754.tar.gz

    Expand it.

    $ docker exec -it jenkins_jenkins-seperates /bin/bash
    
    [email protected]:/$ cd /jenkins_backups/
    
    [email protected]:/jenkins_backups$ tar -zxvf backup_20160519_1754.tar.gz
    secret.key.not-so-secret
    hudson.plugins.git.GitTool.xml
    queue.xml.bak
    secret.key
    credentials.xml
    com.dabsquared.gitlabjenkins.connection.GitLabConnectionConfig.xml
    backup.xml
    com.dabsquared.gitlabjenkins.GitLabPushTrigger.xml
    .java/fonts/1.8.0_72-internal/fcinfo-1-6954e13c3999-Linux-3.13.0-85-generic-en.properties
    .java/fonts/1.8.0_72-internal/fcinfo-1-5d32f67da987-Linux-3.13.0-85-generic-en.properties
    .java/fonts/1.8.0_72-internal/fcinfo-1-7af5373be5aa-Linux-3.13.0-85-generic-en.properties
    .java/fonts/1.8.0_72-internal/fcinfo-1-006c46631950-Linux-3.13.0-85-generic-en.properties
    .java/fonts/1.8.0_72-internal/fcinfo-1-b279a2b21629-Linux-3.13.0-85-generic-en.properties
    .java/fonts/1.8.0_72-internal/fcinfo-1-c14960d10290-Linux-3.13.0-85-generic-en.properties
    users/ryan/config.xml
    .bashrc
    secrets/jenkins.security.ApiTokenProperty.seed
    secrets/hudson.util.Secret
    secrets/jenkins.model.Jenkins.crumbSalt
    .
    .
    .

    Then, you can use the jenkins_backups folder as your $JENKINS_HOME, see below.

    version: "2"
    
    services:
        jenkins-seperates:
            image: jenkins:2.0
            environment:
              - constraint:node!=mha-demo1
            networks:
                - jenkins
            ports:
                - "8080:8080/tcp"
            volumes:
                 - jenkins_backups:${JENKINS_HOME}
                 - jenkins_plugins:${JENKINS_HOME}/plugins/
    
    networks:
        jenkins:
           external:
             name: overlay1
    volumes:
        jenkins_plugins:
            driver: flocker
        jenkins_backups:
            driver: flocker

    Then bring your Jenkins server up and you can login like normal from a point in time backup from a Flocker backup volume.

    $ docker-compose -f compose-jenkins-master-seperates.yml up -d

    Jenkins_BackupUp

    Conclusion

    In conclusion, Jenkins stores a lot of data, from jobs, system configuration, to build output and artifacts. It all can be persisted and saved so you can manage a robust, fungible master.

    Not only can the above persistence techniques by applied to Jenkins masters, but also to Jenkins slaves and other CI/CD tools too.

    Stay tuned for more on running your CI/CD infrastructure with container-based data management tools from ClusterHQ.

    Like what you read?

    Signup for a free FlockerHub account.
    Sign Up

    Get all the ClusterHQ News

    Stay up to date with our newsletter. No spam. Ever.