Migrating stateful containers using native Docker 1.8 Flocker plugin and Compose

Note: This post mentions an outdated version of the Flocker Plugin for Docker. This plugin is now officially supported and can be found on our documentation.

Docker 1.8 includes stable volume plugin support

Today Docker are releasing Docker 1.8 and we’re thrilled that volume plugins are now a first-class feature in Docker 1.8, no longer just experimental. Here are some key points:

  • Swarm and Compose come along for the ride

    Volume plugin support is now present in Docker Swarm 0.4 and Docker Compose 1.4, meaning that the entire stack now works seamlessly together with Flocker.

  • Flocker Docker Plugin 0.4 now available

    Today we are also releasing the Flocker Docker Plugin 0.4 from ClusterHQ Labs. This release fixes several bugs, and introduces support for the new plugin socket path in Docker 1.8.

    You can install or upgrade to version 0.4 of the Flocker Docker Plugin (or Flongle, as we like to call it) by following the instructions in the installation docs.

    We are also hard at work integrating the Flocker Docker plugin directly into Flocker itself.

  • Docker have been great to work with

    As Luke said last month, it has been a pleasure and a privilege to work with the Docker team and it’s exciting to see this project come to fruition.

    We couldn’t agree more :)

  • Please try it out and let us know how you get on!

    We love feedback. Try out the tutorial below and let us know how you get on via #clusterhq on Freenode.

Using Flocker with Docker 1.8 and Compose

The rest of this blog post gives a concrete example of how to use Flocker with Docker 1.8 and Docker Compose 1.4.

The problem

You have just created an amazing new web application and are ready to show to world! Because it is a small project with a limited budget, you decided to deploy it onto some modest servers and gather feedback before committing resources to more powerful servers.

However, the submission to Hacker News went much better than expected and your new application is at the top of the front page! This has created a surge in traffic and your modest little server just does not have the power to cope with the demands your database is putting onto it.

You will need to migrate your application, database and data to a server with:

  • more memory
  • a faster CPU
  • SSD drives

Because we are using Docker together with Docker Compose - it means we can very quickly spin up the web application and database containers onto another more powerful machine.

However - we also need to move the data otherwise disaster will strike and our data will get left behind on the old host.

The solution

Using Flocker and the plugin for Docker we are able to migrate the containers and the data using nothing other than docker-compose commands.

This demonstrates how the promise batteries included but removable has become a reality and – in the case of volume drivers – have made it out of the experimental release and into mainstream Docker.

We will use the Flocker Plugin together with the new --volume-driver flag in Docker 1.8 and Compose 1.4 to migrate the database container and data as a single, atomic unit.

Demo

First we will demonstrate some basic Docker CLI commands that make use of --volume-driver so we can see how this great new feature of Docker works.

Then, we will use Docker Compose to spin up our application on the first node, create some data and then move it to the second node and witness Flocker migrating the data alongside the database container - all using nothing but docker-compose.

screen shot

figure 1. An overview of how Docker and the Flocker plugin interact

Install

First you need to install:

We’ll use Virtualbox to supply the virtual machines that our cluster will run on.

We’ll use Vagrant to simulate our application stack locally. You could also run this demo on AWS or Rackspace with minimal modifications.

Step 1: Start VMs

The first step is to clone this repo and start the VMs:

$ git clone https://github.com/clusterhq/volume-plugins-demo
$ cd volume-plugins-demo
$ vagrant up

Step 2: Check Docker version

We are then going to confirm that we are actually running Docker 1.8 and that it is the non-experimental version, it should look something like this:

$ vagrant ssh node1
vagrant@node1:~$ docker version

Client:
 Version:      1.8.0
 API version:  1.21
 Go version:   go1.4.2
 Git commit:   1027247
 Built:        Mon Aug  3 18:04:07 UTC 2015
 OS/Arch:      linux/amd64

Server:
 Version:      1.8.0
 API version:  1.21
 Go version:   go1.4.2
 Git commit:   1027247
 Built:        Mon Aug  3 18:04:07 UTC 2015
 OS/Arch:      linux/amd64

vagrant@node1:~$ exit

Step 3: Write some data to node1

Now we use the --volume-driver flocker flag to write some data to a Flocker volume:

$ vagrant ssh node1
vagrant@node1:~$ docker run --rm \
    --volume-driver flocker \
    -v simple:/data \
    busybox sh -c "echo hello > /data/file.txt"
vagrant@node1:~$ exit

Step 4: Read the data from node2

Now lets try to read the same data but from a totally different server! Flocker will migrate the data in place before the container is run:

$ vagrant ssh node2
vagrant@node2:~$ docker run --rm \
    --volume-driver flocker \
    -v simple:/data \
    busybox sh -c "cat /data/file.txt"
hello
vagrant@node2:~$ exit

Step 5: Run the application on node1

Next we will use Docker Compose to spin up our web application on the first node:

$ vagrant ssh node1
vagrant@node1:~$ cd /vagrant/app
vagrant@node1:~$ docker-compose up -d
vagrant@node1:~$ exit

Step 6: Load the application in a browser

Now we can fire up a web-browser and visit:

http://172.16.78.250

You should see a blank page - try clicking around to create some Moby Docks!

screen shot

figure 2. A screenshot of the application running in a browser

Step 7: Stop the application on node1

Next lets stop the application running on node1:

$ vagrant ssh node1
vagrant@node1:~$ cd /vagrant/app
vagrant@node1:~$ docker-compose stop
vagrant@node1:~$ exit

Step 8: Run the application on node2

Now the cool part - lets SSH to node2 and use docker-compose just like we did on node1. The difference is this time - Flocker will migrate the data we created on node1 alongside the database container:

$ vagrant ssh node2
vagrant@node2:~$ cd /vagrant/app
vagrant@node2:~$ docker-compose up -d
vagrant@node2:~$ exit

Step 9: Load the application in a browser

Now we can fire up a web-browser and visit:

http://172.16.78.251

This time - we should see the same data we created on node1. This means that docker-compose has given docker the --volume-driver flocker argument and Flocker has kicked in behind the scenes to migrate the data onto node2!

NOTE: The URL is different this time because we have moved the application to node2. I have kept DNS and load-balancing out of this demo to keep it as simple and to the point as possible.

Step 10: Confirm the application is running on node2

As a final confirmation that our application is migrated we can ask Docker to list the containers it is running on node2:

$ vagrant ssh node2
vagrant@node2:~$ docker ps

CONTAINER ID        IMAGE                            COMMAND                  CREATED             STATUS              PORTS                NAMES
38e1920d3994        binocarlos/moby-counter:latest   "node index.js"          25 seconds ago      Up 25 seconds       0.0.0.0:80->80/tcp   app_web_1
2a84d6a262ff        redis:latest                     "/entrypoint.sh redis"   31 seconds ago      Up 25 seconds       6379/tcp             app_redis_1

vagrant@node2:~$ exit

Conclusion

Using this very basic demo, we were able to show that the plugin mechanism in Docker 1.8 is able to integrate with both Flocker and Docker Compose allowing us to migrate a stateful web application from one server to another.

You can read more about Flocker at the ClusterHQ website where you can try a free online demo environment with real VMs so you can play with Flocker with no set-up!

Like what you read?

Signup for a free FlockerHub account.
Sign Up

Get all the ClusterHQ News

Stay up to date with our newsletter. No spam. Ever.