Introducing Mesos-Flocker: Seamless Software-defined Storage for Mesos

Luke Marsden
Posted by Luke Marsden
December 16, 2015 / Comments

Mesos Flocker

Containers have exploded in popularity but most of the focus has been on running “stateless” apps in containers. Stateful apps, like databases, have been left out of the containerization craze. But modern applications like web apps nearly always have state somewhere. To fully take advantage of the container revolution, you need something to manage your stateless and stateful services running in containers. Today we’ll show you how to do that via Apache Mesos and Flocker using the brand new Mesos Flocker plugin we’ve just released.

For the past two weeks we’ve been working with Container Solutions on a proof of concept (PoC) to bring Flocker to Apache Mesos. This blog post will show how to deploy and manage a typical web application and backend using Mesos, while using Flocker to manage its data. This means that as Mesos reschedules a database container around a cluster of servers, Flocker will make sure that the database container always has its data. That means you’re nearly all the way there towards providing HA for your app’s stateful and stateless componants.

You can download Mesos Flocker and start playing around today.

What is Flocker?

For the benefit of anyone that is reading about Flocker for the first time, Flocker is an open-source data volume manager which is able to provision storage through software. By providing tools for data migrations, Flocker gives ops teams the tools they need to run containerized stateful services like databases in production. Unlike a Docker data volume which is tied to a single server, a Flocker data volume, called a dataset, is portable and can be used with any container in your cluster.

Flocker Architecture

Flocker has a simple REST API which is used to instruct the Flocker Control Service, the brain of your Flocker cluster. The Flocker Control Service communicates with Flocker Agents running on each node in the cluster to carry out commands. Flocker can be controlled via popluar container management tools like Mesos, Marathon, Kubernetes and Docker Swarm.

What is Mesos and how is Flocker integrated?

Apache Mesos is an abstraction layer for computing resources. Applications request to receive a certain amount of resources and Mesos is responsible for providing those resources. The applications do not know, or care, where those resources are located. When paired with an orchestration tool like Kubernetes or Marathon, applications become infinitely scalable to cope with many modern computing requirements. For example, if a piece of hardware dies on a legacy system, it takes staff an inordinate amount of time to restore functionality and recover lost data. With Mesos and an orchestration tool, failure is a normal part of an application’s lifecycle. Applications are simply restarted on another machine and functionality is restored almost instantly.

However, until now, dynamic integration with a reliable, redundant back-end storage mechanism was difficult or impossible to use with Mesos. For example, if an application relied heavily on AWS EBS volumes, it would be a significant task to move to a different provider or location. You’d also have to write a lot of code to make sure your volumes were remounted on your new Mesos hosts when an application was rescheduled.

Using an abstraction layer like Flocker, rescheduling stateful applications is as simple as changing a configuration file. This PoC installs a module to Mesos, known technically as an isolator, which intercepts instantiation calls made to applications and mounts volumes provided by Flocker. The isolator communicates with the Flocker Master to manage datasets without any interaction from the user or the user’s application.

Demonstration of the Mesos-Flocker Plugin

To show the promise of Flocker on Mesos, below is a video demonstration of a very simple, but typical application. The goal is to create a web-app that allows users to click on a web page, where the locations of the clicks are saved to a file on the disk. But unbeknown to the application, this is backed by the Mesos-Flocker plugin, which means that the data is actually transparently written to an EBS volume on AWS. When we forcibly kill the application and it restarts on another slave, the data is reattached and the application can continue like nothing had happened. Magic!

This is the just the beginning for the Flocker Mesos plugin and we’d love your feedback below or over on Hacker News.

Feedback

We’d love to hear your feedback!

About Flocker

Flocker is an open-source container data volume manager for your Dockerized application. It gives ops teams the tools they need to run containerized stateful services like databases in production.
Vote on Hacker News