What is FlockerHub?

FlockerHub is like GitHub® for data. Just like you can use GitHub to store, share and organize your code, you can use FlockerHub to store, share and organize your Docker data volumes.

Designed to work with Fli, a git-like CLI for snapshotting, pushing and pulling volumes, FlockerHub lets you keep track of and distribute your volumes to any host, all with access controls that allow you to fulfill your data governance responsibilities.

FlockerHub is currently available as a hosted service. Contact us if you’d like to run FlockerHub privately in your own cloud or data center.

Try flockerhub for free
FlockerHub Volumesets overview screen
FlockerHub Volumesets overview screen
Fli client pushes data to FlockerHub<
Fli client pushes data to FlockerHub

Github for data

Much more than just static volume storage, FlockerHub is a full version-control repository for your data, the way GitHub is a version control repository for your code. You can keep track of every Docker data volume in your cloud or datacenter as a versioned series of snapshots. If your CI environment needs last night’s data backup, you need last week’s data to debug a pesky error report, or Ops needs data from three months ago to provide to the audit committee, FlockerHub can provide each individual a fully independent read/writeable copy of the data, while preserving the original for posterity.

These snapshots can be organized into any number of volume sets which are like repositories in GitHub. Each volume set can include any number of branches, which like GitHub, provide the version-controlled history of the object, in this case a Docker volume.

Fine-grained access controls = Fast and nimble, yet safe

Containers have sped up the build > test > run workflow that DevOps teams all over the world are using to ship higher quality code faster.

But because of the special stewardship requirements that surround data, databases and other data artifacts have largely been on the sideline of the container revolution.

With FlockerHub, you can make your Docker data volumes available to only those users and machines that you authorize. This means data can be where it needs to be, when it needs to be, eliminating the need for slow, manual data-provisioning processes. When teams have the data they need when they need it, without the wait, they can build and deploy better tested software, faster.

FlockerHub user management screen
FlockerHub user management

FlockerHub & Fli in action

Just as it would be impossible to describe everything you can do with git in a single use case, it is impossible to show everything you can build using FlockerHub & Fli in a single page. Let’s just scratch the surface with Tiffany, a DevOps engineer building a microservice using Docker.

Tiffany working on a new feature to increase sales on an ecommerce site

New feature, new microservice

Tiffany is working on a new feature to increase sales on an e-commerce site, a 'You might also like…' widget to be displayed on every product page. Tiffany needs to build, test and deploy the microservice end-to-end and provide a version of the microservice to her teammates so they can test their new features against it.

ecommerce environment on her laptop using a Docker Compose file

Building locally using Docker Compose

Tiffany starts by spinning up the e-commerce environment on her laptop, using a Docker Compose file she got from her manager. She’ll test her feature first locally using this environment.

Create a new branch and writes an initial implementation

Starting her feature branch

Now that she’s got her development environment set up, she creates a new branch and writes an initial implementation of the feature.

She can’t see if the logic is correct

Testing locally is incomplete without data

Tiffany’s got the feature more or less how she wants it, and she wants to test it locally on her machine. She can run her branch against the local environment spun up with Docker Compose, but there is no data in the Order History and Product Catalogue databases that the widget needs, so she can’t see if the logic is correct.

She can’t see if the logic is correct

Realistic data comes from production

Tiffany could put together some test data, but she would almost certainly miss edge cases that she’d find testing against realistic data. Tiffany asks Paul, her colleague in Ops, to send her the anonymized last 5 million entries from the Order History MongoDB database and the entire Product Catalog, all 1.5 million products, PostgreSQL database.

She can’t see if the logic is correct

Snapshotting volumes with Fli

After receiving the data, Tiffany snapshots each database using Fli and pushes these volumes to the FlockerHub. That way, not only can she use this test data, but any of her colleagues or staging/QA environments can too.

There is a wrapping issue

Test data reveals a bug (Shocking!)

With her realistic test data available locally, Tiffany previews her branch. Looks good, except for that wrapping issue on the long product name. Who knew there was something called The Really, Incredibly, Stupendously, Fantastically Good Stain Remover®?

Tiffany fixes the CSS for displaying long names and tries again

A little tweaking and voila...

Tiffany fixes the CSS for displaying long names and tries again. Bingo. Looks good.

She can’t see if the logic is correct

And now for CI

Now that things are looking good locally, Tiffany needs to run her code through her company’s Jenkins environment. Their CI system kicks off for each commit in GitHub, so before pushing her branch, Tiffany makes sure that her Order History MongoDB and Product Catalog PostgreSQL that she pushed to FlockerHub are pulled into Jenkins /home/test-fixtures on all the Jenkins build slaves. This way, no matter which slave her build executes on, it will run her tests against the realistic data.

She can’t see if the logic is correct

When a build fails, capture the state

Tiffany runs her test but the build fails when her integration test notices that one of the three product suggestions fails to load. At first glance, she is not quite sure what happened but can figure it out. Her team sets up Jenkins to snapshot all test databases using Fli as soon as a build fails, and then pushes these snapshots to FlockerHub. This way, Tiffany or any of her colleagues can have a completely consistent environment— code, container images and state–in which to debug.

Mongo and Postgres data volumes from the Jenkins environment only take a few seconds

Debugging locally with actual data

Because Fli uses incremental snapshots, the snapshot and push and subsequent pulls of the Mongo and Postgres data volumes from the Jenkins environment only take a few seconds, so in no time, Tiffany has an identical application environment running locally.

She can’t see if the logic is correct

Sharing a development environment, including data, with a colleague

Tiffany digs around a bit and can’t recreate the bug. She decides to ask her colleague Sam to take a look. She sends Sam a Docker Compose file, which references her data volume on FlockerHub. Sam pulls all the Docker images and data volumes and is running an identical environment in minutes.

She can’t see if the logic is correct

Armed with data, Sam bags the bug

Sam steps through the integration test and notices the bug only appears when there are exactly 4,294,967,296 orders in the Order History database. The code works fine up to 4,294,967,295 orders, then something goes horribly awry. But Sam’s brain always has a background thread calculating base 2 logarithms, so the problem is elementary. She knows exactly where to tweak the code. When she reruns the test, it displays fine.

She can’t see if the logic is correct

Build passes, time for the next feature!

Sam commits her changes, the Jenkins build runs, and the test passes, even though it runs against the exact same database state as before, with 4,294,967,296 orders in the Order History database. High fives all around.

FlockerHub + fli mean flexibility & control for dev & ops

For Developers

  • Makes testing stateful microservices easy
  • Less time waiting for test environment provisioning
  • Collaborate on databases across teams and organizations
  • Programmatically run tests
  • Run multiple database clones on the same machine without increasing storage costs
  • Rollback your database to any point-in-time

For Operations

  • Fine-grained permissions give you control
  • Easy snapshots of any Linux database from the command line
  • Incremental snapshots mean faster data transfers
  • No more one-offs. Easily publish snapshots for your teams to consume
  • Close the loop with your dev team. Work seamlessly on stateful applications

Learn more about FlockerHub and Fli

Push and pull data snapshots

The basics of how to move data with Fli and FlockerHub.

Creating volumes

Learn how to create a data volume using Fli, the FlockerHub CLI.

Updating object metadata

Learn how tag your data snapshots with user-defined attributes so they are easy to find and manage.

How to setup Fli with your ZPool

Learn how to set up a local storage pool to store and take snapshots.

Sign Up Free!

Sign up for the FlockerHub beta and receive 5GB of container volume storage free for 6 months.

Sign up now

Frequently asked questions

To use FlockerHub, you need to install the Fli client. Fli works on any computer that runs Ubuntu 16.04, RHEL 7.2, CentOS 7.2 or Fedora 24. You will also need root access to the server running Fli to configure ZFS, the filesystem that Fli users to take and manage snapshots.

FlockerHub complements the DockerHub and DTR (Docker Trusted Registry) and you should continue to use DockerHub for your container image storage.

FlockerHub gives you a central place to store your version-controlled Docker data volumes. For example, you can use Docker Compose to deploy a containerized application, including its data. Your containers will come from the DockerHub while the data will come from FlockerHub. FlockerHub also works with any other public or private container image repository.

FlockerHub also complements Github or any other version-control system you are using like Bitbucket or Gitlab. FlockerHub provides a version-controlled view of your application data. When you combine this with the code version-control provided by Github and container-image version control provided by DockerHub, you have everything you need to spin up a fully consistent application environment anywhere. This makes debugging a snap, in addition to being a great way to deploy new applications.

Currently you can't push and pull your Flocker volumes to FlockerHub. You also can't pull a volume from FlockerHub and manage it with Flocker. But we're working on this and expect to intergrate Flocker volume orchestrator into FlockerHub soon.

Got a question?

Thanks for your email, we'll be in touch shortly