Git for data is here: Announcing FlockerHub and Fli

FlockerHub and Fli logo

We are very excited to announce the upcoming release on November 8th of two major new products designed to make running stateful apps in containers easy–FlockerHub and Fli (Update- we’ve release FlockerHub and Fli and you can signup today). FlockerHub will be released in beta on November 8th. During the beta, users can store up to 5GB of data volumes and share with an unlimited number of other users for free. Fli will be released as an Apache 2.0 licensed software project.

FlockerHub is like GitHub® for data. With FlockerHub, teams can store and share any Docker data volume with access-controlled users or servers.

Fli is like Git for data. It is command line interface that runs on any Linux server or laptop, and lets developers snapshot, clone, push and pull data volumes to FlockerHub.

With FlockerHub and Fli, DevOps teams can seamlessly move data between laptops, test environments, data centers and clouds, all with version and access controls. This flexibility to have data where you need it, when you need it, no matter the underlying infrastructure, has an immediate and positive impact on DevOps teams doing continuous integration, continuous delivery and automated testing. In addition, FlockerHub and Fli in conjunction with Flocker have the potential to reinvent how disaster recovery, backup and cross-cloud workload migration are done in the enterprise.

Software development before Git (oh, the horror!)

So if FlockerHub and Fli are Git-like, let’s talk a little about Git. It is pretty safe to say that Git has changed the way teams build software for the (much) better. (Quick aside, in reality, it is not just Git. This goes for other source version control systems that are built on a hub/spoke model where there is a central repository that maintains arbitrarily complex trees of code versions–and metadata about the code versions, provide an ability for individual software engineers to pull down any (sub)tree of interest, work on a local copy of the code, make their own arbitrarily complex local code trees, and then share selected code changes with peers, QA, or automated testing systems with an easy and scriptable workflow).

Before Git, developers largely worked in isolation. You worked on your own version of code. Your teammates did too, whether they sat next to you or were half the planet away. Collaborating on complex code-related problems was difficult, unnatural, and at best a royal pain in the neck. Managing code versions in QA was manual and error prone. Continuous test and integration, let alone deployment, weren’t even fantasies.

Because testing was inherently hampered, you wrote very few test cases, so software was ill-tested before being pushed to production. And those production pushes were ever-traumatic. Code release cycles were measured in years or quarters. Rapid innovation – get a bug report or have a feature idea, quickly code it, test it, iterate, iterate, iterate – was impossible. Microservices? DevOps? Inconceivable.

Today, working with data suffers from many of the same problems. Data is hard to move around.
Getting access to data often requires manual processes that take way too much time and as a result cost way too much money.

Just like Git streamlined software development so that eventually 1000 microservices could bloom, we think that FlockerHub and Fli will change the way software teams interact with data, opening up a world of new possibilities.

What exactly is Git for data?

Git and GitHub are about making it easier to work as a team on code. FlockerHub and Fli aim to do the same thing for data.

FlockerHub is like GitHub® for data. Just like you can use GitHub to store, share and organize your code, you can use FlockerHub to store, share and organize your Docker data volumes.

FlockerHub screenshot

Fli, the FlockerHub CLI, is like Git for data. Just like you can use Git to copy, branch, push and pull code, you can use Fli to copy, branch, push and pull your data volumes.


So what can you do with FlockerHub and Fli?

Here are a few ideas to help you get started.

Realistic test data for CI environments

The best tests include realistic test data. But getting data into continuous integration environments can be cumbersome since it usually involves asking operations teams for access. Not only does this manual request take time, slowing down testing, but it is also expensive because multiple copies of the data are needed for parallel testing.

With Fli, Ops teams can automatically snapshot any data volume and send it to FlockerHub, where it can be shared with only access-controlled users or servers. Additionally, Fli can clone the data multiple times without increasing storage consumption, so multiple tests can run in parallel without additional cost. When combined with popular CI software like Jenkins, the data management capabilities provided by FlockerHub and Fli can accelerate incremental and parallel builds, speeding up testing dramatically.

On-demand staging for each commit

As developers work on feature branches, they often preview their code in a staging environment. Docker Compose is an easy way to quickly spin up test environments for each commit, but without realistic test data, bugs can be easily missed, increasing the likelihood of errors in production.

Through an integration with Docker Compose, FlockerHub volumes can be referenced directly in the Docker Compose YAML file, allowing the Dockerized application to be spun up along with its data, providing a truly representative staging environment.

Commit and share error state for simplified debugging

The recent Application Testing Survey conducted by ClusterHQ uncovered that 11 percent of application developers report encountering bugs in production every single day, while an additional 25 percent encounter new bugs multiple times per week. Typically when a bug is reported, a developer wastes several hours trying to reproduce the problem before it can be fixed.

With Fli, database state can be regularly snapshotted and pushed to FlockerHub, so that any time an application needs to be debugged, the accompanying data can be included in the debugging environment. Any developer who needs to work on an issue can pull the error state, along with the correct code version and Dockerized application environment, and have everything needed to isolate the bug.

Stay tuned for more

We are very excited to be releasing these projects on November 8th. Checkout some of the resources below to learn more about how FlockerHub and Fli work and stay turned for more details in the coming days as we share more use cases.

Technical Details

How FlockerHub works

How Fli works


Fli- Taking Snapshots

FlockerHub- Updating Object Metadata

FlockerHub- Creating Volumes

Push and Pull Data Snapshots with Fli and FlockerHub

How to setup Fli with your ZPool

Using fli-docker to bootstrap Docker Compose Apps with data

Use Cases

Better CI: Integrate realistic data, accelerate your pipeline

On-demand staging: Include realistic test data for each commit

Test better: Easily create a scalable, distributed test fixture library

Share error state: Collaboratively debug the most challenging errors

On-demand dev environments: Share your workspace with anyone

Like what you read?

Signup for a free FlockerHub account.
Sign Up

Get all the ClusterHQ News

Stay up to date with our newsletter. No spam. Ever.