In this blog post we will recap the recent FlockerHub and Fli announcements, and present a deeper look at how these tools work. If you haven’t heard about FlockerHub and Fli yet, read all about it here.
In short, by letting DevOps teams share data volumes, certain processes within the DevOps lifecycle become more efficient. FlockerHub and Fli can improve testing and catch more bugs before they make it into production. A recent study we released the same day as FlockerHub showed that 43% of application developers spend between 10-25% of their time debugging errors discovered in production, rather than developing new features. We think that’s crazy, and we think FlockerHub and Fli can help. Read on for the details and get started using FlockerHub today.
Fli, the FlockerHub CLI, is like Git for data. Just like you can use Git to copy,
pull code, you can use Fli to copy,
pull your data volumes.
Fli lets you take incremental snapshots of any database or data volume that runs on Linux and push those snapshots to a hosted volume repository called FlockerHub. There they can be accessed by any person or machine to whom you’ve granted access. Fli can also be used without FlockerHub to manage data volumes locally.
One thing that’s really great about Fli is that it is open source! You check out the code, README and getting started for developers at the ClusterHQ/fli GitHub® page.
You can find documentation at fli-docs-clusterhq.com.
FlockerHub is a Hub for data, like GitHub is a Hub for source code and DockerHub is a Hub for container images. Specifically, FlockerHub lets you store your Docker data volumes and organize them into volumesets and branches that are owned by organizations, teams and individual users.
FlockerHub also provides a user interface for organizing and controlling access to snapshots. In the current implementation, FlockerHub is provided as a public SaaS offering, which you can start using here. However, in future releases, we plan to enable creation of private FlockerHub’s in a VPC or on-premise environments.
You can find documentation at flockerhub-docs-clusterhq.com.
Fli and FlockerHub Concepts
In this section, we will run through the different concepts and abstractions that underlie Fli so you can get the most out of these tools for managing your volumesets, snapshots, branches, organizations, teams and users.
In order to start using FlockerHub to house snapshots, a user must get an Authentication Token to validate them as a genuine user of FlockerHub.
To do this, navigate to FlockerHub and log in using Google or GitHub Connect. Then, go to the Settings tab at the top of the screen.
You will be brought to a screen that lets you create authentication tokens. To do this, add a name and description for your token and click Create Token.
volumeset can belong to one or more users that may be part of a team or organization.
Think of a
volumeset like a repository on GitHub, except volumesets don’t contain source code, they contain volume snapshots.
Snapshots can be organized into branches which help teams and users relate specific trees of snapshots they might be working on.
Volumesets can contain meta-data in the form of attributes and descriptions that allow
volumeset owners to give extra context to users who will be consuming them.
Volumesets can be synced to and from FlockerHub so that your local changes are visible to other users. When you
sync, you are not pushing any data, only meta-data about the
Volumesets can also be used with the
push command in case you might want all snapshots in your
volumeset to be pushed to FlockerHub.
volumeset with Fli is simple.
Volumesets will show up in the FlockerHub web interface in the
volumeset tab at the top.
Note, volumesets or other changes will only show up after a
A volume, as it relates to FlockerHub and Fli, is a working mount-point. We don’t store “working” volumes on FlockerHub, but rather snapshots of those volumes. You can, however, turn any snapshot into a working volume.
Think of a volume as a directory that you can work with like any other directory, except this volume can be snapshotted and that snapshot can be pushed and pulled to and from FlockerHub.
When you create a volume with Fli, it looks like this.
You can move into that directory, create files, or anything you would normally do on POSIX file-systems.
A snapshot is a “picture” of a volume that can’t be mounted itself to a host, but it can be instantiated as a data volume that can be mounted to a host through the
To create a snapshot from an existing volume, you will use the snapshot command.
A clone is a copy of a snapshot that is made available as a volume. Cloning a volume is used when you want to instantiate a volume based on a particular snapshot.
This allows you to create a volume with pre-existing point-in-time versions of your data.
To create a clone from a snapshot you may run the following clone command.
branch is a lineage of a particular data volume made up of multiple snapshots over time.
Think of a
branch like a GitHub
branch, but made of snapshots instead of commits.
One way a Fli branch is not similar to Git is that volume branches do not necessarily come from a single trunk, nor do they have the ability to merge back in; e.g. “Production data” might be one
branch; “staging data” another
branch; and “local data” a third
Branches are created by using the
--branch | -b flag during a snapshot command.
Branches are visible in the FlockerHub Web Interface if you click on a particular
volumeset. Clicking on a
branch shows you which snapshots belong to that
FlockerHub volumesets can be shared among users.
By allowing organizations and teams to share snapshot resources, they can easily enable better testing during CI/CD, development, QA and debugging stages of application development. They can then share necessary storage resources to test and find bugs faster and more efficiently in the DevOps life cycle.
To share a
volumeset, navigate to the Volumesets page. Then, click on the check box to the left of a
volumeset that you want to share. You should see the share button light up red with the number of volumesets you are trying to share.
Next, click the share button and a new window will pop up asking you to enter the email addresses of the users you would like to share the
volumeset with, along with a message about what is being shared.
To set the permissions when sharing to users, you must be the Owner or Admin of a
volumeset. Choose the permission setting from the drop down menu in the sharing windows as shown below.
Once you have entered the email addresses and message you would like, click share and a notification should tell you that you have successfully shared the
Emails will be sent to the users with which you wish to share your
volumeset. If the recipients are existing FlockerHub users, they will see a
Collaborate button in their email to start using the
If the recipients are not existing FlockerHub users, then they will get an email with a
Create Account button.
Fli and FlockerHub Architecture
FlockerHub is made up of a few main parts, and this next section will give a high-level overview of the components make up a FlockerHub instance.
This will help you get a better understanding of the moving parts when you are working with Fli and FlockerHub.
Note, as a user of Fli and FlockerHub, you do not need to get caught up in the details of how FlockerHub components work, it is merely a reference for those interested in what’s behind the covers of FlockerHub.
Many data servers make up the the highly scalable back-end of a FlockerHub deployment. It is a Golang server whose main purpose is to store user snapshots in an intelligent, safe and secure manner. You can think of the data server as the place where your data rests after you
push it up to FlockerHub. As a FlockerHub user, you don’t need to worry about running the data servers. We do this for you.
Data plane servers are another scalable component that handle requests from the Fli client to and from the data servers, browser and authentication services. Data plane servers have the important job of brokering requests for information about users and data with the responsibility of doing this safely and securely. You also don’t need to worry about the data plane. We run this for you, too.
FlockerHub Web Interface
This is where you log in and view, organize and manage volumesets, branches, snapshots, tokens and more.
The user interface will talk to different components of the system, such as the data plane, to gather information about users and snapshots that the user should have access to view via the UI.
Authentication and Access Control
The authentication and access control mechanisms for FlockerHub and Fli are made up of multiple components.
One of the components is responsible for managing authentication tokens and who they belong to. This ensures that only authorized users can access the system.
Another component is responsible for managing users, organizations and permissions settings that relate to the volumesets, branches and snapshots so that access controls to such resources is maintained across Fli and FlockerHub.
A final component is responsible for talking with OAuth2 so that users can be authenticated via their existing GitHub and Google accounts.
Fli Client and FlockerHub Interactions
Fli is the CLI component, and is separate from the FlockerHub deployment. Fli is installed on client machines that wish to
pull and create snapshots.
It must be configured with an authentication token created in FlockerHub so that the the Fli client has the correct access to
sync the volumesets, branches and snapshots it has access to.
Hopefully this has been useful reading that has given you a better understanding of the components and interactions that make up Fli and FlockerHub!
We will continue to post new blogs that dive deeper into some of these components and use cases for Fli and FlockerHub. So, stay tuned!