FlockerHub and Fli: A Deeper Look

FlockerHub and Fli logo

In this blog post we will recap the recent FlockerHub and Fli announcements, and present a deeper look at how these tools work. If you haven’t heard about FlockerHub and Fli yet, read all about it here.

In short, by letting DevOps teams share data volumes, certain processes within the DevOps lifecycle become more efficient. FlockerHub and Fli can improve testing and catch more bugs before they make it into production. A recent study we released the same day as FlockerHub showed that 43% of application developers spend between 10-25% of their time debugging errors discovered in production, rather than developing new features. We think that’s crazy, and we think FlockerHub and Fli can help. Read on for the details and get started using FlockerHub today.

    Fli

    Fli, the FlockerHub CLI, is like Git for data. Just like you can use Git to copy, branch, push and pull code, you can use Fli to copy, branch, push and pull your data volumes.

    Fli lets you take incremental snapshots of any database or data volume that runs on Linux and push those snapshots to a hosted volume repository called FlockerHub. There they can be accessed by any person or machine to whom you’ve granted access. Fli can also be used without FlockerHub to manage data volumes locally.

    One thing that’s really great about Fli is that it is open source! You check out the code, README and getting started for developers at the ClusterHQ/fli GitHub® page.

    You can find documentation at fli-docs-clusterhq.com.

    FlockerHub

    FlockerHub is a Hub for data, like GitHub is a Hub for source code and DockerHub is a Hub for container images. Specifically, FlockerHub lets you store your Docker data volumes and organize them into volumesets and branches that are owned by organizations, teams and individual users.

    FlockerHub also provides a user interface for organizing and controlling access to snapshots. In the current implementation, FlockerHub is provided as a public SaaS offering, which you can start using here. However, in future releases, we plan to enable creation of private FlockerHub’s in a VPC or on-premise environments.

    You can find documentation at flockerhub-docs-clusterhq.com.

    Fli and FlockerHub Concepts

    In this section, we will run through the different concepts and abstractions that underlie Fli so you can get the most out of these tools for managing your volumesets, snapshots, branches, organizations, teams and users.

    Log-in

    When you first visit FlockerHub, you will be prompted to log in using either Google or GitHub Connect. If it’s your first time logging in, you will also be prompted to accept the Terms of Use, and optionally run through a quick Getting Started guide.

    FH Login

    Authentication Tokens

    In order to start using FlockerHub to house snapshots, a user must get an Authentication Token to validate them as a genuine user of FlockerHub.

    To do this, navigate to FlockerHub and log in using Google or GitHub Connect. Then, go to the Settings tab at the top of the screen.

    You will be brought to a screen that lets you create authentication tokens. To do this, add a name and description for your token and click Create Token.

    Create Auth Token

    Volumeset

    A volumeset can belong to one or more users that may be part of a team or organization.

    Think of a volumeset like a repository on GitHub, except volumesets don’t contain source code, they contain volume snapshots.

    Snapshots can be organized into branches which help teams and users relate specific trees of snapshots they might be working on.

    Volumesets can contain meta-data in the form of attributes and descriptions that allow volumeset owners to give extra context to users who will be consuming them.

    Volumesets can be synced to and from FlockerHub so that your local changes are visible to other users. When you sync, you are not pushing any data, only meta-data about the volumeset.

    Volumesets can also be used with the push command in case you might want all snapshots in your volumeset to be pushed to FlockerHub.

    Creating a volumeset with Fli is simple.

     $ fli init my-volumeset -a for=blog_post,owner=ryan -d "volumeset for testing"
     ee6dc98d-105d-4536-ba0e-73ac8314a37b

    Volumesets will show up in the FlockerHub web interface in the volumeset tab at the top.

    Note, volumesets or other changes will only show up after a fli sync

    VSets

    Volume

    A volume, as it relates to FlockerHub and Fli, is a working mount-point. We don’t store “working” volumes on FlockerHub, but rather snapshots of those volumes. You can, however, turn any snapshot into a working volume.

    Think of a volume as a directory that you can work with like any other directory, except this volume can be snapshotted and that snapshot can be pushed and pulled to and from FlockerHub.

    When you create a volume with Fli, it looks like this.

     $ fli create my-volumeset my-volume
     /chq/6b18d8d2-2bca-4409-9b0e-a2ae69173346/0ee998a5-3c44-4a6d-8e5e-518bceae8a5c

    You can move into that directory, create files, or anything you would normally do on POSIX file-systems.

     $ cd /chq/6b18d8d2-2bca-4409-9b0e-a2ae69173346/0ee998a5-3c44-4a6d-8e5e-518bceae8a5c
     /chq/6b18d8d2-2bca-4409-9b0e-a2ae69173346/0ee998a5-3c44-4a6d-8e5e-518bceae8a5c$ touch hi

    Snapshot

    A snapshot is a “picture” of a volume that can’t be mounted itself to a host, but it can be instantiated as a data volume that can be mounted to a host through the clone command.

    To create a snapshot from an existing volume, you will use the snapshot command.

     $ fli snapshot my-volumeset:my-volume my-snapshot

    Clone

    A clone is a copy of a snapshot that is made available as a volume. Cloning a volume is used when you want to instantiate a volume based on a particular snapshot.

    This allows you to create a volume with pre-existing point-in-time versions of your data.

    To create a clone from a snapshot you may run the following clone command.

     $ fli clone my-volumeset:my-snapshot my-cloned-volume-name
     /chq/939470e9-3c12-4e55-a679-1baf6c65404e/6741ef71-2470-4d2f-87f3-259c376949e3

    Branches

    A branch is a lineage of a particular data volume made up of multiple snapshots over time. Think of a branch like a GitHub branch, but made of snapshots instead of commits.

    One way a Fli branch is not similar to Git is that volume branches do not necessarily come from a single trunk, nor do they have the ability to merge back in; e.g. “Production data” might be one branch; “staging data” another branch; and “local data” a third branch.

    Branches are created by using the --branch | -b flag during a snapshot command.

     $ fli snapshot my-volumeset:my-volume my-snapshot --branch my-dev-branch

    Branches are visible in the FlockerHub Web Interface if you click on a particular volumeset. Clicking on a branch shows you which snapshots belong to that branch.

    branches

    Sharing

    FlockerHub volumesets can be shared among users.

    By allowing organizations and teams to share snapshot resources, they can easily enable better testing during CI/CD, development, QA and debugging stages of application development. They can then share necessary storage resources to test and find bugs faster and more efficiently in the DevOps life cycle.

    To share a volumeset, navigate to the Volumesets page. Then, click on the check box to the left of a volumeset that you want to share. You should see the share button light up red with the number of volumesets you are trying to share.

    clickshare

    Next, click the share button and a new window will pop up asking you to enter the email addresses of the users you would like to share the volumeset with, along with a message about what is being shared.

    sharepopout

    To set the permissions when sharing to users, you must be the Owner or Admin of a volumeset. Choose the permission setting from the drop down menu in the sharing windows as shown below.

    readwriteadmin

    Once you have entered the email addresses and message you would like, click share and a notification should tell you that you have successfully shared the volumeset.

    Emails will be sent to the users with which you wish to share your volumeset. If the recipients are existing FlockerHub users, they will see a Collaborate button in their email to start using the volumeset.

    collaborate

    If the recipients are not existing FlockerHub users, then they will get an email with a Create Account button.

    createaccount

    Fli and FlockerHub Architecture

    FlockerHub is made up of a few main parts, and this next section will give a high-level overview of the components make up a FlockerHub instance.

    This will help you get a better understanding of the moving parts when you are working with Fli and FlockerHub.

    Note, as a user of Fli and FlockerHub, you do not need to get caught up in the details of how FlockerHub components work, it is merely a reference for those interested in what’s behind the covers of FlockerHub.

    FHHLArch

    Data Servers

    Many data servers make up the the highly scalable back-end of a FlockerHub deployment. It is a Golang server whose main purpose is to store user snapshots in an intelligent, safe and secure manner. You can think of the data server as the place where your data rests after you push it up to FlockerHub. As a FlockerHub user, you don’t need to worry about running the data servers. We do this for you.

    Data Plane

    Data plane servers are another scalable component that handle requests from the Fli client to and from the data servers, browser and authentication services. Data plane servers have the important job of brokering requests for information about users and data with the responsibility of doing this safely and securely. You also don’t need to worry about the data plane. We run this for you, too.

    FlockerHub Web Interface

    This is where you log in and view, organize and manage volumesets, branches, snapshots, tokens and more.

    The user interface will talk to different components of the system, such as the data plane, to gather information about users and snapshots that the user should have access to view via the UI.

    Authentication and Access Control

    The authentication and access control mechanisms for FlockerHub and Fli are made up of multiple components.

    One of the components is responsible for managing authentication tokens and who they belong to. This ensures that only authorized users can access the system.

    Another component is responsible for managing users, organizations and permissions settings that relate to the volumesets, branches and snapshots so that access controls to such resources is maintained across Fli and FlockerHub.

    A final component is responsible for talking with OAuth2 so that users can be authenticated via their existing GitHub and Google accounts.

    Fli Client and FlockerHub Interactions

    Fli is the CLI component, and is separate from the FlockerHub deployment. Fli is installed on client machines that wish to push, pull and create snapshots. It must be configured with an authentication token created in FlockerHub so that the the Fli client has the correct access to push, pull and sync the volumesets, branches and snapshots it has access to.

    branches

    Conclusion

    Hopefully this has been useful reading that has given you a better understanding of the components and interactions that make up Fli and FlockerHub!

    We will continue to post new blogs that dive deeper into some of these components and use cases for Fli and FlockerHub. So, stay tuned!

    More Details

    How FlockerHub works

    How Fli works

    Videos

    Fli Fundementals

    Better CI with FlockerHub and Fli

    Fli- Taking Snapshots

    FlockerHub- Updating Object Metadata

    FlockerHub- Creating Volumes

    Push and Pull Data Snapshots with Fli and FlockerHub

    How to setup Fli with your ZPool

    Using fli-docker to bootstrap Docker Compose Apps with data

    Use Cases

    Better CI: Integrate realistic data, accelerate your pipeline

    On-demand staging: Include realistic test data for each commit

    Test better: Easily create a scalable, distributed test fixture library

    Share error state: Collaboratively debug the most challenging errors

    On-demand dev environments: Share your workspace with anyone

    Like what you read?

    Signup for a free FlockerHub account.
    Sign Up

    Get all the ClusterHQ News

    Stay up to date with our newsletter. No spam. Ever.