Fli is the CLI for FlockerHub. You can think of it like Git, but for data. Just like you can use Git to copy, branch, push and pull code, you can use Fli to copy, branch, push and pull your data volumes.
When Fli, available for download as an Apache 2.0 open source project, is combined with FlockerHub, you can share your data with any other user or computer on the Internet to which you’ve granted access, the way that GitHub lets you share your code version-controlled with Git. Think of Fli as a way to make your data fly. See what we did there?
With version-controlled data and fine-grained access controls, you can speed up development and debugging without losing control of your data.
Fli lets you take incremental snapshots of any database or data volume that runs on Linux, and push those snapshots to a hosted volume repository called FlockerHub where they can be accessed by any person or machine to whom you’ve granted access.
Because the snapshots are incremental, each additional push or pull to the FlockerHub sends only part of the data, speeding up transfer times and reducing bandwidth consumption.
Additionally, just like you can use Git to organize your code into repos and branches, Fli gives you the ability to organize your data into volume sets and branches. You can use Fli to manage these snapshots locally, or push them to FlockerHub to share with others.
Just as it would be impossible to describe everything you can do with git in a single use case, it is impossible to show everything you can build using FlockerHub & Fli in a single page. Let’s just scratch the surface with Tiffany, a DevOps engineer building a microservice using Docker.
New feature, new microservice
Tiffany is working on a new feature to increase sales on an e-commerce site, a 'You might also like…' widget to be displayed on every product page. Tiffany needs to build, test and deploy the microservice end-to-end and provide a version of the microservice to her teammates so they can test their new features against it.
Building locally using Docker Compose
Tiffany starts by spinning up the e-commerce environment on her laptop, using a Docker Compose file she got from her manager. She’ll test her feature first locally using this environment.
Starting her feature branch
Now that she’s got her development environment set up, she creates a new branch and writes an initial implementation of the feature.
Testing locally is incomplete without data
Tiffany’s got the feature more or less how she wants it, and she wants to test it locally on her machine. She can run her branch against the local environment spun up with Docker Compose, but there is no data in the Order History and Product Catalogue databases that the widget needs, so she can’t see if the logic is correct.
Realistic data comes from production
Tiffany could put together some test data, but she would almost certainly miss edge cases that she’d find testing against realistic data. Tiffany asks Paul, her colleague in Ops, to send her the anonymized last 5 million entries from the Order History MongoDB database and the entire Product Catalog, all 1.5 million products, PostgreSQL database.
Snapshotting volumes with Fli
After receiving the data, Tiffany snapshots each database using Fli and pushes these volumes to the FlockerHub. That way, not only can she use this test data, but any of her colleagues or staging/QA environments can too.
Test data reveals a bug (Shocking!)
With her realistic test data available locally, Tiffany previews her branch. Looks good, except for that wrapping issue on the long product name. Who knew there was something called The Really, Incredibly, Stupendously, Fantastically Good Stain Remover®?
A little tweaking and voila...
Tiffany fixes the CSS for displaying long names and tries again. Bingo. Looks good.
And now for CI
Now that things are looking good locally, Tiffany needs to run her code through her company’s Jenkins environment. Their CI system kicks off for each commit in GitHub, so before pushing her branch, Tiffany makes sure that her Order History MongoDB and Product Catalog PostgreSQL that she pushed to FlockerHub are pulled into Jenkins /home/test-fixtures on all the Jenkins build slaves. This way, no matter which slave her build executes on, it will run her tests against the realistic data.
When a build fails, capture the state
Tiffany runs her test but the build fails when her integration test notices that one of the three product suggestions fails to load. At first glance, she is not quite sure what happened but can figure it out. Her team sets up Jenkins to snapshot all test databases using Fli as soon as a build fails, and then pushes these snapshots to FlockerHub. This way, Tiffany or any of her colleagues can have a completely consistent environment— code, container images and state–in which to debug.
Debugging locally with actual data
Because Fli uses incremental snapshots, the snapshot and push and subsequent pulls of the Mongo and Postgres data volumes from the Jenkins environment only take a few seconds, so in no time, Tiffany has an identical application environment running locally.
Sharing a development environment, including data, with a colleague
Tiffany digs around a bit and can’t recreate the bug. She decides to ask her colleague Sam to take a look. She sends Sam a Docker Compose file, which references her data volume on FlockerHub. Sam pulls all the Docker images and data volumes and is running an identical environment in minutes.
Armed with data, Sam bags the bug
Sam steps through the integration test and notices the bug only appears when there are exactly 4,294,967,296 orders in the Order History database. The code works fine up to 4,294,967,295 orders, then something goes horribly awry. But Sam’s brain always has a background thread calculating base 2 logarithms, so the problem is elementary. She knows exactly where to tweak the code. When she reruns the test, it displays fine.
Build passes, time for the next feature!
Sam commits her changes, the Jenkins build runs, and the test passes, even though it runs against the exact same database state as before, with 4,294,967,296 orders in the Order History database. High fives all around.