Table of Contents
What is Redis? It is an open source (BSD licensed), in-memory data structure store, used as database, cache and message broker. It supports data structures such as strings, hashes, lists, sets, sorted sets with range queries, bitmaps, hyperloglogs and geospatial indexes with radius queries. Redis has built-in replication, Lua scripting, LRU eviction, transactions and different levels of on-disk persistence, and provides high availability via Redis Sentinel and automatic partitioning with Redis Cluster.
Flocker is a container data volume manager that is designed to allow databases like Redis to easily run inside containers in production. When running a database in production, its important to think about things like recovering from host failure and availability of data. Flocker provides tools for managing data volumes across a cluster of machines like those in a production environment. For example, as a Redis container is scheduled between hosts in response to server failure, Flocker can automatically move its associated data volume between hosts at the same time. This operation can be scheduled using the Flocker API or CLI, or automatically by a container orchestration framework that Flocker integrates with such as Docker Swarm, Kubernetes and Mesos.
In this example, we’ll be using Kubernetes to move Docker containers among nodes in a cluster.
Why run Redis in Docker?
Redis has a number of persistence options that have certain advantages such as increased durability and advanced snapshots. By using these optiones along with Flocker, Redis can get increased flexibility and safety for persisted data in a container environment.
Also, as database workloads like Redis scale up or down, we need to make sure that Redis Server has enough or as little CPU, RAM, and network bandwidth as needed to handle our capacity needs. Running Redis Server in a container makes it portable so it may manually or automatically move to a different, possibly more powerful server with ease. Additionally, running Redis in a container means it can be automatically rescheudled to a new host in failure senarios such as crashed or failing servers.
This is where Flocker comes in. By running Redis in a Docker container managed by Flocker, when the container is rescheduled, its data moves to the new host along with the container, reducing downtime and headaches. Flocker can be used to manage volumes attached in a 1:1 fashion with replicated Redis slaves or individual Redis masters.
What you will learn in this blog
In this tutorial you’ll learn how to run a Redis master-slave cluster deployment on Kubernetes. Additionally, we’ll show you how to migrate a Redis container and its data stored on AWS EBS between hosts in case of scheduled maintenance or other operational tasks.
The tutorial will take advantage of Kubernetes to easily deploy Redis slaves and easily connect them to the cluster. We will create a Replication Controller and a Service for the Redis master and every Redis slave. This will enable Redis slaves to easily query Kubernetes’ built-in methods for discovering services to easily join the cluster over the network.
In order to run the tutorial, we’ll first start by installing and configuring Kubernetes and Flocker. If you already have Kubernetes and Flocker installed, skip to the good bits.
NOTE: The redis containers in this tutorial use AOF (Append Only File) To learn about reasons why you might want to do this, click the link.
In this tutorial there are:
- 5 Nodes total.
- 1 master node with Kubernetes Master and Docker installed.
- 1 Control Service for the Flocker Control Service node.
- 3 Nodes with our Flocker Agent Services, Docker, and Kubernetes node services installed (Our Redis cluster is going to be deployed and move between these nodes inside of PODs).
In this example we will be running our nodes on Amazon EC2 and creating and attaching volumes from Amazon’s EBS service.
QuickStart: Setting up a Flocker + Kubernetes Cluster
This QuickStart installs Kubernetes v1.1
Install Kubernetes on Ubuntu using this guide
The guide above uses some environments variables for installation. Export some optional settings if you would like to follow along.
NOTE: Using larger instance types will cost more. Lastly, make sure you have AWS CLI installed and configured.
Create the Kubernetes Cluster
NOTE: You will need EC2 Full access and S3 Access for this process and it could take up to 10-15 minutes to complete.
You can view your UI by going to the
KubeUI URL above and entering your username and password from the
kubtctl tool to use k8s
kubectl.shpath is from the installation output.
NOTE: We suggest having a seperate node for your Flocker Control Service and Kubernetes Master Services.
We’ll be creating a seperate node for the Flocker Control Service
You can use the AWS Console Launch Wizard for this.
Use the same AMI that the Kubernetes Install is using.
Launch in the correct VPC, you can find the VPC information by clicking any existing Kubernetes node.
Use the same kubernetes KeyPair created with the above tool. It should look something like this.
Add the security groups for kubernetes.
Finally create an elastic IP for the Flocker Control node and associate the control service. You can do this by selecting “Elastic IPs” from the menu on the left of the AWS console.
Find the security group ids for the
Worker nodes. You can find then with
aws ec2 or in your AWS Console.
Open up the Flocker ports using
Install the Flocker CLI and create a
yaml file for Flocker installation.
Install CLI on your local machine.
Example Flocker YAML.
You will need to replace the Public IPs, Private IPs, Key location and Control Node DNS name.
Hint: You can find the IPs easily with the AWS CLI if you want. Examples below.
flocker.yaml file is complete, install Flocker with these commands.
Configure your Flocker + K8S Cluster to use Flocker
NOTE: You can follow manual instruction for configuration flocker here or use the script below for automation.
Run the configuration script
You will need Python Twisted, and Python YAML installed to run this script.
You should see output like the following if the script was succesful.
Make sure you certs are correct so you can use
flockerctl against your cluster.
Once Flocker and Kubernetes are installed, Install the Volume Hub Dashboard on the Flocker/Kubernetes nodes.
Setup Volume Hub Dashboard
The Volume Hub displays a catalog of all the data volumes in a Flocker cluster, gives visibility into your logs, and makes for a more seamless user experience.
Configure your cluster for the Volume Hub
The Volume Hub Agents push the latest metadata about your cluster into the Volume Hub.
1. On the control service node
2. On just one of your agent nodes
3. On all the rest of your agent nodes
Go back to your Volume Hub account and you should see the cluster and agent nodes displayed with no attached volumes.
Getting Started Running Redis on Kubernetes
Now that Kubernetes and Flocker are installed we can begin the steps needed to deploy Redis.
The first thing that needs to be done is to create the volume resources so Kubernetes can use them when deploying PODs. Let’s start by creating two volumes, one for a Redis master and another for a Redis slave.
- Create 2 seperate volumes, one for the Redis master and one for the Redis slave
- Deploy Redis Master Replication Controller and Service Endpoint
- Generate some sample data into Redis
- Deploy the Redis slave Replication Controller and Service Endpoint
Creating Volumes for our Kubernetes PODS
The first step is to create the volume resources that our Kubernetes PODS will use to store the Redis data.
List the nodes, and choose one UUID.
Volume for Redis Master
Volume for Redis Slave
You should be able to look at your Volume Hub dashboard and see the volume being created and attached.
Volumes being created
Running a Redis Master Replication Controller
Now that we have created the Volumes we can deploy the Replication Controllers, PODs, and Services. The first thing we want to do is define a Replication Controller which defines the containers that will make up a POD and the minimum number of PODs to run at any given time.
This POD will have a replica of 1 (because we only want 1 container using our 1 volume), a Redis Master Docker container and a Flocker Volume that is used by the Redis Master.
Below is an example of how this will look, you can save this file as
To run this this POD use the kubectl command with the
create command and pass it the
yaml file from above.
NOTE: The script automatically uses the
kubeconfigfile created by the installation above. Read the documentation above on
kubectlfor more information.
The Master POD should become available if you issue the
get po command.
To verify the 10G volume is available inside your container, you can use the following command to check the
/var/lib/redis directory and device.
<Redis POD NAME>with the name from above command.
Next, create a Kubernetes Service for the master to expose it within your cluster as a service. Below is an example YAML you can use, save it as
create the service the same way as the controller
Redis Master should then be available as a Service and Endpoint. Use the
get svc and
get ep commands to see this.
Here is an example of
This endpoint can now be used by other PODs becuase it is exposed as a service.
The service is available at the IP from above or in a environment variable
NOTE: Kubernetes exposes services this way, it is one way out of others for accessing services. Please see the Kubernetes documentation for more detail.
Add data to Redis Master
To add data to Redis, log into the container and run the redis CLI. Below is an example of this.
NOTE: The IP (10.2.51.8) used in the example comes from the
get epcommand above.
Adding a Redis Slave for Replication
The next step is to create a Redis slave and add it to our Redis master to achieve a small replicated Redis cluster.
NOTE: You may add 1 or more slaves at this point, adding one is only an example.
To do this, repeat the process from above, except this time we define a Service Spec and a Replication Controller Spec for the slave.
Here is the Spec for the controller.
Notice it uses the
redis-slave volume created earlier.
Save this one as
Here is the spec for the service.
Save this one as
Then, create them with
Verify that the slave is running by running
This command should output both Redis Master and Redis Slave PODs.
View that our Redis Slave SYNCed with Master
To check on the status of the Redis Slave, run the following.
The logs should display output showing our Master/Slave syncing and replicating our sample data from our master into memory.
HINT: Notice the
MASTER <-> SLAVE sync startedportion of the logs.
To see that our Redis Slave actually SYNCed the data, let’s log-in and list the contents.
NOTE: The IP (10.2.64.6) used in the example comes from the
get epcommand above.
There you have it! All our data is in the database. We can also check that the data is persisted to disk by running the following command. The output is the Redis protocol saving data to disk. This handy to save for example because it can be replayed for recovery purposes.
Performing a data migration
A common operation might be to migrate your Redis containers / PODs in the case of scheduled downtime or scheduled upgrades. One way we can do this is by labeling Kubernetes nodes and migrating our application while letting Flocker worry about where data should be moved to.
Label a node
The first thing we want to do is label a node that wont be affected by the upgrade or maintenance.
You can do this with
kubectl. Let’s label a node with
Now we can list the nodes and see the label is there.
To move our POD we can add the
nodeSelector attribute to the spec file. To move the Redis slave we open our
redis-slave-controller.yaml file and add the
To redeploy the container and make sure it moves to a the selected server along with its data, stop and start the POD.
This will cause the Redis slave POD to be rescheduled onto the specific node with
During this rescheduling, Flocker will move the data volume and attach it to the host where the POD started on, this way your data remains available!
NOTE: If you had more than one server with this label, Kubernetes would choose one of them.
You can view the fact that Flocker has moved your volume by using the ClusterHQ Volume Hub and taking note of the changes.
Volume is being moved
Volumes re-attached to
staging Kubernetes node
Containers view and search for
You can view your Redis Master and Slave in the Kube UI both configured with Flocker EBS Volumes
nodeSelector can be a useful scheduling filter and in this case we use it for migration.
Flocker and Kubernetes will react the same way in the case of a node failure where Kubernetes will recognize the failed node and reschedule the container automatically while Flocker will react to placing your data where its needed.
Running a Redis cluster on Kubernetes with Flocker volumes is useful in single instance examples but to run Redis is a more production like configuration, using clustering options is preferable. The main take aways from this post should be:
- Using Redis and a Master-Slave configuration works well with Flocker and Kubernetes
- Using the Volume Hub makes it easy to visualize containers, volumes and movements with Kubernetes.
- Spinning up new Redis Slaves and persisting the data to shared storage can easily be achieved by using Flocker
- Getting the extra level of safety for your data can be achieved by not only running a cluster but using shared storage behind the Redis database.
- Migrations and movements done by the Kubernetes scheduler are atomic with Flocker, your data moves with the container.
We’d love to hear your feedback!