Deploying and migrating an Elasticsearch-Logstash-Kibana stack using Docker Part 2

Flocker + ELK

This is Part 2 of 2 in a series about using Docker to deploy and migrate an Elasticsearch-Logstash-Kibana stack. Part 1 showed you how to deploy all three containers to a single host using the Docker CLI and Docker Compose. In Part 2, we will deploy the three containers across multiple hosts using Docker Swarm.

##Part 1 Summary

  • Elasticsearch is a distributed RESTful search engine ususally used in conjunction with Logstash and Kibana
  • Docker is a great place to run ElasticSearch because Docker makes it easy to move your ElasticSearch, Logstash and Kibana containers between servers.
  • While Logtash and Kibana are stateless, ElasticSearch is stateful. If you want to be able to migrate your ElasticSearch data along with your ElasticSearch container between hosts, you can use Flocker.

##What you will learn in Part 2 Tutorial 1 deployed all three containers in the ELK stack to a single host. While useful for exploratory purposes, this is not a very realistic production set up. ElasticSearch is notoriously memory hungry, so you probably want to give it its own dedicated machine, maybe one with access to high performance storage for fast reads and writes.

In this Tutorial, you will learn:

  • How to deploy the ELK stack across multiple hosts.
  • How to orchestrate your containers using Docker Swarm.
  • How to automatically move your three ELK containers and ElasticSearch’s data to a new server.

Like before, we will use the official ElasticSearch, Logstash and Kibana images from Docker Hub for images.

We’ll use Docker Swarm and Flocker, an open-source data volume for deployment, networking and migrations ops tasks.

We will be using this repository for the examples in the blog post, feel free to try it out.

Multi Node Deployment

Architecture

In this tutorial there are:

  • 3 nodes.
  • 1 master node with our Flocker Control Service and Docker installed.
  • 2 nodes with our Flocker Agent Services and Docker installed (our ELK stack is going to be deployed across these nodes via Swarm.)

In this example we will be running our nodes on Amazon EC2 and creating and attaching volumes from Amazons EBS service.

To run an ELK stack in a multi-node fashion, we can use Docker Swarm for orchestrating the placement of our containers.

First, make sure Swarm is setup on your cluster. Below is an example of what this may look like, notice we have 2 nodes in our cluster ip-10-0-70-72 and ip-10-0-202-37.

Swarm setup will not be covered in this post. See Swarm docs for more on installing Swarm

$ user@localhost:~# export DOCKER_HOST=tcp://ec2-54-172-223-122.compute-1.amazonaws.com:2357
$ user@localhost:~# docker info
Containers: 2
Images: 2
Role: primary
Strategy: spread
Filters: health, port, dependency, affinity, constraint
Nodes: 2
 ip-10-0-70-72: 10.0.70.72:2375
  └ Status: Healthy
  └ Containers: 1
  └ Reserved CPUs: 0 / 2
  └ Reserved Memory: 0 B / 7.67 GiB
  └ Labels: executiondriver=native-0.2, kernelversion=3.13.0-63-generic, operatingsystem=Ubuntu 14.04.3 LTS, storagedriver=aufs
 ip-10-0-202-37: 10.0.202.37:2375
  └ Status: Healthy
  └ Containers: 1
  └ Reserved CPUs: 0 / 2
  └ Reserved Memory: 0 B / 7.67 GiB
  └ Labels: executiondriver=native-0.2, kernelversion=3.13.0-63-generic, operatingsystem=Ubuntu 14.04.3 LTS, storagedriver=aufs
CPUs: 4
Total Memory: 15.34 GiB
Name: 122731d0708a

From this repository you can cd multi-node/ for an example docker-compose-multi.yml file you can use for deploying ELK against a Swarm cluster.

Edit docker-compose-multi.yml to contain the node #1 name from docker info output in the Elasticsearch section of the Compose file. Also add the ip address of the same node in the Logstash and Kibana configuration sections.

This will make sure Kibana points correctly at the right IP address of the Elasticsearch node.

An alternate solution would be to enable Docker overlay networks but it will not be covered in this post.

Here is a snippet of what your docker-compose-multi.yml may look like.

elasticsearch:
  image: elasticsearch:latest
  command: elasticsearch -Des.network.host=0.0.0.0
  ports:
    - "9200:9200"
    - "9300:9300"
  volume_driver: flocker
  volumes:
    - elasticsearch1:/usr/share/elasticsearch/data
  environment:
    - constraint:node==ip-10-0-70-72
logstash:
  image: logstash:latest
  command: logstash -e 'input { tcp { port => 5000 } } output { elasticsearch { hosts => "10.0.70.72:9200" } }'
  ports:
    - "5000:5000"
kibana:
  image: kibana
  ports:
    - "5601:5601"
  environment:
    - ELASTICSEARCH_URL=http://10.0.70.72:9200

Now, run your ELK stack in the same fashion as you did in Part 1, but this time were talking to our Swarm Master

$ user@localhost:~/elk-flocker-compose/multi-node# docker-compose -f docker-compose-multi.yml up -d
Pulling elasticsearch (elasticsearch:latest)...
ip-10-0-70-72: Pulling elasticsearch:latest... : downloaded
ip-10-0-202-37: Pulling elasticsearch:latest... : downloaded
Creating multinode_elasticsearch_1
Pulling logstash (logstash:latest)...
ip-10-0-70-72: Pulling logstash:latest... : downloaded
ip-10-0-202-37: Pulling logstash:latest... : downloaded
Creating multinode_logstash_1
Pulling kibana (kibana:latest)...
ip-10-0-70-72: Pulling kibana:latest... : downloaded
ip-10-0-202-37: Pulling kibana:latest... : downloaded
Creating multinode_kibana_1

During this process you should see a volume getting created for the Elasticsearch container, here is the output of the Flocker CLI during the process.

Volume is being created

$ user@localhost:-> flockerctl list
DATASET                                SIZE     METADATA              STATUS     SERVER
a5c1f04e-6d49-43b4-8090-eea134309f6d   75.00G   name=elasticsearch1   detached   6709227e (10.0.70.72)

Volume is created and attached to the host

$ user@localhost:-> flockerctl list
DATASET                                SIZE     METADATA              STATUS         SERVER
a5c1f04e-6d49-43b4-8090-eea134309f6d   75.00G   name=elasticsearch1   attached ✅   6709227e (10.0.70.72)

Now, if we run a docker ps command, we can see our containers are running on different nodes in our cluster. Elasticsearch is on ip-10-0-70-72 and Logstash and Kibana are on ip-10-0-202-37

$ user@localhost:# docker ps
CONTAINER ID        IMAGE                  COMMAND                  CREATED             STATUS              PORTS                                                  NAMES
6013e3b548bc        kibana                 "/docker-entrypoint.s"   3 minutes ago       Up 3 minutes        10.0.202.37:5601->5601/tcp                             ip-10-0-202-37/multinode_kibana_1
5588b5abf326        logstash:latest        "/docker-entrypoint.s"   3 minutes ago       Up 3 minutes        10.0.202.37:5000->5000/tcp                             ip-10-0-202-37/multinode_logstash_1
70a35ab5857d        elasticsearch:latest   "/docker-entrypoint.s"   3 minutes ago       Up 3 minutes        10.0.70.72:9200->9200/tcp, 10.0.70.72:9300->9300/tcp   ip-10-0-70-72/multinode_elasticsearch_1

Now if we login to our node running Logstash, we can add some logs.

$ user@localhost:-> ssh -i ~/user.pem root@54.152.42.22
$ root@node-with-logstash-container:~# nc localhost 5000 < /var/log/flocker/flocker-dataset-agent.log

Now we can visit our Kibana UI and see we have some data.

Flocker ELK data 3

Now we can move our Elasticsearch database to perform a migration. You may want to do this as your workload scales up or the server doesn’t have enough CPU, RAM, and network bandwidth to handle near and longterm capacity needs. Running your ELK stack in containers makes it portable so you can manually or automatically move that container to a more powerful machine with ease. You can also more easily respond to system failures like crashed servers. Flocker comes in by making sure your data directory moves to the new host along with your container reducing downtime and headaches. The same thought process can be used if you wanted to downgrade the host server to something more affordable with moderate performance attributes.

Migrating our ELK stack with Docker Swarm means re-deploying Docker Compose against Swarm with a new configuration. Here is our new docker-compose-multi.yml:

Notice, our IP address and node have changed to another node in the configuration.

elasticsearch:
  image: elasticsearch:latest
  command: elasticsearch -Des.network.host=0.0.0.0
  ports:
    - "9200:9200"
    - "9300:9300"
  volume_driver: flocker
  volumes:
    - elasticsearch1:/usr/share/elasticsearch/data
  environment:
    - constraint:node==ip-10-0-202-37
logstash:
  image: logstash:latest
  command: logstash -e 'input { tcp { port => 5000 } } output { elasticsearch { hosts => "10.0.202.37:9200" } }'
  ports:
    - "5000:5000"
kibana:
  image: kibana
  ports:
    - "5601:5601"
  environment:
    - ELASTICSEARCH_URL=http://10.0.202.37:9200

We can re-deploy our ELK stack by running the following with our new yml file.

$ user@localhost:~/elk-flocker-compose/multi-node# docker-compose -f docker-compose-multi.yml stop
Stopping multinode_kibana_1 ... done
Stopping multinode_logstash_1 ... done
Stopping multinode_elasticsearch_1 ... done

$ user@localhost:~/elk-flocker-compose/multi-node# docker-compose -f docker-compose-multi.yml up -d
Creating multinode_elasticsearch_1
Creating multinode_logstash_1
Creating multinode_kibana_1

Now, if we run a docker ps command, we can see our containers are running on different nodes in our cluster. Elasticsearch and Kibana are on ip-10-0-202-37 and Logstash and is on ip-10-0-70-72

$ user@localhost:~/elk-flocker-compose/multi-node# docker ps
CONTAINER ID        IMAGE                  COMMAND                  CREATED              STATUS              PORTS                                                    NAMES
a8396236c3a2        kibana                 "/docker-entrypoint.s"   56 seconds ago       Up 54 seconds       10.0.202.37:5601->5601/tcp                               ip-10-0-202-37/multinode_kibana_1
02efcc91132a        logstash:latest        "/docker-entrypoint.s"   56 seconds ago       Up 55 seconds       10.0.70.72:5000->5000/tcp                                ip-10-0-70-72/multinode_logstash_1
df85d85b5816        elasticsearch:latest   "/docker-entrypoint.s"   About a minute ago   Up 56 seconds       10.0.202.37:9200->9200/tcp, 10.0.202.37:9300->9300/tcp   ip-10-0-202-37/multinode_elasticsearch_1

We can also see that Flocker responded to the migration by moving Elasticsearch data to our new node.

Notice the IP address is now 10.0.202.37 now.

$ user@localhost:-> flockerctl list
DATASET                                SIZE     METADATA              STATUS         SERVER
a5c1f04e-6d49-43b4-8090-eea134309f6d   75.00G   name=elasticsearch1   attached ✅   6d2b7d6b (10.0.202.37)

What we saw during this migration was:

Flocker ELK Movement

So now if we refresh our Kibana UI since it’s on the same host as it was before, we should see the same data even though our Elasticsearch database now lives on a different host.

Flocker ELK data 4

Feedback

We’d love to hear your feedback!

Like what you read?

Signup for a free FlockerHub account.
Sign Up

Get all the ClusterHQ News

Stay up to date with our newsletter. No spam. Ever.