Skip to content

Latest commit

 

History

History
188 lines (126 loc) · 7.66 KB

README.md

File metadata and controls

188 lines (126 loc) · 7.66 KB

Docker EFK stack

Playground to play with Fluentd and Elasticsearch+Kibana. Prepared for Cloud-Native-Singapore August 2016 meetup.

This is fully based on Jeff Sogolov's Presentation visualizing Logs using ElasticSearch and Kibana.

Video on Youtube & GitHub Repository

Sample Dashboard:

Image of Dashboard

Introduction

slides

In a world of Ephemeral containers, how can we centralise the logs and get meaningfull analysis out of them?

One of the major struggles with any large deployment is logging. Having a central place to aggregate logs makes troubleshooting and analysis considerably easier to do.

As part of the presentation, this playground environment was created to get familiar with the log management tools leveraged by Kubernetes.

The Kubernetes Logging Add-on

The official Kubernetes Provisioning scripts come with the option to deploy several add-ons. One such add-on is the Fluentd based log aggregation setup. In this setup, the goal is for every node in the cluster to forward the logs of its containers as well as its agent (Kubelet) to a log indexing service.

For the containers, the default Docker logging driver (json), captures all data sent to stdout and stderr and stores it in json files in a directory per container. As part of starting containers, the Kubelet also ensures a symbolic link is created to each json log file in a central directory. It is this central directoy on the host node which the Fluentd agent, running itself in a container, is configured to monitor through a volume mount.

At the time of writing the logging add-on did not yet leverage the powerful DaemonSet controllers available in Kubernetes to ensure every cluster node ran a log shipping agent. Refer to the references section below on how to deploy the Fluentd agents to each node using DaemonSets instead.

Additionally, as part of the logging add-on, a 2 node Elasticsearch cluster for indexing is deployed and exposed as an internal cluster service. It is important to note that this Elasticsearch cluster does not follow the best practices of running Master, Client and Data Elasticsearch nodes recommended for performance and resiliency. Configuration and Manifest files to run a production ready ES cluster, with integrated Kubernetes cloud discovery for the latest Elasticsearch version are available and should be considered instead.

Nevertheless, the Fluentd+Elasticsearch & Kibana (EFK) stack provides very powerful analysis, as will be demonstrated using the setup in this repository.

This Repository Setup

To play with the full EFK stack locally, this repository currently leverages Docker Compose and Docker For Mac.

A sample log generator is used to simulate activity on an e-commerce website. The stack defined in Docker Compose will stand up inter-connected Elasticsearch, Kibana, Fluentd and Log-generator containers.

The Fluentd configuration is mounted from the current working directory, allowing you to experiment with its settings to parse the provided sample logs.

Running the playground

Contents of this repository:

.
├── Makefile                            Main Makefile to run demoscript
├── README.md                           This Readme
├── docker-compose.yaml                 Definition of full stack
├── fluentd                             >> Fluentd container setup
│   ├── Dockerfile                      Fluentd image recipe
│   ├── Makefile                        Build script for fluentd image
│   ├── elasticsearch-template.json     Index template 
│   ├── fluent.conf                     Fluentd configuration
│   ├── fluentd.vim                     Simple syntax highlighting
│   ├── plugins/                        Placeholder for custom ruby parse scripts
│   └── versioning.mk                   Versioning Makefile
├── kibana-sense                        >> Kibana container setup
│   ├── Dockerfile                      Kibana image with Sense plugin
│   ├── Makefile                        Build script for Kibana image
│   └── versioning.mk                   versioning Makefile
├── log-generator                       >> Sample Log generator
│   ├── Dockerfile                      OpenJDK based image recipe
│   ├── Makefile                        Build script for Log generator
│   ├── README.md                       Original readme from log generator
│   ├── pom.xml                         Maven package
│   ├── src/                            Source for Log Generator
│   └── versioning.mk                   Versioning Makefile
└── versioning.mk                       Main versioning Makefile

Log generator (./log-generator/)

Go to log-generator sub folder

Building:

make build

Run as a stand-alone container:

make run

Follow the logs generated:

make logs

Review the logs generated by this sample application to understand the data they contain. Notice IPs are present but no Geoip data exists.

Stop & Remove the container

make stop

Kibana Image (./kibana-sense)

Building the image:

make build

This is based on the official Kibana image, but with the Sense plug-in made available.

Fluentd Image (./fluentd)

Building the image:

make build

Full Stack (Repository Root)

From the root directory:

Start Elasticsearch and create Index template:

make init

This will create the networks and persistent volume for Elasticsearch, it will automatically detect when Elasticsearch becomes available and run curl -XPUT ...index-template.json to create the index template for the log data sent by the Fluentd container

Bring up the full stack (Kibana/Fluentd & Log-Generator) - Note: it is required to run init first!

make up

Once all services started, You should be able to access Kibana on port 80 of your localhost (assuming port 80 is available)

Follow the logs generated by Fluentd:

make logs

You may now edit the fluentd/fluent.conf file on your local machine, to apply your config changes use:

make reload

If you are following the logs, you will notice the Fluentd container restarting and using the new configuration.

You may use Sense to understand the Elasticsearch data better

Image of Sense

Cleaning up / Resetting

Once finished, stop and remove all containers with:

make down

Delete the Elasticsearch data (Visualizations / Dashboards / Indices / ...)

make clean

References