Playground to play with Fluentd and Elasticsearch+Kibana. Prepared for Cloud-Native-Singapore August 2016 meetup.
This is fully based on Jeff Sogolov's Presentation visualizing Logs using ElasticSearch and Kibana.
Video on Youtube & GitHub Repository
Sample Dashboard:
In a world of Ephemeral containers, how can we centralise the logs and get meaningfull analysis out of them?
One of the major struggles with any large deployment is logging. Having a central place to aggregate logs makes troubleshooting and analysis considerably easier to do.
As part of the presentation, this playground environment was created to get familiar with the log management tools leveraged by Kubernetes.
The official Kubernetes Provisioning scripts come with the option to deploy several add-ons. One such add-on is the Fluentd based log aggregation setup. In this setup, the goal is for every node in the cluster to forward the logs of its containers as well as its agent (Kubelet) to a log indexing service.
For the containers, the default Docker logging driver (json), captures all data sent to stdout and stderr and stores it in json files in a directory per container.
As part of starting containers, the Kubelet also ensures a symbolic link is created to each json log file in a central directory. It is this central directoy on the host node which
the Fluentd agent, running itself in a container, is configured to monitor through a volume mount.
At the time of writing the logging add-on did not yet leverage the powerful DaemonSet controllers available in Kubernetes to ensure every cluster node ran a log shipping agent. Refer to the references section below on how to deploy the Fluentd agents to each node using DaemonSets instead.
Additionally, as part of the logging add-on, a 2 node Elasticsearch cluster for indexing is deployed and exposed as an internal cluster service. It is important to note that this Elasticsearch cluster does not follow the best practices of running Master, Client and Data Elasticsearch nodes recommended for performance and resiliency. Configuration and Manifest files to run a production ready ES cluster, with integrated Kubernetes cloud discovery for the latest Elasticsearch version are available and should be considered instead.
Nevertheless, the Fluentd+Elasticsearch & Kibana (EFK) stack provides very powerful analysis, as will be demonstrated using the setup in this repository.
To play with the full EFK stack locally, this repository currently leverages Docker Compose and Docker For Mac.
A sample log generator is used to simulate activity on an e-commerce website. The stack defined in Docker Compose will stand up inter-connected Elasticsearch, Kibana, Fluentd and Log-generator containers.
The Fluentd configuration is mounted from the current working directory, allowing you to experiment with its settings to parse the provided sample logs.
Contents of this repository:
.
├── Makefile Main Makefile to run demoscript
├── README.md This Readme
├── docker-compose.yaml Definition of full stack
├── fluentd >> Fluentd container setup
│ ├── Dockerfile Fluentd image recipe
│ ├── Makefile Build script for fluentd image
│ ├── elasticsearch-template.json Index template
│ ├── fluent.conf Fluentd configuration
│ ├── fluentd.vim Simple syntax highlighting
│ ├── plugins/ Placeholder for custom ruby parse scripts
│ └── versioning.mk Versioning Makefile
├── kibana-sense >> Kibana container setup
│ ├── Dockerfile Kibana image with Sense plugin
│ ├── Makefile Build script for Kibana image
│ └── versioning.mk versioning Makefile
├── log-generator >> Sample Log generator
│ ├── Dockerfile OpenJDK based image recipe
│ ├── Makefile Build script for Log generator
│ ├── README.md Original readme from log generator
│ ├── pom.xml Maven package
│ ├── src/ Source for Log Generator
│ └── versioning.mk Versioning Makefile
└── versioning.mk Main versioning Makefile
Go to log-generator sub folder
Building:
make build
Run as a stand-alone container:
make run
Follow the logs generated:
make logs
Review the logs generated by this sample application to understand the data they contain. Notice IPs are present but no Geoip data exists.
Stop & Remove the container
make stop
Building the image:
make build
This is based on the official Kibana image, but with the Sense plug-in made available.
Building the image:
make build
From the root directory:
Start Elasticsearch and create Index template:
make init
This will create the networks and persistent volume for Elasticsearch, it will automatically detect when Elasticsearch becomes available and run curl -XPUT ...index-template.json to create the index template for the log data sent by the Fluentd container
Bring up the full stack (Kibana/Fluentd & Log-Generator) - Note: it is required to run init first!
make up
Once all services started, You should be able to access Kibana on port 80 of your localhost (assuming port 80 is available)
Follow the logs generated by Fluentd:
make logs
You may now edit the fluentd/fluent.conf file on your local machine, to apply your config changes use:
make reload
If you are following the logs, you will notice the Fluentd container restarting and using the new configuration.
You may use Sense to understand the Elasticsearch data better
Once finished, stop and remove all containers with:
make down
Delete the Elasticsearch data (Visualizations / Dashboards / Indices / ...)
make clean
