Elastic Stack Deployment on Docker Swarm

This repository provides a deployment of the Elastic Stack (Elasticsearch, Kibana, Logstash, Beats [Filebeat, Metricbeat], and APM Server) on a Docker Swarm cluster.

Architecture

The architecture of the deployment is illustrated in the provided diagram.

• Elasticsearch Containers: Each Elasticsearch container is bound to a specific swarm node using deployment constraints and labels.
• Logstash and Kibana Containers: These containers can be scheduled on any node, as decided by the swarm managers. You can set the number of replicas for these services using the KIBANA_REPLICAS and LOGSTASH_REPLICAS environment variables in the .env file.
• Metricbeat: Deployed in global mode, ensuring one replica is running on every swarm node.
• Filebeat: Deployed in global mode, ensuring one replica is running on every swarm node. Ships container(elasticsearch, kibana, and logstash) logs to elasticsearch

Additional Notes

• You can scale the swarm cluster beyond the default three nodes. In such cases, you can assign different roles (hot, warm, cold) to Elasticsearch containers running on different swarm nodes. To achieve this, use the compose.extend.yml file to define additional Elasticsearch services such as es04, es05, etc.

Prerequisites

1. Docker Engine (Tested with version 24.09.01)
2. Docker Compose Plugin: (Tested with version 2.27.0)
3. Docker Swarm Cluster: A working cluster with at least three nodes, each properly labeled (Covered in the Deployment section).
   • Setting up Swarm (Follow the official documentation for more info. Getting started with Swarm mode):

     docker swarm init --advertise-addr 10.0.0.1

Environment Variables

To run this project, you will need to add the following environment variables to your .env file. These variables are used in two ways:

1. Variables used only in compose file itself like STACK_VERSION
2. Variables that are also passed to containers like LOGSTASH_WRITER_USER

Variable	Description	Default	Required
IMAGE_REGISTRY_PREFIX	Optinal Image registry prefix (Should end with /) . For example `registry.mycompany.com/'		No
DATA_DIR	This is the directory where all container data is stored. Sub-directories created during setting up the cluster will be bind mounted to containers		Yes
ELK_DIR	This is the full path of this project clone.	/opt/elkswarm	Yes
APM_SERVER_REPLICAS	APM Server service replica count	1	Yes
FILEBEAT_REPLICAS	Filebeat service replica count	3	Yes
KIBANA_REPLICAS	Kibana service replica count	2	Yes
LOGSTASH_REPLICAS	Logstash service replica count	2	Yes
STACK_VERSION	Version of Elastic product. Used in compose.yml image tags	8.13.2	Yes
RUN_AS_USER	The user id used to run the containers	1000	Yes
CLUSTER_NAME	The Elasticsearch cluster name		Yes
ES_PORT	Port to expose Elasticsearch HTTP API on swarm nodes	9200	Yes
KIBANA_PORT	Port to expose Kibana service on swarm nodes	5601	Yes
NFS_SERVER_IP	In case of using shared file system repository for snapshots, this variable is the IP of the NFS server		No
NFS_SERVER_DEVICE_PATH	In case of using shared file system repository for snapshots, this variable is the device path of the NFS server		No
KIBANA_SERVER_PUBLICBASEURL	Public URL of kibana server including port, used as kibana's server.publicBaseURL		Yes
KIBANA_HOSTNAMES	Comma-separated list of Kibana hostnames. Used for generating certificates		Yes
KIBANA_HOSTS_IP	Comma-separated list of Kibana IPs. Used for generating certificates		Yes
ELASTIC_JVM_HEAP_SIZE	Sets Elasticsearch JVM heap size. This value is used as both -Xms and -Xmx to prevent resizing heap at runtime. DO NOT set this value more than half of the available physical memory of the host	2g (GB)	Yes
LOGSTASH_JVM_HEAP_SIZE	Sets Logstash JVM heap size. This value is used as both -Xms and -Xmx to prevent resizing heap at runtime	2g (GB)	Yes
ELASTIC_HOSTS	Space-separated list of Elasticsearch data nodes. Used in Logstash pipelines output In case of the default three node cluster, every node has all the roles	'https://es01:9200 https://es02:9200 https://es03:9200'	Yes
ELASTIC_HOSTS_LIST	List of Elasticsearch nodes	["https://es01:9200","https://es02:9200","https://es03:9200"]	Yes
ELASTIC_MASTER_ROLES	Comma-separated list of elasticsearch master node roles. In case of the default three node cluster, all of the 3 nodes should have all the roles.	master,ingest,ml,remote_cluster_client,data_warm, data_cold,transform,data,data_hot,data_content	Yes
ELASTIC_HOT_ROLES	Comma-separated list of elasticsearch hot node roles. Only used when scaling the cluster to more than 3 nodes in compose.extend.yml	data_hot,data_content,ingest	Yes*
ELASTIC_WARM_ROLES	Comma-separated list of elasticsearch warm node roles. Only used when scaling the cluster to more than 3 nodes in compose.extend.yml	data_warm,ingest	Yes*
LOGSTASH_HOSTS_LIST	List of Logstash nodes		Yes
ELASTIC_PASSWORD	Password for the 'elastic' user (at least 6 characters)		Yes
KIBANA_ENCRYPTEDSAVEDOBJECTS_ENCRYPTIONKEY	Kibana encryption key (must be 32 characters)		Yes
KIBANA_SECURITY_ENCRYPTIONKEY	Kibana encryption key (must be 32 characters)		Yes
APM_SYSTEM_PASSWORD	Password of apm_system user		Yes
BEATS_SYSTEM_PASSWORD	Password of beats_system user		Yes
KIBANA_SYSTEM_PASSWORD	Password of kibana_system user		Yes
LOGSTASH_SYSTEM_PASSWORD	Password of logstash_system user		Yes
REMOTE_MONITORING_USER_PASSWORD	Password of remote_monitoring_user user		Yes
LOGSTASH_WRITER_USER	A user created automatically during initialization. Used by beats, logstash, and apm-server to connect to authenticate with elasticsearch cluster	logstash_writer	Yes
LOGSTASH_WRITER_PASSWORD	Password for the custom 'logstash_writer' user (created automatically during initialization)		Yes
USERS_DEFAULT_PASSWORD	Password for non-system users created automatically during initialization by script elasticsearch/elastic_initial_setup_script.sh		Yes

Deployment

1. Clone this repository

2. Fill the .env file. Then export .env variables in the current shell by using the below command

   while IFS= read -r variable; do export "${variable?}"; done < <(grep -vE '^#|^$' .env)

3. Create mountpoints with the right permission on all of the nodes

   mkdir -p "${DATA_DIR}"/{elasticsearch/{data,snapshot},fleet-server/data,elastic-agent/data,kibana/data,logstash/data,filebeat/data,metricbeat/data} ${ELK_DIR} && \
   chown -R "${RUN_AS_USER}:docker" "${ELK_DIR}" "${DATA_DIR}"/{elasticsearch,logstash,kibana,filebeat} && \
   chmod 775 "${DATA_DIR}" && \
   chmod 777 "${DATA_DIR}"/{elasticsearch,kibana,metricbeat,logstash}/data/;

4. (Optinal) In case of using shared file system repository for snapshots, setup the NFS server and ensure all client nodes can successfully connect and mount the share filesystem. These options are needed for configuring the nfs server to work correctly with docker volumes: (rw,sync,no_subtree_check,no_root_squash). Then uncomment the snapshot-nfs from both volumes and es01 section of the compose.yml file.

5. Update kernel parameters on nodes running Elasticsearch container:

   grep -q 'vm.max_map_count=262144' /etc/sysctl.conf || echo 'vm.max_map_count=262144' >> /etc/sysctl.conf && sysctl --load /etc/sysctl.conf;

6. Properly label each node of the swarm:

   docker node update --label-add name=node1 HOSTNAME1
   docker node update --label-add name=node2 HOSTNAME2
   docker node update --label-add name=node3 HOSTNAME3

7. Generate elasticsearch certificates

   docker compose --env-file .env -f generate-certs-compose.yml up --force-recreate

8. Deploy ELK to swarm as a stack named elk:

   docker stack deploy --resolve-image never -c compose.yml elk

9. See the status of the elk stack

   docker stack ps --no-trunc elk