[DOCS] Replace Docker Compose files and setup

[lockewritesdocs]

Current state

In the current documentation for starting a multi-node cluster with Docker Compose, we use the following files:

• instances.yml file to outline the nodes that will be created
• .env file to hold environment variables
• create-certs.yml file to generate security certificates
• docker-compose.yml file for starting a multi-node cluster with TLS configured

@bytebilly created a docker-compose.yml file that incorporates the certificate generation scripts from create-certs.yml, and eliminates the need for that file and instances.yml. Also, users do not need to complete the enrollment flow for Kibana because the updated configuration handles this task.

Proposed solution

We would like to replace the current files in the documentation with this updated docker-compose.yml file, which requires only a single command to set up and provides a much more streamlined UX. I'm pinging a few teams for input so that we can determine if this is the optimal solution going forward.

I created a gist that contains the updated .env and docker-compose.yml files. To run this setup:

1. Make sure that you have at least 4GB of memory allocated in Docker.
2. Copy both files into a single directory.
3. Change to the directory, modify .env, and enter a value for ELASTIC_PASSWORD and KIBANA_PASSWORD.
4. Start the Docker containers with docker compose up -d.

The included healthchecks in the updated docker-compose.yml file should also resolve the issues in elastic/stack-docs#1846.

[elasticmachine]

Pinging @elastic/es-docs (Team:Docs)

[elasticmachine]

Pinging @elastic/es-delivery (Team:Delivery)

[elasticmachine]

Pinging @elastic/es-core-infra (Team:Core/Infra)

[elasticmachine]

Pinging @elastic/es-security (Team:Security)

[mark-vieira]

I assumed with auto-security all this certificate generation stuff would be unnecessary now. Why would we still have to manually generate certs?

[jkakavas]

I assumed with auto-security all this certificate generation stuff would be unnecessary now. Why would we still have to manually generate certs?

Docker compose, along with other use cases where there is orchestration, has been out of scope on purpose for Security on by default, as we believe that in these use cases it makes sense to take care of certificate generation and configuration on the orchestration layer.

The devised enrollment process for security on by default that is the basis of TLS auto-configuration for multi-node clusters assumes a set of manual actions in sequence and communication between joining and existing nodes and doesn't fit well with docker compose.

[bytebilly]

Yes, we are considering future use of the enrollment flow in Docker Compose scenarios, but it's not part of this effort.
The file has been tested on Linux, Windows, and macOS. Anyone is welcome to test more!

Maybe @mgreau has feedback too.

[bytebilly]

@lockewritesdocs in the gist I see that you modified the .env file to set placeholders for <password>.
This is a major concern, as the check in the compose file doesn't work anymore and people could actually start the deployment with a password set to the default verbatim<password>.

We don't want to provide a default password intentionally, to be sure that users set their own.

If you strongly feel that we need that placeholder, we should modify checks so that the specific verbatim is not allowed.

Another note is that the STACK_VERSION variable should be dinamically set based on the documentation version, if possible.

[lockewritesdocs]

in the gist I see that you modified the .env file to set placeholders for .

@bytebilly, yes, this is just in the gist. I only included it there, but would leave the passwords blank in the actual files we put in our docs so that the password checks you included will work if users try to start Docker Compose without a password in the .env file.

Another note is that the STACK_VERSION variable should be dinamically set based on the documentation version, if possible.

We can certainly add a variable for the STACK_VERSION 👍

[bytebilly]

Perfect, thanks!
FYI I did some other minor improvement to the script, probably the easiest way is that you ping me in the PR and we can check which is the best version to include.

[lockewritesdocs]

@bytebilly, I opened this PR to update the configuration: elastic/stack-docs#1906