[Elastic Agent] Allow separation of runtime state location and installation and executable location.

[pebrc]

We ran into an issue on Kubernetes with ECK and Elastic Agent. The released version of ECK has a bug in that it does not use a hostPath volume we configure for the Elastic Agent data directory. The intention behind using a hostPath volume was to create a location for the runtime state of Agent to be persistent across container restarts. Fixing this issue has surfaced a problem with the planned approach in that we cannot just configure --path.home to be the hostPath volume on Kubernetes because Elastic Agent installs the programs it wants to run into that filesystem and tries to execute them from there as well but hostPath volumes are mounted noexec on Kubernetes. Running Elastic Agent as DaemonSet, as probably many users will want to do, means there is no good way to use persistent volumes as an alternative, which might not have the noexec restriction.

The current filesystem structure for the Elastic Agent Docker container seems to be roughly:

/usr/share/elastic-agent => home --path.home
/usr/share/elastic-agent/data => data directory derived from home
/usr/share/elastic-agent/data/elastic-agent-${hash}/run => runtime state derived from home
/usr/share/elastic-agent/data/elastic-agent-${hash}/install => install location for the programs to execute agent.download.install_path
/usr/share/elastic-agent/data/elastic-agent-${hash}/download => download location for packages to install agent.download.target_directory

If we had a way to separately configure the directory for the runtime state (presumably everything below run) that would allow us to sidestep this issue. The fact that the hash of the Elastic Agent build is part of the directory structure makes it very hard for an orchestrator to mount something on top of the directory without knowing that hash upfront or having such an explicit config option.

But even with the ability to configure a separate directory for runtime state, it would still be problematic that Elastic Agent would download and execute the packages in the filesystem of the Docker container. But I don't see a good way of addressing this other than effectively reimplementing a DaemonSet in the ECK operator and using persistent volumes that are not hostPath based.

[elasticmachine]

Pinging @elastic/ingest-management (Team:Ingest Management)

[blakerouse]

This is actually a core issue that we need to solve for the K8s enablement for Elastic Agent. I linked this to the overall meta issue. #23613.

[elasticmachine]

Pinging @elastic/agent (Team:Agent)

[blakerouse]

@ph I believe this needs to be done for 7.13.

[ph]

@blakerouse I have added to 7.13 iteration, can you add bit of details what need to changed in the Agent architecture?

[blakerouse]

@ph The bug description gives a good break down of the issue and the paths. I would need to dig into more on a proper way of fixing this.

I think I will pick this one, up because I believe that the new container subcommand I added will be the best place to handle this.

[ph]

Thanks @blakerouse I was missing context to assign it. thanks for grabbing it.

[simitt]

++ on the requirements for the cloud setup.
Created #24271 for a similar issue with the logs path configuration.