[Elastic Agent] fork/exec from within STATE_PATH incompatible with K8s hostPath volumes

[pebrc]

#24817 introduced a new predictable directory for the state of the Elastic Agent container. However it also executes the programs it runs from within that state directory.

As mentioned in #24160 on Kubernetes one common way to run Elastic Agent is as a DaemonSet. A DeamonSet cannot use persistent volumes to persist the Agent's state across container restarts. So to maintain state across upgrades or reconfigurations of the container ECK uses a hostPath volume. HostPath volumes however have a noexec restriction. Agents using a hostPath volume will therefore never be able to successfully run their configured programs.

2021-04-26T19:14:43.019Z    ERROR    log/reporter.go:36    2021-04-26T19:14:43Z: type: 'ERROR': sub_type: 'FAILED' message: Application: metricbeat--7.13.0-SNAPSHOT[ │
│ c3de84c-1dc6-41d7-9639-46957471a06d[]: State changed to FAILED: failed to start '/usr/share/elastic-agent/state/data/install/metricbeat-7.13.0-SNAPSHOT-linux-x86_64/ │
│ metricbeat': fork/exec /usr/share/elastic-agent/state/data/install/metricbeat-7.13.0-SNAPSHOT-linux-x86_64/metricbeat: permission denied

Unless we say DaemonSets are not a supported way of running Agent on Kubernetes I am thinking that it would be good to distinguish the different kinds of state agent uses:

• derivative state: IIUC builds are copied into the state directory to avoid downloading them again. However losing this state would not affect the correct operation of the Agent. Agent configuration dictates which binaries are required. It would just be inefficient to re-download the binaries. This should be a separate state location without the noexec restriction
• essential state: I am assuming that ingest state for example for a Filebeat process is essential if we want to avoid double ingesting the same log files and should be inside the state directory that is persisted across restarts.

[elasticmachine]

Pinging @elastic/agent (Team:Agent)

[pebrc]

I realise I have not tried working around this issue with agent.download.install_path will reopen this if needed. Apologies for the noise.

[pebrc]

Tweaking agent.download.install_path and agent.download.target_directory does not help as agent still tries to copy/link into the state directory:

│ 2021-04-27T08:37:21.476Z    ERROR    log/reporter.go:36    2021-04-27T08:37:21Z: type: 'ERROR': sub_type: 'FAILED' message: Application: metricbeat--7.13.0-SNAPSHOT[ce414fa1-4d5b-4b1b-9da4-ec3d51c13f1c]: State changed to FAILED: rename /usr/share/elastic-agent/state/data/tmp/elastic-agent-install181662629/metricbeat-7.13.0-SNAPSHOT-lin │
│ ux-x86_64 /usr/share/elastic-agent/data/elastic-agent-a5c2d3/install/metricbeat-7.13.0-SNAPSHOT-linux-x86_64: invalid cross-device link

[ph]

@urso @blakerouse Could we fix that for 7.14?

[ruflin]

We should spend some time on rethinking all the paths we need for the different environments. From my perspective we should find ways to support the above.

[blakerouse]

We could possibly place the download directory in STATE_PATH and then the install directory inside the container. The run directory would need to live in the STATE_PATH so the run time state of the beat is kept across restarts.

Should be do-able in 7.14.