elastic-agent docker: support healthcheck for the container

[mtojek]

With the Fleet Server enabled in the agent's Docker container, we need to find a way to signal that the container is healthy. Before 7.13.0-SNAPSHOT we used the following healthcheck: https://github.com/elastic/elastic-package/blob/master/internal/install/static_snapshot_yml.go#L85

Do you have any recommendation how to signal that the container is healthy - it has a default policy assigned? Do you think you can add the healthcheck definition to the official Docker image?

[elasticmachine]

Pinging @elastic/agent (Team:Agent)

[ruflin]

@simitt How are you handling this for the Cloud container?

[simitt]

@ruflin no special handling for the healthcheck yet; instead, if the legacy APM Server cannot be started or dies, it sends a signal to terminate the Elastic Agent, resulting in the whole container being terminated.

For sub processes managed by Elastic Agent, we have discussed in the past that the Elastic Agent should provide a healthcheck endpoint providing details per sub process but also an overall health indicator.

[ruflin]

@michalpristas @ph Do we have this overall healthcheck already tracked somewhere? I remember we discussed this in the past. Does Agent already has some http endpoint or similar?

[ph]

We track health status internally and we should be able to expose it in any way necessary, @ruflin I believe this is linked to the #24091

[mtojek]

From our perspective (users) it's valuable if the healthcheck signals green if the default policy is assigned for the first time.

[ruflin]

I think an Agent should be healthy, as soon as the first policy is received and acked. This does not have to be the default policy.

We should improve this healthcheck later on to have more fine grained status information depending on processes / inputs status.

[simitt]

IMO if the agent is started in Fleet Server mode, then the Agent's health should consider the Fleet Server's health, ideally by consuming a health endpoint from the Fleet Server.

[mtojek]

As we're observing some flakiness with booting the agent I did a short exercise to check responses fro /api/status:

$ for I in `seq 1 1 10000`; do curl http://localhost:8220/api/status ; sleep 1; done
curl: (52) Empty reply from server
curl: (52) Empty reply from server
curl: (52) Empty reply from server
curl: (52) Empty reply from server
curl: (52) Empty reply from server
curl: (52) Empty reply from server
curl: (52) Empty reply from server
curl: (52) Empty reply from server
curl: (52) Empty reply from server
curl: (52) Empty reply from server
curl: (52) Empty reply from server
curl: (52) Empty reply from server
curl: (52) Empty reply from server
curl: (52) Empty reply from server
curl: (52) Empty reply from server
{"name":"fleet-server","version":"","status":"STARTING"}{"name":"fleet-server","version":"","status":"HEALTHY"}curl: (52) Empty reply from server
curl: (52) Empty reply from server
curl: (52) Empty reply from server
{"name":"fleet-server","version":"","status":"HEALTHY"}{"name":"fleet-server","version":"","status":"HEALTHY"}{"name":"fleet-server","version":"","status":"HEALTHY"}{"name":"fleet-server","version":"","status":"HEALTHY"}{"name":"fleet-server","version":"","status":"HEALTHY"}{"name":"fleet-server","version":"","status":"HEALTHY"}{"name":"fleet-server","version":"","status":"HEALTHY"}{"name":"fleet-server","version":"","status":"HEALTHY"}{"name":"fleet-server","version":"","status":"HEALTHY"}{"name":"fleet-server","version":"","status":"HEALTHY"}{"name":"fleet-server","version":"","status":"HEALTHY"}{"name":"fleet-server","version":"","status":"HEALTHY"}{"name":"fleet-server","version":"","status":"HEALTHY"}{"name":"fleet-server","version":"","status":"HEALTHY"}{"name":"fleet-server","version":"","status":"HEALTHY"}{"name":"fleet-server","version":"","status":"HEALTHY"}{"name":"fleet-server","version":"","status":"HEALTHY"}{"name":"fleet-server","version":"","status":"HEALTHY"}{"name":"fleet-server","version":"","status":"HEALTHY"}{"name":"fleet-server","version":"","status":"HEALTHY"}{"name":"fleet-server","version":"","status":"HEALTHY"}{"name":"fleet-server","version":"","status":"HEALTHY"}{"name":"fleet-server","version":"","status":"HEALTHY"}{"name":"fleet-server","version":"","status":"HEALTHY"}{"name":"fleet-server","version":"","status":"HEALTHY"}ć{"name":"fleet-server","version":"","status":"HEALTHY"}

Please mind the gap between HEALTHY states. It seems that the Fleet Server got restarted then which means that we need a different workaround :)