Add /liveness endpoint to elastic-agent

[fearful-symmetry]

What does this PR do?

Closes #390

Why is it important?

This adds a /liveness endpoint to elastic agent that we enable along with the processes endpoint. However, unlike the /processes endpoint, liveness will return a 500 is any components are in an Unhealthy or Degraded state.

This is a pretty simple change as-is, since I figured it would be easier to discuss other theoretical changes when we can actually see code. In particular:

1. Should this be toggled by something other than the monitoring.http flag? Should we enabled it based on some k8s autodiscover config or env variable or something else? If this feature is fundamentally tied to a k8s workflow, it seems like enabling it should also be tied to k8s in some way.
2. Should we fail the health check based on any other conditions?

I'm also holding off on changing any docs until we're sure we have what we want with regards to config.

How to test this PR

• set monitoring.http.enabled to true
• Check /liveness/ and /liveness/[component-id] using either curl or a k8s health check.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in ./changelog/fragments using the changelog tool
• I have added an integration test or an E2E test

[elasticmachine]

Pinging @elastic/elastic-agent (Team:Elastic-Agent)

[mergify]

This pull request does not have a backport label. Could you fix it @fearful-symmetry? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-v./d./d./d is the label to automatically backport to the 8./d branch. /d is the digit

NOTE: backport-skip has been added to this pull request.

[pchila]

Code looks ok, just a couple of questions about when we want to fail the liveness probe and copying potentially big structures in memory

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b liveness-endpoint upstream/liveness-endpoint
git merge upstream/main
git push upstream liveness-endpoint

[fearful-symmetry]

So, I'm still doing manual testing, but updating this as-is while I figure out how integration tests should work.

[leehinman]

question on this. This looks like we will put something on the channel as soon as possible, so if the coordinator gets blocked after that, you would read from the heartbeatChan and think that it was up, because in the past it had been able to write to the channel. On your next read it would fail. Is that correct?

If it is, I'd rather see something that records a timestamp every time runLoopIteration is called, then we can check to see if that timestamp is within our timeout window.

[fearful-symmetry]

so if the coordinator gets blocked after that, you would read from the heartbeatChan and think that it was up, because in the past it had been able to write to the channel. On your next read it would fail. Is that correct?

So, if I understand you correctly, yes. We could end up in a state where the coordinator blocks right after a heartbeat call. Because the liveness endpoint is meant to be repeated on some kind of regular period, I'm not too worried about that.

I specially didn't go with some kind of timestamp mechanism on @faec 's advice, since she was worried about the added complexity/edge cases of a time comparison, compared to a simple signal like this.

[leehinman]

I think this just tells us that the runLoopIteration function can run and the select can hit the heartbeat case. It doesn't tell us if the coordinator can/has processed any of the other cases.

I was thinking to be "alive", we want to know that one of the case statements besides heartbeat statement has run. We would probably need to bound that comparison with the check-in Interval. Or to put another way, the runLoopIteration function should happen at least every check-in Interval (2x is probably safer) and it might happen more frequently.

[fearful-symmetry]

Yeah, I sort of agree, this only really works as a basic heartbeat.

My concern is that "has the coordinator done anything else?" would be a bit flaky, as we can't really guarantee what state other sub-components will be in at any given time. @faec can comment more, but for that to work we might need to refactor part of the coordinator to allow for a more sophisticated health check.

[cmacknz]

Mostly LGTM, a couple of minor things in particular the changelog has a typo.

[cmacknz]

Am I blind or is this test just starting the server and not actually reloading anything?

[fearful-symmetry]

Yeah, the name is a tad deceptive, the integration test is the only one doing the reloading.

[elastic-sonarqube]

Quality Gate passed

The SonarQube Quality Gate passed, but some issues were introduced.

2 New issues
0 Security Hotspots
68.3% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

[cmacknz]

Thanks, sanity checked in a local kind cluster to make sure it interoperates with k8s as expected.

Using

          livenessProbe:
            httpGet:
              path: /liveness
              port: 6792
            initialDelaySeconds: 3
            periodSeconds: 3

With a Fleet managed agent configured with:

PUT kbn:/api/fleet/agent_policies/3de09e84-54ec-479e-851b-f4947ff95262
{
  "name": "Policy",
  "namespace": "default",
  "overrides": {
      "agent": {
        "monitoring": {
          "http": {
            "enabled":true,
            "host": "0.0.0.0",
            "port": 6792
          }
        }
      }
  }
}

Works as expected.

[rgarcia89]

@cmacknz is there also a possibility to enable monitoring via the UI of the agent policies?

[cmacknz]

Logs and Metrics monitoring has always been possible to configure in the UI.

This PR makes it possible to configure the agent to expose the liveness endpoint via the UI, but what polls the /liveness endpoint is deployment specific and Fleet itself can't/won't monitor it since it is local to the machine the agent is on.

[rgarcia89]

Thanks I am well aware about the functionality of the liveness and readiness probes. However adding the probe definition to the daemonset is not enough. By default the agents are not listening on the port which is being used in your snipped. Therefore, I am trying to figure out where to enable the exposure of it, so that it can be used for the probing. I might have overseen the section in the UI to enable it. Could you maybe provide a screenshot?

PS: I am talking about fleet managed agents.

[cmacknz]

The first problem you have is that the code in this PR is not released yet, and won't be until 8.15.0. The steps in #4499 (review) won't work until then.

The second thing with a Fleet managed agent prior to 8.15.0 is that the agent.monitoring.http settings need to be set in the local configuration when the agent container first starts up. This is because you can't change them from Fleet yet. Following the process in https://www.elastic.co/guide/en/fleet/current/advanced-kubernetes-managed-by-fleet.html to modify the initial contents of the elastic-agent.yml should get you around this problem, but the /liveness endpoint wont' exist until 8.15.0.

The third thing is that you likely need to change the default host the monitoring server binds to to 0.0.0.0 from localhost for the liveness probe to work.