Add a /-/healthy endpoint for monitoring component health

[ptodev]

PR Description

Adding a /-/healthy endpoint which returns an error if at least one component is not healthy. The /-/ready endpoint doesn't check for component health, so this could be useful for Kubernetes liveliness probes.

Which issue(s) this PR fixes

Fixes #2061

Notes to the Reviewer

Prometheus also has a /-/healthy endpoint, and so does Grafana Agent Static mode. However, both of these were not very useful because they always return HTTP 200. There is an open issue in the Prometheus repo to make it more useful.

PR Checklist

• CHANGELOG.md updated
• Documentation added
• Tests updated
• Config converters updated

[thampiotr]

Looks nice, thanks! Just minor comments.

[clayton-cornell]

I added a few suggestions here to expand an clarify some doc things. :-)

[rfratto]

Hi, I wanted to share some concerns with using component health for liveness probes.

Kubernetes intends for liveness probes to be an indication of when the pod is "stuck" and needs to be killed. Kubernetes will eventually kill the pod if the liveness probe fails enough times.

With this definition of liveness, component health is not a good indicator. Components like remote.s3 can report themselves as unhealthy if an object in S3 gets deleted, and custom components can fail if you give them an invalid configuration (such as someone rolls out a bad config from fleet management).

Killing the pod when components are unhealthy will not only not fix the problem, but since Alloy requires all components to be healthy on startup, it will also completely halt all telemetry collection until the problem is manually fixed.

[thampiotr]

Hi, I wanted to share some concerns with using component health for liveness probes.

Oh yeah, I 100% agree and I would not recommend using it as liveness probe. But people still wanted it. I think adding this endpoint is harmless, because it can be used for other health checking purposes or diagnostics, not just as a liveness probe. We do know people who wanted to use it as liveness probe, but this would be doing it against our advice. They may have some special requirements though and it may work okay for them.

[ptodev]

Hi @rfratto, thank you for opining! I agree that in the component health is not a good indicator for whether Alloy as a whole should be restarted.

Apparently, the issue which originally prompted this feature request is an informer timeout like the one referenced in #2161.
@thampiotr - if that timeout does happen, maybe the real issue is that the component should keep retrying to sync the informers rather than giving up? Judging by the code, the prometheus.operator components will only sync the informers when the config is updated?

I do think there is a benefit in the /healhy endpoint as a whole, although I do agree that we shouldn't recommend that users use it as a liveliness probe. It could be useful in situations where users update the Alloy config and then want to double check that it works as expected. They might do rolling deployments - e.g. 1% of Alloy instances first, and then if they are healthy, the deployment can continue to more instances. I mistakenly thought that liveliness probes in k8s can do similar things, but apparently that's not the case.

[ptodev]

I opened another PR for the k8s informers to keep retrying, as mentioned in the comment above.

[clayton-cornell]

Minor tweaks to the docs and all good