Allow monitoring of etcd when etcd_deployment_type=host

[sathieu]

What would you like to be added:

Currently, monitoring etcd is very hard for the following reasons:

• etcd is listening on https with client cert verification
• the keys are rotated
• etcd endpoints are not known inside the cluster

Why is this needed:

A production cluster needs etcd to be properly monitored.

See also https://gitlab.com/kubitus-project/kubitus-installer/-/issues/13.

Proposal:

A possible way forward would be to expose the etcd metrics endpoint with a static pod on each etcd nodes (similar to /etc/kubernetes/manifests/nginx-proxy.yml).

[maxpain]

The same problem. I switched container_manager from docker to containerd and can't monitor the etcd anymore.

[sathieu]

@maxpain How was monitoring working with container_manager=docker?

[maxpain]

@maxpain How was monitoring working with container_manager=docker?

@sathieu Everything was working fine. etcd in docker can be found by Prometheus via service discovery, but it is impossible with etcd, running as a host process (not in a container).

[sathieu]

@maxpain Reading the code, I couldn't find how this was working.

With etcd_deployment_type=docker, a raw container is created, and not a static Pod. A static pod is only created when etcd_kubeadm_enabled=true. A raw container is not seen in the API.

etcd_kubeadm_enabled is still experimental and has known bugs (#7667, #7765).

Another thing: etcd as installed by kubespray uses server and client certificates, this makes /metrics API unreachable.

[nikitka]

@sathieu You can define etcd_metrics_port and scrape metrics from it without TLS.

[sathieu]

@nikitka Thanks 🙏! Now I only need a way to create endpoints or endpointslices!

Ref: etcd_metrics_port implemented in #6092

[floryut]

Closing then, thanks @nikitka

[sathieu]

@floryut Please reopen, only half of the answer is implemented. I'll write a PR for the second half (endpoints).

[sathieu]

Thanks for reopening @floryut. PR proposed in #8203.