Prometheus metrics for Kubernetes components

[anton-johansson]

Note: Technically not related to this repository other than the fact that I might need custom switches on other components than kube-apiserver. But I'll give it a go here anyway, maybe it's a good discussion topic. :)

I noticed that all Kubernetes components and etcd exposes a /metrics path with Prometheus metrics. So I was thinking that I should start scraping these, and see if I can find any pre-built dashboards for Grafana.

I just have something to ask/discuss here.

kube-apiserver should easily be accessible by my Prometheus pod, as long as I give the serviceaccount access to the /metrics path (not sure how I do that, though, will need to investigate).

Regarding kube-scheduler and kube-controller-manager, I can access them over HTTP on ports 10259 and 10257 respectively. However, they have quite some strange CA certificates and I'm not able to use my own access token. I suppose switches --tls-cert-file and --tls-private-key-file will solve the strange CA certificate, but I'm not sure how to actually authenticate (avoiding 401 Unauthorized). Do you any ideas?

When it comes to etcd, I can access that pretty easily. However, I need to use the client certificate and key stored on the masters (etcd.pem and etcd-key.pem), and I can't really access them from my Prometheus pod. I'm not sure I want to either. I guess this is something that is interesting here.

kube-proxy should be fairly simple. It only listens to 127.0.0.1:10249 by defualt, but that's changable with a switch, so it should be fine.

Finally: I wouldn't want to hardcode all server IPs in my Prometheus configuration file. It would be great if I could use Kubernetes services for this. I see that I have some endpoints (kubectl get endpoints -n kube-system), like kube-controller-manager, but they're set to <none>. I guess I could create my services manually (once) and utilize them. But I wouldn't want Prometheus to round-robin requests to them. I would want it to perform a DNS-lookup and scrape all targets of that DNS-lookup. Somehow... :) Ideas? For the worker nodes, it would be nice if I could utilize kubectl get nodes to find IP adresses of nodes, and there reach kube-proxy.

Just close this if you feel it's too off-topic, and I'll try elsewhere.

[amimof]

Prometheus has a very nice Kubernetes auto discovery feature which will use in-cluster credentials using generated pod tokens. I usually use this as a base to deploy Prometheus and get most cluster metrics out.

[anton-johansson]

Oh, that looks very interesting! I was already using the auto-discovery for pods with the scaping annotations, I did not know that it could autodiscover other things too.

Thanks a bunch for this!

[anton-johansson]

By the way, @amimof: Do you know of any good "complete" dashboards for these components available on https://grafana.com/dashboards, or have you created your own?

[amimof]

I either create my own or use others on grafana.com as a base. One great dashboard to use with the node-exporter is https://grafana.com/dashboards/3320

[anton-johansson]

Great, thanks a bunch!