[ENH] - Setup uptime monitoring for nebari deployment

[aktech]

Feature description

Uptime monitoring for Nebari and all services.

Value and/or benefit

• Know general health of system

Anything else?

No response

[dcmcand]

Kuberhealthy https://github.com/kuberhealthy/kuberhealthy looks like it might be a promising tool for this. However, I think this should be a meta issue. There are different meanings to up for different services. I think we should create a task that is part of the meta issue for each service that we want to monitor

[dcmcand]

@aktech, @kcpevey mentioned that you had thoughts about just using Prometheus for an initial implementation. What metrics were you thinking of using there?

[aktech]

I was thinking we can add something like the blackbox_exporter in prometheus, to export uptime metrics for all the services and then create a dashboard in Grafana for the same.

Here is an example configuration file for the blackbox_exporter: https://github.com/prometheus/blackbox_exporter/blob/3dd5dfeaabc630ca0c2ec722a07f9755159ae0dd/example.yml

[dharhas]

@dcmcand does using the k8s python client instead of tf for kuberhealthy have any implications related to cross cloud compatibility. Any other pros/cons?

[dcmcand]

closed by #2667