Monitoring of cluster stats via grafana and prometheus

[costrouc]

No description provided.

[laisbsc]

According to @costrouc : this is a fun issue!

It is useful for demos, a cool to have. Not sure when the team will be able to implement it.

[costrouc]

Acceptance Criteria:

• grafana and prometheus deployment in Qhub
• pre made dashboards in grafana for viewing:
  • jupyterhub health
  • kuberenetes pods
  • traefik health and traffic

[Adam-D-Lewis]

I'll try to work on this if no one else is planning on working on it at the moment. I would add documentation to the acceptance criteria as well.

Acceptance Criteria:

• grafana and prometheus deployment in Qhub
• pre made dashboards in grafana for viewing:
  • jupyterhub health
  • kuberenetes pods
  • traefik health and traffic
• documentation detailing how to access prometheus and grafana.

[Adam-D-Lewis]

I'll probably add these via helm charts similarly to ClearML.

• https://github.com/grafana/helm-charts
• https://github.com/prometheus-community/helm-charts
• https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack

[costrouc]

@Adam-D-Lewis this would be really useful for the 0.4 release and would be awesome if you worked on it. We've needed this feature for awhile!

[Adam-D-Lewis]

To update my progress, I was able to deploy the grafana helm chart manually via helm install ..., but I'm still working on setting up some traefik routes to access grafana.

[Adam-D-Lewis]

To provide further updates, with Chris's help I was able to access grafana at grafana.myqhub.qhub.dev, but was not able to access grafana at myqhub.qhub.dev/monitoring/. Also, in consultation with Chris, he advised me we could either strip the /monitoring prefix with a middleware or we can let grafana know its baseUrl similar to how it's done at https://github.com/Quansight/qhub-hpc/blob/main/templates/grafana.ini.j2#L54. The preference is on doing the latter which needs more research on my end to see if the helm chart exposes the grafana.ini somehow.

[brl0]

Hey @Adam-D-Lewis, hope all is well. :)

I am definitely +1 on this idea and am looking forward to trying it out.

I don't know if this is particularly helpful, but I thought it was an interesting project and at least tangentially related: https://github.com/yuvipanda/jupyterhub-grafana

This project provides some standard Grafana Dashboards as Code

[brl0]

Incidentally, it looks like the project I mentioned above just got moved to the jupyterhub organization:
https://github.com/jupyterhub/grafana-dashboards

[costrouc]

@brl0 yeup we saw that and certainly want to use that! We'll also be tying in additional stats from other services e.g. traefic

[Adam-D-Lewis]

Thanks for the references @brl0, hoping we can tie those in eventually. I'm working through some Traefik/grafana routing issues at the moment.

[Adam-D-Lewis]

To provide further updates, with Chris's help I was able to access grafana at grafana.myqhub.qhub.dev, but was not able to access grafana at myqhub.qhub.dev/monitoring/. Also, in consultation with Chris, he advised me we could either strip the /monitoring prefix with a middleware or we can let grafana know its baseUrl similar to how it's done at https://github.com/Quansight/qhub-hpc/blob/main/templates/grafana.ini.j2#L54. The preference is on doing the latter which needs more research on my end to see if the helm chart exposes the grafana.ini somehow.

I've specified the baseUrl in the grafana config, but I'm getting an infinite loop of redirects when navigating to https://github-actions.qhub.dev/monitoring/login (local deployment) rather than the grafana login page being loaded. I'm continuing to work on this.

[brl0]

Another resource that may or may not be helpful... I stumbled across NeuroHackademy's JupyterHub which deploys prometheus and grafana via helm dependencies.

[Adam-D-Lewis]

I still am hitting the infinite loop of redirects in grafana mentioned in my last comment. I want to try out using traefik ingress instead of traefik ingressroute and see if that helps. If not, I may need to just open an issue on grafana's repo and then use the subdomain approach (monitoring.myqhub.qhub.dev) to access grafana instead of prefix paths (myqhub.qhub.dev/monitoring).

[Adam-D-Lewis]

Finally got past the infinite loop of redirects. I had to add a middleware to strip the path prefix, and then it worked.