Fix our Grafana / Prometheus data connection

[choldgraf]

Description

Our Grafana dashboards are currently broken because of a regression in the grafana-dashboards project. Our Grafana dashboards will be broken until this is resolved.

Benefit

This benefits / impacts two important groups of users:

• Hub engineers use Grafana as a first point of contact to understand what is going on with the infrastructure, and spot problems
• Hub administrators use Grafana to get a more user-friendly understanding of the infrastructure that is more well-matched to their expertise.

This is particularly useful for hubs with admins that are more knowledgeable about operations, since they'll care about things like this (so, Pangeo + CarbonPlan at least)

Implementation

This is technically an issue in the grafana-dashboards support service (jupyterhub/grafana-dashboards#30), but we might be able to get around it by just downgrading Prometheus first. We'll try that and see if it works as a stop-gap.

Instructions for deploying Grafana to our clusters might be helpful.

Tasks to complete

• Pin Prometheus to the previous major version, and see if that gives us the data that we need for our Grafana dashboards to work.

[choldgraf]

I was just doing the Statement of Work for the new Pangeo sub-award and realized that this is a deliverable on there too (about creating reporting dashboards that show off hub activity).