[Monitoring] Document the central Grafana

[GeorgianaElena]

Context

We can now monitor the 2i2c clusters at https://grafana.pilot.2i2c.cloud, also called "the central grafana". However, there is little info about this in the docs https://infrastructure.2i2c.org/en/latest/howto/operate/grafana.html?highlight=central%20grafana#the-central-grafana

Proposal

I propose adding more info about this central Grafana to our docs.

Updates and actions

No response

[colliand]

Can I please have access to the central Grafana dashboard? I am not certain how to associate my credentials.

[GeorgianaElena]

@colliand, unfortunately until #1437 gets sorted out, getting access to the Grafana means being able to decrypt the file at https://github.com/2i2c-org/infrastructure/blob/master/helm-charts/support/enc-support.secret.values.yaml and get the admin password from there.

We have docs on how to do this here and it implies setting up sopsand gcloud per instructions in this section.

Someone in @2i2c-org/tech-team would also need to add you as an owner of the two-eye-two-see gcloud project, since this is where the encryption key is stored and I don't see your address being listed there. Is colliand@2i2c.org the correct address that you'd like to use to access the project above?

I would also like to signal-boost your request to @2i2c-org/2i2c-team, in case anyone can think of a quicker way of giving you access to the central grafana credentials ✨ Thank you!

[GeorgianaElena]

I opened #1445 to discuss access to central grafana and to signal-boost the request

[choldgraf]

A quick thought from me, I think that we should put the "how to log in / use / etc" information in a different place than https://infrastructure.2i2c.org/en/latest/howto/operate/grafana.html . The information on that page is very engineering-heavy, and talks about things like Kubernetes, Helm Charts, etc. We want the documentation about how to use and understand this Grafana to be discoverable for non-engineers.

Maybe the easiest path forward would be to create a Topic Guide like "Monitoring and reporting", and add a page under this section like "Grafana Dashboards"...this could cross-link to the "how to deploy with Kubernetes" docs. I'm not positive on that though - if others think there's a better spot for it, please suggest!

[GeorgianaElena]

Actually, thinking about it now, how about instead we create these docs upstream, in jupyterhub/grafana-dashboards and reference it from 2i2c?

[choldgraf]

That works for me! I think that the 2i2c docs should have a brief and discoverable section that can explain what this is in general, and cross link to the upstream for more details like how to understand the graphs