Upgrade monitoring stack to use Grafana 8.x and maybe 9.x

[npepinpe]

Description

Our SaaS offering recently upgraded their monitoring stack such that they now use Grafana 8.x. In order for us to take advantage of the new features in our own dashboard, our development clusters should also be updated.

The simplest way to do so is to modify the benchmarks/docs/setup/prometheus-values.yaml file.

• Remove the pinned grafana.image.tag

Now, for each cluster, run:

• Follow these instructions
• Run helm upgrade metrics --dry-run --debug --namespace default --dependency-update -f prometheus-operator-values.yml --version 17.0.0 prometheus-community/kube-prometheus-stack
• Inspect the output from above to make sure everything seems OK. The important parts is that the PVC for Prometheus remains the same and we reuse the old data.
• If all looks good, run helm upgrade metrics --debug --namespace default --dependency-update -f prometheus-operator-values.yml --version 17.0.0 prometheus-community/kube-prometheus-stack
• Keep an eye on your pods (kubectl get pods -w) and make sure everything comes up green.

Do this for:

• zeebe-cluster
• zeebe-long-running

Once done, open a PR with the modified values and get it merge.

[megglos]

Triage: should be a long hanging fruit,

[npepinpe]

I would propose to upgrade sooner than later to Grafana 9 actually, our SaaS environment is already on Grafana 9.

[menski]

@deepthidevaki as medic you can try to achieve that next week, with the help of @npepinpe