Setup alerts on the MPC service

[DavidM-D]

We need to keep track of

• Whether the nodes are up
• Whether we have an elevated set of errors

Us and SRE need to be alerted if these things change and we need to design a checklist of things to do when something goes wrong.

Tasks

Preview Give feedback

1. Set up Prometheus to scrape, collect and monitor metrics
2. Set up Alertmanager to dispatch the alerts generated by Prometheus
3. Create Grafana board to display the metrics
4. Create Alerts

[volovyks]

Blocked on SRE side. We need to get access to Grafana for the setup.

[DavidM-D]

I guess this is unblocked now?

[itegulov]

Yep, unblocked and in progress

[itegulov]

Blocked again, waiting for SRE to add a notification policy for us