[Optional Spike: 2hrs] Investigate prometheus monitoring and grafana alerting for EBS volumes approaching full

[sgibson91]

Warning

To be pursued if #5005 fails

Tasks

• Investigate prometheus metrics to track the size of EBS volumes, if deployed
• Investigate grafana alerting capabilities
• Report on learnings (can be a comment on this issue)
• Scope and refine future tasks in the associated epic

Definition of Done

• We understand what is required to provide alerting on EBS volume size from a prometheus and grafana perspective
• Tasks in [EPIC] [devseed] Set up alerting for an EBS volume getting close to full on AWS, and a runbook on how to resize it when needed #5004 have been refined

[sunu]

Yet to test this end-to-end, but I think we should be able to reuse the Prometheus metric exporter set up defined in https://github.com/2i2c-org/infrastructure/blob/main/helm-charts/basehub/templates/home-space-reporter.yaml by mounting the whole NFS disk as a volume mount.

[sgibson91]

Next steps defined in #5062