[EPIC] [devseed] Set up alerting for an EBS volume getting close to full on AWS, and a runbook on how to resize it when needed

[sgibson91]

Context

Previously using AWS EFS instances for home directories, they would be automatically resized when they got full. However, to provide home directory quotas, we had to migrate to EBS volumes which do not automatically expand. Therefore we need to develop a monitoring and alerting system that let's us know that these volumes are approaching full in good time to act before the service is disrupted.

Task list

Tasks

Beta Give feedback

1. [Spike: 2hrs] Investigate AWS alerting systems for EBS volumes and associated terraform config #5005
   +0
   
2. [Optional Spike: 2hrs] Investigate prometheus monitoring and grafana alerting for EBS volumes approaching full #5006
   +0
   
3. Enable disk size alerting for AWS EBS volumes on VEDA staging hub #5062
   4 of 6
   +0
   

Definition of Done

No response