Enable disk size alerting for AWS EBS volumes on VEDA staging hub

[sgibson91]

Context

• Default JupyterHub dashboards live in https://github.com/jupyterhub/grafana-dashboards
• Start in the infra repo, then have a discussion about upstreaming

Task list

Tasks

Beta Give feedback

1. Use Prometheus metric exporter and mount the whole NFS disk as a volume
2. Verify the graphs that report disk size continue to make sense when referring to EBS volumes, and both staging and prod volumes are captured in a way that is distinguishable from on another
3. Investigate and enable Grafana alerting based on some threshold, alerts sent to support[at]2i2c.org (alerting was manually enabled)
4. Deploy to VEDA staging hub
5. Test by setting the threshold to something really low and see if an alert is triggered
6. Document the feature, the notifications, and how to resize the EBS volume

Definition of Done

• The feature/service is technically complete
• The feature/service been tested with one or more users (if applicable)
• Deployed to VEDA staging hub
• Feature has been tested to fire an alert
• The feature/service is well documented, and the documentation is accessible for the target user base

[GeorgianaElena]

@sgibson91, raising here for visibility so you're not caught by surprise in case you haven't seen #4923 (for more context https://2i2c.slack.com/archives/C055A1J1DRP/p1727942694562319)

[sgibson91]

Thank you @GeorgianaElena I did see it and did remember about it, but hadn't done the work digging it out yet! Thank you!

@sunu, #4923 will need to be reverted for Grafana alerting :)

[sgibson91]

@sunu found it difficult to programatically enable grafana alerting via grafonnet. I have proposed that we add the enablement of alerts as a one-time manual step in our hub deployment guide for now to close this issue out. I will open an issue to track a spike investigating grafonnet further in the new year, maybe with more 2i2c folks helping him.

[sgibson91]

Grafana needs access to 2i2c's Freshdesk SMTP server to send emails to support[at]

[yuvipanda]

@sgibson91 let's send these alerts to pagerduty rather than use smtp directly. I think the integration will be more straightforward this way, given we already do that in https://github.com/2i2c-org/infrastructure/blob/main/terraform/uptime-checks/pagerduty.tf

[sgibson91]

Amazing, thank you for the suggestion @yuvipanda!

[yuvipanda]

Another simpler suggestion is to try to use prometheus alertmanager. We already have it deployed, but disabled (

infrastructure/helm-charts/support/values.yaml

Line 72 in ace761c

alertmanager:

(with an inline comment that is not true anymore)).

It should be easier than trying to automatically do this in grafana:

1. The pagerduty service key can be set in yaml config (https://prometheus.io/docs/alerting/latest/configuration/#pagerduty_config) in our secret config (https://github.com/2i2c-org/infrastructure/blob/main/helm-charts/support/enc-support.secret.values.yaml) as that's the same for all our clusters
2. Alerts are generated by prometheus itself, using YAML based alerting rules (https://prometheus.io/docs/prometheus/latest/configuration/alerting_rules/). This would go in our support chart config
3. The alert configuration itself is a prometheus expression, which is exactly the same as it is for grafana! So while what does the alerting and where it goes is different, the source of the data and how it's expressed is the same.

This would also allow us to add more alerting in the future without having to directly tie it to a grafana graph.

[yuvipanda]

I have proposed that we add the enablement of alerts as a one-time manual step in our hub deployment guide for now to close this issue out.

These alerts may get overwritten when we deploy the grafana dashboard next, so will have to be checked to see if they persist.

[sunu]

Another simpler suggestion is to try to use prometheus alertmanager

Oh, that's a great suggestion! Let's go with Prometheus Alertmanager if that's an option.