[alerting] Alerting for consistent server spawn failures on a hub

[damianavila]

Automated alerts for server spawn failures

If a JupyterHub is somehow failing to start user pods consistently, we should have an alert for that that hopefully reaches us before a user does. Note that this is fairly rare. The use case is to just provide enough evidence that this rare thing isn't happening right now.

Parent issue: #1804

Rationale: This is the most common way people report ‘the hub is not working’ that affects all the users, and requires immediate attention. This metric is already collected by prometheus. Setting this up will also lay the groundwork for future alerts based on prometheus metrics. This is also a useful alert, as it alerts on specific user facing symptoms, rather than causes that may have user specific symptoms.

Proposal

Definition of done:

• Define way to add alerts for hubs and clusters
• Create tuneable alerts for ‘server failed to start’, where thresholds can be set
• Runbooks on what people responding to these alerts should try, and how to navigate possible outages.
• Method where we can intentionally cause a hub to have failed spawns, and this triggers an alert to test and for training.

Out of scope:

• Paging people - alerts should currently perhaps go to slack, but definitely nobody should be getting called or actively paged.

Updates and actions

2023-03-28: @pnasrat update to be a Q2 goal

[yuvipanda]

Me and @pnasrat discussed moving this to next quarter, based on capacity of the team this quarter. I think we should! Thoughts, @damianavila?

[damianavila]

Thoughts, @damianavila?

I concur with the move, although we should evaluate it alongside the other potential goals for Q3 and decide if we really want to include it for the next quarter.