Q4 2022 Goal: Monitoring & alerting for our infrastructure

[yuvipanda]

I want us to feel confident that our infrastructure exists in a state that is not on fire. This means we need monitoring and alerting in place to make sure we have evidence to believe that - that 'everything is ok'.

For this, we need to have good monitoring and alerting that we trust. We should be able to realistically say 'there are currently no alerts, so we believe there are no outages right now'. And have enough trust in the process to believe that.

Here's a bunch of things that will help get us there:

• Black box alerting for all the hubs we maintain, so if they aren't accessible on the web we'll get an alert. This would've caught https://github.com/2i2c-org/incident-reports/blob/main/reports/2022-08-30-utoronto-inaccessible.pdf without it needing to be reported Add simple HTTPS uptime checks for all hubs #1750
• Same as above for our 'support' setup, primarily prometheus. Add health checks for our prometheus instances #1778
• Alerts to make sure we are not going to suddenly run out of quota in a project #2265
• Alerts for when our clusters will use up more than forecasted amount of $$$ #2266
• [alerting] Alerting for consistent server spawn failures on a hub #2267
• Alerts for consistent login failure on a hub #2268

Alerts are of two kinds - immediate outage (delivered via pagerduty) and a 'cliff alert' (something will go bad in a few days if you do not deal with something now) (delivered via freshdesk). Each alert should be clearly actionable - it's better to not have an alert at all than one we ignore.

[jmunroe]

(I have a clear memory of already commenting here last week .. but perhaps I had neglected to click 'Comment' and lost my thought! Apologizes if I already typed this up somewhere else.)

We've received a request to provide more analytics to our community partners. Specifically:

It would be great if 2i2c could deliver monthly usage reports in the form of an email summarizing:

• Number of active users in the month
• Average number of hours per user session in the month
• Breakdown of machine types used
• Total cost of the hub in the month

This information will help our project decide whether usage of our hub is growing and will help us justify whether to continue with this service.

Since this Q4 2022 Goal is tackling 'monitoring', could it be expanded in scope to include analytics as well?

I don't want to download extra work to @2i2c-org/tech-team so my request is to understand more about the scope of work is planned here and to offer my assistance in the development work, if appropriate.

[damianavila]

Since this Q4 2022 Goal is tackling 'monitoring', could it be expanded in scope to include analytics as well?

I would avoid expanding the scope.

I don't want to download extra work to https://github.com/orgs/2i2c-org/teams/tech-team so my request is to understand more about the scope of work is planned here and to offer my assistance in the development work, if appropriate.

Your help will be always welcome, @jmunroe.
I would suggest opening a new goal issue for "Analytics and Reporting" and keeping it separated from this one.

[damianavila]

During Q4 health checks were deployed.
I have captured the remaining items in the top message as dedicated child issues to deal with in the next monitoring theme iteration (realistically, proto-Q2 content).