Our alerts

[jlangy]

Overview

Our alerting for sso-keycloak uses sysdig for monitoring and alerting, which integrates with opsGenie for escalating issues. Please use this thread to recommend new alerts, procedures or ask questions!

The Sysdig alerts and dashboards are managed exclusively through the terraform files stored in the sso-sysdig repo. Any update to them made in the Sysdig GUI will be overwritten by the next PR to main branch in that repos. Further documentation can be found in the sso-sysdig repo,

For team members with access to our private repos, you can find more information on the team's bcgov-c wiki.

Tools

• Sysdig: Sysdig monitors our cluster stats and raises alerts at different severity levels (low, medium, or high). Our events are metric-based to allow for self-healing, e.g an alert for the Number of Ready Pods will self-close if a new pod can replace the lost one.
• OpsGenie: OpsGenie is setup to consume sysdig alerts, and escalate them to on-call staff if the sysdig issue cannot self-heal. Depending on the severity of the alert, OpsGenie will wait a given amount of time to call staff. The current escalation times are:

Severity	On-call Escalation Time	Team Escalation Time (If not acknowledged/closed)
Low1	30 minutes	45 minutes
Medium	15 minutes	30 minutes
High	Immediate	5 minutes

1. Low level alerts are only escalating through opsgenie. Higher priority alerts go through email and Rocketchat as well

Alerts

Ready Pods

Due to a significant number of false positives we were getting from these alerts we created a new set of alerts (July 2022). These alerts measure the difference between the number of desired pods and the number of ready pods. If sysdig fails on a given kubernetes node and does not report the status of a pod on it, it will not trigger an alert. Meaning there will be no false positive triggered.

Alert	Severity	Trigger
1 keycloak pod not ready Silver (15+ minutes)	Low	1 pod not ready for more than 15 min
More than 1 keycloak pod not ready Silver	Med	> 1 pod not ready for 8 min
More than 2 keycloak pods not ready Silver	High	>2 pods not ready for 5 min
1 patroni pod not ready Silver	Med	1 pod not ready for at least 5 min
More than 1 patroni pod not ready Silver	High	>1 pod not ready for at least 5 min

A recent upgrade to sysdig made pod count alerts prone to false triggers. These alerts were disconnected from opsgenie, though they still send an alert to rocketchat with the prefix OLD ALERT. Our instance monitors an rh-sso deployment as well as a Patroni stateful set, and alerts are set up based on the number of available pods.

Alert	Severity	Trigger
SSO - Ready Pods Low	Low	< 7 pods
SSO - Ready Pods Med	Med	< 6 pods
SSO - Ready Pods High	High	< 4 pods
Patroni - Ready Pods Med	Med	< 3 pods
Patroni - Ready Pods High	High	< 2 pods

---

CPU usage

CPU has been our best indicator to date of cluster health. We monitor the CPU usage of our rh-sso deployment for spikes.

Alert	Severity	Trigger
SSO - CPU usage Med	Med	SSO pods summed CPU use over 3 cores
SSO - CPU usage High	High	SSO pods summed CPU use over 5 cores

Available Space

Alerts are setup to notify when our database volumes are approaching maximum capacity

Alert	Severity	Trigger
Patroni - PVCs > 60%	Low	The volume use is over 60%
Patroni - PVCs > 80%	Med	The volume use is over 80%
Patroni - PVCs > 90%	High	The volume use is over 90%
Patroni - Backup PVC > 60%	Low	The backup container's volume use is over 80%
SSO - Log PV Usage over 90%	Hight	The PVC holding keycloak logs is almost full

Other

These alerts have been setup earlier, but may fail to measure what we want. We are taking the approach of turning them off if they start to false fire, and we are leaving them active for the time being.

Alert	Severity	Trigger	Description
App - Prod - SSO-PGSQL - Pod Restart	Med	Patroni restart count > 1	Using restart count will keep the alert in an active status until the restarted pod is deleted
App - Prod - SSO - Patroni Backup storage > 85%	Low	The filesystem of the backup pod is over 85%	This measures the pods filesystem, not the connected PVC use
App - Prod - SSO - CPU Spike Over 5	High	The average of the maximum CPU cores is over 5	The pods cpu.cores.used seems to spike very high when pods are starting up, so measuring off of the maximum can cause this to be overly sensitive when pods restart