Monitoring Gatekeeper service health with Prometheus

[laimison]

Hi all,
how do you monitor Gatekeeper itself to ensure that it’s healthy as a service?

I think standard Prometheus metrics to see if no pods are restarting kube_pod_container_status_restarts_total will be helpful.

How to catch that audit completed successfully? Does it confirm that Total Violations count is updated successfully (if audit is completed)?

Anything else to be sure just from alerts?

Thank you

[peteroneilljr]

Maybe @maxsmythe or @ritazh might have some suggestions?

[peteroneilljr]

Actually I see that there is some information already here.

#18 (comment)

[maxsmythe]

Hi! Thanks for digging through the context Peter :)

WRT monitoring health, there is a lot that could be looked at for thorough monitoring, also its possible to monitor Gatekeeper from a number of different points... metrics, pods, the status field of individual resources. Here are some highlights:

audit

• The status.auditTimestamp should be reasonably recent for all resources (this only gets updated after audit updates totalViolations
• There is also a prometheus metric: gatekeeper_audit_last_run_time
• You could monitor audit events written to the pod logs, as described here: https://open-policy-agent.github.io/gatekeeper/website/docs/audit#audit-logs

constraint/template ingestion

• make sure the status.byPod field exists for all pods on all constraints and templates, make sure observedGeneration matches metadata.generation
• make sure status.byPod shows no errors
• gatekeeper_constraint_templates metric should have the expected number of constraint templates, the value of the metric for the error tag should be 0
• same thing for gatekeeper_constraints

Webhook

• There are standard webhook metrics issued by the controller-runtime library (they have the controller_runtime prefix). If you find best practices for those metrics, they would be a good starting point
• You can watch the gatekeeper_validation_request_count metric

[laimison]

Thanks for comprehensive answer. I'm going to start implementing monitoring in a week or so from existing metrics, not sure if everything could be exposed as metrics without an additional exporter.

[peteroneilljr]

@laimison be sure to share what you create! The OPA community would love to see it!

[laimison]

Sure, I always share the answer that worked for me (in terms of Gatekeeper community, there are previous issues in Slack). At least basic monitoring is in my to do list. So I'll share my findings here in a while.

[laimison]

Due to changes in tasks, never got a chance to work on it.

[peteroneilljr]

Thanks for the update @laimison :)