You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Describe the bug
Excessive Prometheus metrics cardinality causes server degradation. MetricsManager middleware creates a new label for all unique URLs, which results in server resources depletion and eventually, panics (see https://github.com/ory/hydra/blob/master/metrics/prometheus/middleware.go#L30). So even a very short-lived pentest scan can generate hundreds of thousands of new metrics which could cause Hydra, Prometheus, and all related monitoring tools to fail.
Expected behavior
Ideally, only endpoints registered in HTTP routers should show up as a separate label in Prometheus metrics. All other requests should generate metrics with the URL label unmatched.
PR with a fix will be submitted soon.
The text was updated successfully, but these errors were encountered:
Describe the bug
Excessive Prometheus metrics cardinality causes server degradation.
MetricsManager
middleware creates a new label for all unique URLs, which results in server resources depletion and eventually, panics (see https://github.com/ory/hydra/blob/master/metrics/prometheus/middleware.go#L30). So even a very short-lived pentest scan can generate hundreds of thousands of new metrics which could cause Hydra, Prometheus, and all related monitoring tools to fail.A few examples:
Reproducing the bug
curl http://localhost:4444/randomRequest
curl http://localhost:4445/metrics/prometheus
Expected behavior
Ideally, only endpoints registered in HTTP routers should show up as a separate label in Prometheus metrics. All other requests should generate metrics with the URL label
unmatched
.PR with a fix will be submitted soon.
The text was updated successfully, but these errors were encountered: