OPA generates a lot of bundle metrics and floods system

[costimuraru]

Short description

We have OPA deployed in Kubernetes as a standalone service (ie. 3 pods). We generate a new bundle every 30 seconds (bundle contains updated rego policies), which OPA downloads. We also scrape the metrics on each OPA pod (via Prometheus) to monitor them. We've noticed that the number of metrics emitted by OPA increases dramatically - from ~400 metrics when OPA starts to around 200 000+ after a few days. This is per pod. It seems that the increase is attributed to a series of metrics that relate to the bundle:

• bundle_loading_duration_ns
• last_success_bundle_activation
• last_success_bundle_download
• last_success_bundle_request
• etc.

Each of this metric has a label named active_revision, which I believe is the bundle id. Given that we load a new bundle every 30 seconds, the number of metrics increases fast.

You can see the output of OPA /metrics with 200k+ metrics in this Gist.

Examples:

• OPA version
  0.39
• Example query, input, data, and policy that OPA was given
• Example output that OPA returned
  curl http://opa-pod:8182/metrics | wc -l
  229129

Steps To Reproduce

1. Start OPA server, which downloads a new bundle every 30 seconds.
2. Check the metrics outputted by /metrics over time

Expected behavior

A constant number of metrics should be outputted over time.

Actual behavior

The number of metrics increases over time dramatically. It starts with 400 metrics and in 2 days, in our case, it reached 229 000 metrics. This floods our systems (Prometheus/Cortex), where we have quota on the number of unique metrics scrapped. The scrapping itself also takes a lot of time (10-20 seconds).

Additional context

[srenatus]

Thanks for reporting! Indeed it sounds like having the active revision label is the problem. Would the metrics still be useful without that? 🤔

[srenatus]

Cc @rafaelreinert what do you think? I guess we could either make it a feature you'd need to enable, or drop the labels?

[costimuraru]

@srenatus in our case the metrics are valuable, even without the active_revision label. In fact, we already have recording rules in place that discard that label. The problem is that the number of metrics on the OPA increases constantly, to the point where Prometheus can’t scrape it anymore.
We want to know how the system behaves over time, regardless of what was the actual bundle that misbehaved. If we detect an anomaly, say bundle_loading_duration > x seconds at dateY, we can always go to the logs and see what happened at that time, what was the bundle that generated the issue and so forth.

[rafaelreinert]

Hey @srenatus and @costimuraru , about the flag it already exists, in order to enable the status metrics on Prometheus it must be configured and the default value is false ( see https://www.openpolicyagent.org/docs/latest/management-status/#prometheus-status-metrics). I've done that just to not overflow the metrics in workloads like yours. I really believe someone activated it in your infrastructure. (Please let me know if the flag is not set, maybe a bug).
About the high cardinality of the metrics, it depends on the workload. For example, on my infrastructure, the OPA is distributed as a sidecar on every pod and is really important for monitoring which version of the bundle each pod has (the bundle doesn’t change too often ).
But I understand that doesn’t fit well when a workload has too many bundles in a short period of time, One alternative maybe is to create a new flag on the status config that configure the level of the status metrics. Maybe:

• full : all metrics with all bundles
• simple: only global metrics

What do you think about this approach?

[rafaelreinert]

Hey @srenatus and @costimuraru, I was thinking again about this issue and I realized what is maybe root cause, The problem is that, this function https://github.com/open-policy-agent/opa/blob/main/plugins/status/plugin.go#L478 is not resetting the older bundles metrics, I've not seen that as an issue because my environment has few bundles and the opa instance lifespan is less than 3 days. But in an environment that has a long lifespan and many bundles that become an issue because the number of metrics exported increase a lot. This week I will try to fix it and reset the metrics because if the bundle is not used anymore it doesn't be exported.
@costimuraru what do you think ?

[costimuraru]

Thanks for the details on the rationale behind this behavior, @rafaelreinert.
I'm trying to understand your suggestion with resetting the metrics. Are you saying that each time a bundle is loaded, the metrics will be reset and they will take the values based on that latest bundle? I worry that would not work fine with the counters. From the docs in Prometheus:

A counter is a cumulative metric that represents a single monotonically increasing counter whose value can only increase or be reset to zero on restart.

For instance, the bundle_failed_load_counter should (ideally) increase from the moment the opa server starts till it gets terminated. By this logic, it should not get reset when a new bundle is loaded. This allows us to run Prometheus queries such rate on this counter.

For our use case, I think what we need is to just remove the active_revision label from these metrics. Here is a change that seems to work fine for our use case: https://github.com/open-policy-agent/opa/compare/main...costimuraru:fix-metrics-cardinality?expand=1 - let me know if you see issues with this, we plan to give it a try (in prod), to have a quick fix :-D. In this way we have an aggregated view on these bundle metrics, which is what we need (a global view as you mentioned).

Coming back to the metric types, the gauge is probably not that important, cause it retains the latest value, right?
As for the histogram, I think having one across all bundles is actually a good thing: we can infer things like "95% of the bundles took less than X seconds to load", which should be possible now that we don't have one histogram per bundle, no?

Your suggestion to have a flag which makes it possible to select between these 2 behaviors is probably best (metrics with and without the active_revision label).
Wdyt?

[costimuraru]

After stripping the activeRevision label, OPA is looking much better:

[rafaelreinert]

I am thinking about that, maybe the best solution is to remove the active_revision for all metrics (as you have done) and create another metric (gauge) last_active_revision with the active_revision as label and reset it each update.
With this solution, we can have the active_revision from every bundle and we eliminate the other metrics' high cardinality.

[srenatus]

@rafaelreinert that sounds reasonable. @costimuraru what do you think?

@rafaelreinert would you be able to pick up making this change? 😃 (I'll take care of it if it's too much on your plate right now.)

[costimuraru]

Thanks @srenatus.
I've incorporated the feedback from @rafaelreinert and opened #4600. I'm gonna test it today and tomorrow in our env and see how it works. (I plan to use Grafana annotations with the newly added metrics that @rafaelreinert suggested and see how it behaves).