Understanding the Performance metrics better

[deshmukhrajvardhan]

Hello Team,

I am using these https://www.openpolicyagent.org/docs/latest/rest-api/#performance-metrics
to measure performance.

I don't understand the performance parameters well. e.g.

1. Is timer_server_handler_ns = sum of above parameters + extra ?
2. Is timer_rego_query_eval_ns calculated before timer_rego_module_compile_ns ? (is the query evaluated before rego module is compiled? that doesn't make sense)
3. What is the difference between timer_rego_query_eval_ns and timer_rego_query_compile_ns (what happens in these 2 steps)

Would like help with the above.

Thanks,
Raj

[philipaconrad]

@deshmukhrajvardhan Thank you for your patience! Here's what I was able to dig up from a cursory look around the OPA REST API sources, hunting for metrics-related stuff:

1. timer_server_handler_ns is a holistic measurement; it tells you how long everything took to process on the API endpoint you hit. So, it may wrap up some/most/all of the other parameters mentioned on that page, depending on the situation. It's meant to mimic common web framework endpoint timing metrics, so that you can understand the overall latency of a request/response on the server.
2. So, there's 2x separate things going on:
   • timer_rego_module_compile_ns should only come up when a new policy is created/updated with a PUT request. We try not to recompile modules in the server code unnecessarily, so I only see measurement for this on PUT/DELETE endpoints.
   • timer_rego_query_eval_ns should come up much more often, since each time you want to query a policy and get a decision back, the server will have to evaluate your query. There's definitely GET endpoints measuring this.
3. There's a lot of machinery around query eval, but as best I can tell, it looks like timer_rego_query_compile_ns is the measuring the time to compile the query, and timer_rego_query_eval_ns should be the overall time that eval took to (parse+compile+run) the query.

Disclaimer: I only spent about 10 min hunting around, so one or two points there may be wrong, but if you want to go spelunking yourself with grep, I'd recommend looking under server/ and topdown/ mostly, and look for instances of stuff like metrics.RegoQueryEval and friends, and that'll get you 90+% of the way there if you want to double-check. 😄

[deshmukhrajvardhan]

Thanks @philipaconrad ! This has been helpful.
It would be a good idea to put the above explanation in the opa REST API docs, Many developers will find this helpful
I will follow your guidance to double-check (^_^)

Update:
I found the metric names on which the timer is run

https://github.com/open-policy-agent/opa/blob/main/metrics/metrics.go#L21

And used these to double-check.
Thanks again @philipaconrad !