Prometheus choice of bucket boundaries for http_request_duration_seconds

[luong-komorebi]

Hi, first of all, thanks for the OPA. Awesome work.

I am creating a Grafana dashboard for OPA. When visualizing http_request_duration_seconds, I face a problem when the data of average response time of all HTTP requests to opa doesnt match the percentile of duration of requests.

Here're some images showing the problem.
You can see that we have the same spike in two graphs, which is a good thing. On the avg request duration, I am able to find out the avg request duration down to microseconds.

However, the quantile stops at seconds

and the value doesnt change much overtime

which leads me to thinking that the we havent had the right bucket configuration for http_request_duration_seconds for our OPA.

From my view, I can see #1638 is where the work for http_request_duration_seconds is done. This is using the default bucket configuration that prometheus provides, but these buckets are for typical web application and in OPA's case the numbers are not granular enough.

Another case may happen that my queries are wrong, but these queries are pretty standard I dont think I messed up anything. Feel free to view the code at https://github.com/luong-komorebi/opa-grafana-dashboard

Steps to Reproduce the Problem

• OPA version: 0.26.0
• Have OPA, prometheus configured and running
• Make some http call to OPA
• If you have grafana up and running, install my grafana dashboard from Github or Grafana.com
• If you dont, go to prometheus and query average request duration as well as, for example, 50th percentile for http_request_duration_seconds in 5 min interval
• Check the query for average request duration and see why they differ so much from the http request duration quantiles.

Additional Info

If possible and my assumption is right, I suggest we lower the number for the buckets.

[srenatus]

Thanks for bringing this up! 👏 We've started discussing this internally, the current thinking is that ditching some higher granularity buckets in favour of adding a few smaller ones is probably the right move. You'd agree, I presume? 😃

[luong-komorebi]

Yes, I totally agree @srenatus

[luong-komorebi]

I tried to turn my idea into codes and it looks like this #3214 Maybe this is what you guys are looking for