fix(1170): Use scope guard to decrement reducer queue length #1172
Conversation
Fixes #1170. Also updates the bucket values for the queue length histogram. Also removes the max queue length metric, since the histogram should suffice.
| let _guard = QueueMetric::inc(db); | ||
| let permit = acq.await.map_err(|_| PoolClosed)?; |
There was a problem hiding this comment.
I'm fairly confident we should not have any more issues with this metric not resetting, or this metric going negative, with this scope guard in place. Would you agree @coolreader18?
| let queue_length = WORKER_METRICS.instance_queue_length.with_label_values(&self.db).get(); | ||
| WORKER_METRICS | ||
| .instance_queue_length_histogram | ||
| .with_label_values(&self.db) | ||
| .observe(queue_length as f64); |
There was a problem hiding this comment.
why do we need a separate histogram? shouldn't grafana accumulate the data for us over time anyway?
There was a problem hiding this comment.
if you didn't need to set the histogram you could just use the inc_scope utility function I made a while ago, which does this same sorta thing.
There was a problem hiding this comment.
why do we need a separate histogram? shouldn't grafana accumulate the data for us over time anyway?
The instance_queue_length metric is a gauge which is only sampled once per sampling interval. So all it gives us is the queue length at the end of the sampling interval over time. But it doesn't tell us anything about the queue length at the beginning or middle of the sampling interval.
instance_queue_length_histogram on the other hand will build a histogram of all the different values the queue length takes throughout the sampling interval, and this will give us percentiles.
It is true that instance_queue_length really doesn't give us any new information over the histogram. We just need some variable to track the current length of the queue. I just chose a prometheus metric for that purpose mainly for ease of use.
There was a problem hiding this comment.
you could just use the inc_scope utility function I made a while ago
I would have done that if it weren't for the get
Fixes #1170.
Also updates the bucket values for the queue length histogram. Also removes the max queue length metric, since the histogram should suffice.
Description of Changes
Please describe your change, mention any related tickets, and so on here.
API and ABI breaking changes
If this is an API or ABI breaking change, please apply the
corresponding GitHub label.
Expected complexity level and risk
How complicated do you think these changes are? Grade on a scale from 1 to 5,
where 1 is a trivial change, and 5 is a deep-reaching and complex change.
This complexity rating applies not only to the complexity apparent in the diff,
but also to its interactions with existing and future code.
If you answered more than a 2, explain what is complex about the PR,
and what other components it interacts with in potentially concerning ways.
Testing
Describe any testing you've done, and any testing you'd like your reviewers to do,
so that you're confident that all the changes work as expected!