Align S3 Gateway Prometheus Histogram Buckets with lakeFS API Buckets #7622
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
No linked issues found. Please add the corresponding issues in the pull request description. |
E2E Test Results - DynamoDB Local - Local Block Adapter |
guy-har
changed the title
guy-har
added
include-changelog
arielshaqed
approved these changes
Apr 3, 2024
pkg/gateway/stats.go
Outdated
| Help: "request durations for lakeFS storage gateway", | ||
| Name: "gateway_request_duration_seconds", | ||
| Help: "request durations for lakeFS storage gateway", | ||
| Buckets: []float64{0.01, 0.05, 0.1, 0.2, 0.25, 0.5, 1, 2.5, 5, 10, 30, 60}, |
There was a problem hiding this comment.
Not a fan of the intervals, especially the middle octave 0.1, 0.2, 0.25, 0.5, 1.0. The bucket-to-bucket size ratios are really weird!
| limit | 0.01 | 0.05 | 0.1 | 0.2 | 0.25 | 0.5 | 1 | 2.5 | 5 | 10 | 30 | 60 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| next limit / this limit | - | 5.0 | 2.0 | 2.0 | 1.25 | 2.0 | 2.0 | 2.5 | 2.0 | 2.0 | 3.0 | 2.0 |
The jumps 0.1 -> 0.2 -> 0.25 -> 0.5 are pretty poorly spaced: 2x, 1.25x, 2x. I would get rid of 0.2.
I don't care that the spacing on the buckets >10s is weird, it's not as different and the numbers there will barely matter (until they do).
Suggested change
| Buckets: []float64{0.01, 0.05, 0.1, 0.2, 0.25, 0.5, 1, 2.5, 5, 10, 30, 60}, | |
| Buckets: []float64{0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10, 30, 60}, |
There was a problem hiding this comment.
@arielshaqed, I changed both here and in the API, PTAL
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
The Prometheus histogram buckets currently used for monitoring the lakeFS API (
api_requests_total) are specifically tailored to the lakeFS service's response times, providing valuable insights into API performance. However, the histogram for the S3 gateway (gateway_request_duration_seconds) utilizes the default Prometheus buckets, which may not accurately reflect the performance characteristics of the S3 gateway interactions.Aligning the S3 gateway's Prometheus histogram buckets with those defined for the lakeFS API can enhance monitoring accuracy and offer better insights into S3 gateway performance. This adjustment would allow for a more consistent and informative analysis of request durations across both interfaces.