Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Increase indexheader_lazy_load_duration_seconds buckets #5005

Merged

Conversation

pracucci
Copy link
Contributor

  • I added CHANGELOG entry for this change.
  • Change is not relevant to the end user.

Changes

In Cortex we've noticed that the 99th percentile computed onindexheader_lazy_load_duration_seconds is always capped to 5s even if the average can be way higher. Reason is that the largest histogram bucket is 5s but lazy loading can take way longer than that (mostly depending on index size and the disk speed).

In this PR I proposes to add more buckets to the metric, to have a better estimation of 99th percentile on high durations.

Change is not relevant to the end user.

Is it? Do you track such changes in CHANGELOG?

Verification

N/A

squat
squat previously approved these changes Dec 27, 2021
Copy link
Member

@squat squat left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @pracucci

Signed-off-by: Marco Pracucci <marco@pracucci.com>
@pracucci pracucci force-pushed the increase-index-header-lazy-loading-buckets branch from d95d664 to 718d999 Compare December 28, 2021 08:13
Copy link
Contributor

@yeya24 yeya24 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @pracucci. I think 300s is little bit too large but overall LGTM.

@pracucci
Copy link
Contributor Author

Thanks @pracucci. I think 300s is little bit too large but overall LGTM.

We've actually seen several minutes too, on a store-gateway under heavy load with indexes each of few tens GB each.

@pracucci
Copy link
Contributor Author

If there are no concerns, can we get this merged, please?

@squat squat merged commit abd4f46 into thanos-io:main Dec 29, 2021
@squat
Copy link
Member

squat commented Dec 29, 2021

thanks @pracucci 🌮

@pracucci
Copy link
Contributor Author

Thanks to you @squat ! 🌮

@pracucci pracucci deleted the increase-index-header-lazy-loading-buckets branch December 30, 2021 07:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants