Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

metric: optimize Histogram.RecordValue (1.2% cpu on sysbench) #133306

Open
tbg opened this issue Oct 23, 2024 · 1 comment
Open

metric: optimize Histogram.RecordValue (1.2% cpu on sysbench) #133306

tbg opened this issue Oct 23, 2024 · 1 comment
Labels
branch-master Failures and bugs on the master branch. C-performance Perf of queries or internals. Solution not expected to change functional behavior. o-perf-efficiency Related to performance efficiency P-2 Issues/test failures with a fix SLA of 3 months T-observability

Comments

@tbg
Copy link
Member

tbg commented Oct 23, 2024

on current master, histogram observations account for >1% of cpu.1
This may be due to the bucket count, though the profile only spends a portion of the time in the bucket search.
Sadly most of the time is spent in dependencies. Still, this is a sizeable chunk of CPU.

https://pprof.me/fcd1cbfe64d478dd1e6211970ff9107e/?profileType=profile%253Acpu%253Ananoseconds%253Acpu%253Ananoseconds%253Adelta&filter_by_function=Observe&search_string=Observe

image

Jira issue: CRDB-43623

Footnotes

  1. https://github.com/cockroachlabs/perf-efficiency-team/pull/4/commits

@tbg tbg added C-performance Perf of queries or internals. Solution not expected to change functional behavior. branch-master Failures and bugs on the master branch. P-2 Issues/test failures with a fix SLA of 3 months o-perf-efficiency Related to performance efficiency T-observability labels Oct 23, 2024
@tbg tbg changed the title metric: optimize Histogram.RecordValue metric: optimize Histogram.RecordValue (1.2% cpu on sysbench) Oct 25, 2024
@tbg
Copy link
Member Author

tbg commented Dec 16, 2024

Took a quick look here and nothing jumps out. This is just a very very hot code path. It doesn't help that we maintain two histograms per histogram (one for windowing, one for the cumulative counters). We could save half the work if we updated only the most recent one, and then merged into the cumulative at each rotation interval - that would give a factor of ~0.5 here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
branch-master Failures and bugs on the master branch. C-performance Perf of queries or internals. Solution not expected to change functional behavior. o-perf-efficiency Related to performance efficiency P-2 Issues/test failures with a fix SLA of 3 months T-observability
Projects
None yet
Development

No branches or pull requests

1 participant