Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: disable vt proto pools #3034

Merged
merged 6 commits into from
Feb 28, 2024
Merged

Conversation

kolesnikovae
Copy link
Collaborator

@kolesnikovae kolesnikovae commented Feb 27, 2024

Should fix #3024

I tested distributors without VT protobuf pools and I think that it's probably better to disable them altogether:

  • We do not benefit from reduced allocs as it's compensated by the cost of sync pools and resets (CPU time).
  • Memory consumption is 10-15% lower without pools.
  • Code is simpler. I suspect that Excessive memory consumption in distributors causes OOM #3024 is caused by wrong handling of pooled items. By removing the pools we eliminate this factor.

Version w/o pools deployed at ~10:10:
image

The impact on GC is understood and expected:

image

@kolesnikovae kolesnikovae marked this pull request as ready for review February 27, 2024 12:20
@kolesnikovae kolesnikovae requested a review from a team as a code owner February 27, 2024 12:20
Copy link
Contributor

@aleks-p aleks-p left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The code is simpler and the impact looks good. I only left one remark about a change in a generated file.

Copy link
Contributor

@simonswine simonswine left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for getting some real numbers on this ❤️

LGTM, after we adapt the generation

@kolesnikovae
Copy link
Collaborator Author

Yeah, thanks for pointing this out guys! That's a leftover – we had a dependency in the JFR parser, so I hacked it quickly for testing and forgot to rollback. I removed the dependency and the generated code for pools. Next step is to write/revive our very own efficient pprof parser

@kolesnikovae kolesnikovae merged commit c1ccb50 into main Feb 28, 2024
19 checks passed
@kolesnikovae kolesnikovae deleted the fix/distributor-pprof-proto-pool branch February 28, 2024 03:21
aleks-p added a commit that referenced this pull request Mar 7, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Excessive memory consumption in distributors causes OOM
3 participants