Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use buffer pools for decompression #2484

Merged
merged 1 commit into from
Jul 25, 2023
Merged

Conversation

ronanh
Copy link
Contributor

@ronanh ronanh commented Jul 4, 2023

While profiling my application, I've noticed that Sarama spends a significant amount of time growing slices when it's decompressing messages:

growSlice

This patch uses an intermediate buffer from a pool for decompression, to avoid these allocations.
When the final size is know, the result buffer can be allocated and copied.

After the patch:

withpool2

@ronanh
Copy link
Contributor Author

ronanh commented Jul 4, 2023

I have signed the CLA!

@dnwe dnwe force-pushed the improve_decompress branch 3 times, most recently from 13ec122 to 832e358 Compare July 17, 2023 11:15
@dnwe dnwe force-pushed the improve_decompress branch from 832e358 to c2e8186 Compare July 23, 2023 10:24
Signed-off-by: Ronan Harmegnies <ronan.harmegnies@gmail.com>
@dnwe dnwe force-pushed the improve_decompress branch from c2e8186 to cf5b296 Compare July 24, 2023 11:09
@hindessm
Copy link
Collaborator

Looks very good to me:

before:
BenchmarkDecompression/lz4_2048_compressable-4          578866    9368 ns/op    8533 B/op   11 allocs/op
BenchmarkDecompression/lz4_2048_lessCompressable-4      830926    6521 ns/op    8506 B/op   11 allocs/op
BenchmarkDecompression/lz4_2048_compressable-16         804277    7391 ns/op    8544 B/op   11 allocs/op
BenchmarkDecompression/lz4_2048_lessCompressable-16    1221556    4904 ns/op    8538 B/op   11 allocs/op

after:
BenchmarkDecompression/lz4_2048_compressable-4          792148    7512 ns/op    2373 B/op    7 allocs/op
BenchmarkDecompression/lz4_2048_lessCompressable-4     1302768    4614 ns/op    2395 B/op    7 allocs/op
BenchmarkDecompression/lz4_2048_compressable-16         920294    5854 ns/op    2364 B/op    7 allocs/op
BenchmarkDecompression/lz4_2048_lessCompressable-16    1544532    3909 ns/op    2420 B/op    7 allocs/op

@dnwe dnwe merged commit 735f33b into IBM:main Jul 25, 2023
@dnwe dnwe added the feat label Jul 25, 2023
ioanzicu pushed a commit to ioanzicu/sarama that referenced this pull request Jul 31, 2023
Signed-off-by: Ronan Harmegnies <ronan.harmegnies@gmail.com>
Signed-off-by: Ioan Zicu <ioan.zicu@nokia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants