Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

zstd.go initializes shared global encoder+decoder regardless of whether zstd is being used #1937

Closed
dnwe opened this issue May 7, 2021 · 2 comments
Labels
needs-investigation Issues that require followup from maintainers stale Issues and pull requests without any recent activity

Comments

@dnwe
Copy link
Collaborator

dnwe commented May 7, 2021

Currently we initialise a global zstd encoder+decoder at import time:
https://github.com/Shopify/sarama/blob/8dbbfb50f4698c577c49ff8142c90a9185eb3114/zstd.go#L7-L8

Under the covers the compress library spins up goroutines to handle the decompression/compression

If the Sarama user doesn't ever intend to use zstd then this is a bunch of wasted resource.

I'm also not sure if a shared global encoder+decoder is the correct behaviour, or whether we should have per-producer encoders and WithEncoderConcurrency(1) and per-consumer decoders and WithDecoderConcurrency(1)

I also wonder if we should be surfacing the WithDecoderLowmem and WithWindowSize options in sarama.Config{...} so that users can tune the performance vs memory usage tradeoffs themselves.

@github-actions

This comment was marked as outdated.

@github-actions github-actions bot added the stale Issues and pull requests without any recent activity label Aug 26, 2023
@dnwe dnwe added needs-investigation Issues that require followup from maintainers and removed stale Issues and pull requests without any recent activity labels Aug 27, 2023
Copy link

Thank you for taking the time to raise this issue. However, it has not had any activity on it in the past 90 days and will be closed in 30 days if no updates occur.
Please check if the main branch has already resolved the issue since it was raised. If you believe the issue is still valid and you would like input from the maintainers then please comment to ask for it to be reviewed.

@github-actions github-actions bot added the stale Issues and pull requests without any recent activity label Dec 27, 2023
@github-actions github-actions bot closed this as not planned Won't fix, can't repro, duplicate, stale Jan 30, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
needs-investigation Issues that require followup from maintainers stale Issues and pull requests without any recent activity
Projects
None yet
Development

No branches or pull requests

1 participant