Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix kafka benchmarks #1382

Open
rukai opened this issue Dec 4, 2023 · 0 comments
Open

fix kafka benchmarks #1382

rukai opened this issue Dec 4, 2023 · 0 comments

Comments

@rukai
Copy link
Member

rukai commented Dec 4, 2023

kafka benchmarks are currently quite wonky.
We currently use the cpp rdkafka driver which does not map well to rust async.
Things to do from here:

  • add benches using https://github.com/influxdata/rskafka this will give us a point of comparison with a driver that properly uses async.
  • Need to use unique keys to ensure spread across topology3 cluster, otherwise it all gets routed to a single node 🙃 - Windsock kafka fix routing #1417
  • rdkafka maintains a queue of outgoing produce requests which maps poorly to rust async. In order to alleviate this we can:
    • use multiple producer objects to get multiple polling threads? not sure if this works like that.
    • do not start the request timer on produce requests that fail to queue
    • Maybe dropping down to BaseProducer and implementing our own threaded logic would be better?
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant