How does Kafka Consumer Polling work when multiple partitions are assigned? #1416
jindalshivam09
started this conversation in
General
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
We want to have a Kafka consumer which should subscribe to multiple topics (each topic can have multiple partitions too). Now we also want to prioritize a particular topic such that say if there are two topics: topic_1 and topic_2, consumer should consume from topic_1 first and when all messages are consumed, then consumer from topic_2.
To implement such functionality, I started looking into how polling work in case of single topic but with multiple partitions are assigned to the consumer.
I have read various different answers over stack overflow and other sources, but still not sure with actual implementation. Could any please explain how is it implemented? How do we ensure fairness among partitions?
Also any ideas to implement priority among topics which a consumer is subscribed to multiple topics?
Couple of ideas I am thinking are:
Beta Was this translation helpful? Give feedback.
All reactions