how can I dynamically increase consumers #19067
ramprasadp
started this conversation in
General
Replies: 1 comment
-
Yes, Pulsar addresses these problems by having more ways to consume the data. |
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
I am currently using kafka , sorry no experience of pulsar
Our production load is very peaky and distributed across multiple topics .
Number of topics I restrict to "n" , usually 100 , because kafka does not support too many topics. And fixed 12 partitions
The biggest challenge I see is that suddenly when the load ( number of events ) entering a single topic increases There is no way I can increase the number consumers for just that topic beyond 12
The only way to scale seems to increase number of partitions which is not possible in runtime. Also the consumers are sometimes unevenly loaded and rebalancing kills the performance , for a good 5-10 minutes the consumers go starved when that h
Does Pulsar address these problems
Beta Was this translation helpful? Give feedback.
All reactions