Consuming messages from each partition separately

[everhardt]

A Consumer that is subscribed to multiple partitions cannot control the mix of messages it consumes from each partition.
There are various reasons why controlling that is relevant, see eg. #719 and https://github.com/confluentinc/librdkafka/wiki/FAQ#what-are-partition-queues-and-why-are-some-partitions-slower-than-others.

That last link shows that librdkafka supports consuming from each partition separately. It would be great if that implementation is available in node-rdkafka as well. An example could be a PartitionConsumer class, that does not have the normal .consume methods, but .consume(topic, partition, number, callback).

[martijnimhoff]

We're working on a PR: withthegrid#2

[pveller]

re: #719 . node-rdkafka would probably need to migrate to using rd_kafka_queue_new / rd_kafka_consume_queue. You can't do better if you are on rd_kafka_consume

https://github.com/confluentinc/librdkafka/wiki/Consuming-from-multiple-topics-partitions-from-a-single-thread