Usage examples of BalancedConsumer

[har777]

I'm a bit confused about how to use the BalancedConsumer. Say you have 12 partitions in Kafka. Do you create 12 instances of BalancedConsumer on different threads for max consumption ? Any code examples ?

Thanks !

[emmettbutler]

Hi @har777. This does highlight the need for some slightly deeper usage examples that just what you find in the README.

You don't need to create one BalancedConsumer per partition. The optimal ratio of partitions to consumer instances isn't a set number, since it can differ based on your application's needs. I can tell you that at Parsely we use a topic with 100 partitions, and that each consuming service has about 100 instances of BalancedConsumer running. However, the 100 consumers is more a function of the total throughput of the topic (which is about 30k messages per second) than of the number of partitions. In fact, we use 100 partitions exactly because the number of partitions must be at least the number of consumers.

Essentially you need to figure out what the target throughput of your pipeline is, then assign a number of partitions and consumer processes appropriate for that. See these benchmarks for an idea of how much a single consumer can do per second. Also I don't recommend running multiple BalancedConsumer instances in the same process - I've done it before, but the more proven use case is to create a new process for each KafkaClient/BalancedConsumer pair.