Choose partition based on key

[lhrolim]

Is there any way to choose which partition to send a message based on a message key?

Other libraries seem to provide such feature out of the box:

https://github.com/SOHU-Co/kafka-node#sendpayloads-cb (node)
https://docs.spring.io/spring-kafka/api/org/springframework/kafka/core/KafkaTemplate.html#send-java.lang.String-java.lang.Integer-K-V- (java)

At this point, would we need to pick the topic metadata and implement the algorithm based on the key hash for instance?

[joshuawscott]

KafkaEx uses murmur2 hashing by default, which matches the Java client.

If you want to define your own, you can look at https://hexdocs.pm/kafka_ex/KafkaEx.Partitioner.html - the default implementation of that is https://hexdocs.pm/kafka_ex/KafkaEx.DefaultPartitioner.html

[mjparrott]

@joshuawscott Is the fact that KafkaEx uses the default partitioner (matching the Java client) documented anywhere? I was looking around to see what KafkaEx would do if I only provided the key and this issue is the only reference to this I could find.

[jbruggem]

@mjparrott it is documented in the hex docs of KafkaEx.DefaultPartitioner.

If you believe it could be relevant to document it somewhere else, could you please provide a PR for that ? Thanks a lot !