Default partitioner does not match Java client

[mjparrott]

When using KafkaEx to produce to a topic that is also produced to by an application that is using the default Java client, I noticed that the partitions that were being produced to weren't the same in KafkaEx and the Java client.

The reason appears to be how umurmur2 compares to the Java client.

In KafkaEx, the hash is calculated like so:

kafka_ex/lib/kafka_ex/utils/murmur.ex

Line 32 in 403024e

key |> murmur2() |> band(0xFFFFFFFF)

In the default Java client, it is calculated here: https://github.com/apache/kafka/blob/717c55be971df862c55f55d245b9997f1d6f998c/clients/src/main/java/org/apache/kafka/clients/producer/internals/DefaultPartitioner.java#L59

The difference here is in the value we use to bitwise AND the key hash and the number of partitions. In KafkaEx, it is 0xFFFFFFFF. In the Java client, it is 0x7fffffff.

Changing the 0xFFFFFFFF to 0x7FFFFFFF seems to make things match. I'm wondering if just changing the umurmur2 method is the correct approach here? And if so, how would versioning work? Since this would affect anyone using KafkaEx's default partitioner. Happy to make a PR if we have an approach.

[mjparrott]

I've made a PR that could possibly fix this issue, see the linked #399 .

[jbruggem]

Good catch @mjparrott 👍

[dantswain]

Closed by #399 thanks!