Add functionality for custom hash partitioner. #837
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The wrapper function NewCustomHashPartitioner is a wrapper around the PartitionerConstructor, allowing for specifying the hash algorithm, which will be used.
|
Thanks, this makes sense to me. It was a couple of years ago now, but I swear we chose FNV at the time because that's what the upstream kafka libs were using. I'm surprised to see so much variation now. Anyway. |
|
Thanks! I can now update to the new version and get rid of the crufty current solution I mentioned above. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The function NewCustomHashPartitioner is a wrapper around the PartitionerConstructor, allowing for specifying the hash algorithm, which will be used.
Background for the proposed change: messages published on compacted topics by services, written in different languages (Go, Java, etc) end up on different partitions, although they have the same key. We were able to narrow the problem down to different hashing algorithms used by the different libraries, when determining the partition: murmur2 for a Java lib, FNV in sarama, crc32 for .NET and xxHash in librdkafka.
A quick workaround is of course to copy/paste the hashPartitioner implementation to change the implementation of the hash.Hash32 interface ...which is basically one line out of ~50. So I decided I'd propose this PR to facilitate the use of different hashing algorithms, based on specific needs.