Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support FNV-1a hash for partitioning (for Sarama compatibility) #2724

Closed
Manicben opened this issue Feb 19, 2020 · 0 comments · Fixed by #2753
Closed

Support FNV-1a hash for partitioning (for Sarama compatibility) #2724

Manicben opened this issue Feb 19, 2020 · 0 comments · Fixed by #2753

Comments

@Manicben
Copy link
Contributor

Manicben commented Feb 19, 2020 •
edited
Loading

This issue is more of a feature request derived from this confluent-kafka-go issue: confluentinc/confluent-kafka-go#16 (comment)
Raising here for awareness, I'll prob have a go at a PR to add this.
Please let me know if this needs to be in a specific format!

Description

Sarama uses the FNV-1a hashing algorithm as its default partitioner. This means that anyone migrating from Sarama to confluent-kafka-go will either have to implement the partitioner themselves (making an extra metadata round trip call to get number of partitions) or change algorithms to one of the librdkafka supported algorithms (potentially causing short term out of order processing by downstream consumers).
I propose adding the FNV-1a hash as a supported partitioner in librdkafka, similar to how Murmur2 was added for Java compatibility.
@Manicben Manicben mentioned this issue Mar 10, 2020
Merged
Manicben added a commit to Manicben/librdkafka that referenced this issue Mar 11, 2020
@Manicben
partitioner: Add FNV-1a partitioner (confluentinc#2724)
6fe229a 
Adds a new partitioner using the FNV-1a hashing algorithm, with some
tweaks to match Sarama's default hashing partitioner behaviour.
Main use case is for users switching from Sarama to librdkafka
(or confluent-kafka-go) and wanting to maintain ordering guarantees.
edenhill pushed a commit that referenced this issue Mar 11, 2020
@Manicben @edenhill
partitioner: Add FNV-1a partitioner (#2724)
76795b9 
Adds a new partitioner using the FNV-1a hashing algorithm, with some
tweaks to match Sarama's default hashing partitioner behaviour.
Main use case is for users switching from Sarama to librdkafka
(or confluent-kafka-go) and wanting to maintain ordering guarantees.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Add FNV-1a partitioner (#2724) Manicben/librdkafka
1 participant
@Manicben