Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Consumer: handle compressed relative offsets #735

Merged
merged 1 commit into from
Aug 30, 2016

Conversation

eapache
Copy link
Contributor

@eapache eapache commented Aug 23, 2016

New message format does something weird with these. See
https://cwiki.apache.org/confluence/display/KAFKA/KIP-31+-+Move+to+relative+offsets+in+compressed+message+sets

Fixes #720. Supercedes #721. Thanks to @dynamix for the first draft of the fix.

@dynamix @Dieterbe could you verify this fix works correctly?

cc @wvanbergen

New message format does something weird with these. See
https://cwiki.apache.org/confluence/display/KAFKA/KIP-31+-+Move+to+relative+offsets+in+compressed+message+sets

Fixes #720. Supercedes #721. Thanks to @dynamix for the first draft of the fix.
@Dieterbe
Copy link
Contributor

hey, when i rebuild my tool with your branch and send it dummy data, snappy compressed, the first time it actually seemed to work, but in subsequent tries i seemed to get the following again:

[Sarama] 2016/08/25 08:40:00 kafka: error while consuming mdm/0: kafka: response did not contain all the expected topic/partition blocks
[Sarama] 2016/08/25 08:40:00 consumer/broker/0 abandoned subscription to mdm/0 because kafka: response did not contain all the expected topic/partition blocks
[Sarama] 2016/08/25 08:40:02 consumer/mdm/0 finding new broker
[Sarama] 2016/08/25 08:40:02 client/metadata fetching metadata for [mdm] from broker kafka:9092
[Sarama] 2016/08/25 08:40:02 consumer/broker/0 added subscription to mdm/0
[Sarama] 2016/08/25 08:40:02 kafka: error while consuming mdm/0: kafka: response did not contain all the expected topic/partition blocks
[Sarama] 2016/08/25 08:40:02 consumer/broker/0 abandoned subscription to mdm/0 because kafka: response did not contain all the expected topic/partition blocks

However i was also having some other issues with our dev-stack. Will need to do more proper testing.

@dynamix
Copy link

dynamix commented Aug 26, 2016

Currently testing this in combination with the consumer group stuff I'm working on. This looks better than my initial fix - which was indeed incorrect and did lead to invalid offsets. So far it seems to work. 👍

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Consumer with API version set to V0_10_0_0 fails to consume compressed topics
3 participants