Fix partial messages handling #1149
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Kafka 2.0 introduced chunked message down conversions which might produce partial messages at the end of the set. These partial messages are not well formed beyond offset and length, so they might cause strange decoding errors down the line. This fix makes `lengthField` a `dynamicPushDecoder` so it checks that the `packetDecoder` has at least that many bytes. It fixes the problem above and I think it's a more robust check in general.
|
Thanks for digging into this! I've pushed a test branch on CI but still seeing failures https://travis-ci.org/Shopify/sarama/jobs/415607362? |
Also, this fixes the case when there's more than one message in a message set with a partial trailing message.
|
Test branch is 🍏 , could please chery-pick 3de182b to enable Kafka 2.0 on CI? |
|
Many thanks for contributing this! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Kafka 2.0 introduced chunked message down conversions which might
produce partial messages at the end of the set. These partial messages
are not well formed beyond offset and length, so they might cause
strange decoding errors down the line.
This fix makes
lengthFieldadynamicPushDecoderso it checks thatthe
packetDecoderhas at least that many bytes. It fixes the problemabove and I think it's a more robust check in general.
Fixes #1144