Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fixes #327: Improve error message when Kafka Streams magic number found #482

Merged
merged 13 commits into from
Dec 29, 2022

Conversation

nachomdo
Copy link
Contributor

@nachomdo nachomdo commented Nov 22, 2022

Fixes: #327

36:31.314 INFO  [pc-broker-poll]  (AbstractParallelEoSStreamProcessor.java:387#onPartitionsAssigned) Assigned 3 total (3 new) partition(s) [input-topic-0, input-topic-1, input-topic-2]
36:31.343 ERROR [pc-broker-poll]  (PartitionStateManager.java:116#onPartitionsAssigned) Error in onPartitionsAssigned
io.confluent.parallelconsumer.offsets.KafkaStreamsV2EncodingNotSupported: Kafka Streams offset metadata V2 not supported
	at io.confluent.parallelconsumer.offsets.EncodedOffsetPair.getDecodedIncompletes(EncodedOffsetPair.java:120) ~[classes/:na]
	at io.confluent.parallelconsumer.offsets.OffsetMapCodecManager.decodeCompressedOffsets(OffsetMapCodecManager.java:246) ~[classes/:na]
	at io.confluent.parallelconsumer.offsets.OffsetMapCodecManager.deserialiseIncompleteOffsetMapFromBase64(OffsetMapCodecManager.java:168) ~[classes/:na]
	at io.confluent.parallelconsumer.offsets.OffsetMapCodecManager.deserialiseIncompleteOffsetMapFromBase64(OffsetMapCodecManager.java:158) ~[classes/:na]
	at io.confluent.parallelconsumer.offsets.OffsetMapCodecManager.decodePartitionState(OffsetMapCodecManager.java:172) ~[classes/:na]
	at io.confluent.parallelconsumer.offsets.OffsetMapCodecManager.lambda$loadPartitionStateForAssignment$0(OffsetMapCodecManager.java:133) ~[classes/:na]
	at java.base/java.util.HashMap.forEach(HashMap.java:1421) ~[na:an]

Checklist

  • [] Documentation (if applicable)
  • [] Changelog

@nachomdo nachomdo force-pushed the bugs/fix-magic-number-ks branch from f0dd4f4 to 33140bf Compare November 22, 2022 21:53
@nachomdo nachomdo force-pushed the bugs/fix-magic-number-ks branch from 33140bf to 02d9efe Compare November 22, 2022 22:25
Copy link
Contributor

@astubbs astubbs left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good work!

We only need one exception - if that's the pattern to use
can you include a reference for where you got the magic number info from? byte 1 and 2

if this is the error that's printed:
"Kafka Streams offset metadata V1 not supported"
we need something more friendly. should be something like "it looks like you're reusing a kafka streams consumer group id with PC. This isn't supported - you should use a fresh consumer group, unique to your use of pc" - something like that?

@nachomdo nachomdo requested a review from astubbs November 24, 2022 13:37
@astubbs astubbs changed the title Fixes #327: Improve error message when Kafka Streams magic number found fixes #327: Improve error message when Kafka Streams magic number found Dec 8, 2022
@nachomdo nachomdo force-pushed the bugs/fix-magic-number-ks branch from 54923fa to d3a25b6 Compare December 12, 2022 11:41
@what-the-diff
Copy link

what-the-diff bot commented Dec 12, 2022

  • Added KafkaStreamsEncodingNotSupported exception
  • Added support for Kafka Streams magic number in OffsetEncoding enum
  • Updated CloseAndOpenOffsetTest to skip new encoding types (KafkaStreams and KafkaStreamsV2) when running tests on offsets open/close cycle
  • Updated ensureEncodingGracefullyWorksWhenOffsetsAreVeryLargeAndNotSequential test to skip new encoding types (KafkaStreams and KafkaStreamsV2)
  • Update WorkManagerOffsetMapCodecManagerTest with two additional deserialise tests, one for each version of the kafka streams metadata format: v1 & v2

Copy link
Contributor

@astubbs astubbs left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

just about there! can close this one up quick now.

astubbs and others added 6 commits December 21, 2022 02:03
…umer/offsets/OffsetEncoding.java

Co-authored-by: Antony Stubbs <antony.stubbs@gmail.com>
…umer/offsets/WorkManagerOffsetMapCodecManagerTest.java

Co-authored-by: Antony Stubbs <antony.stubbs@gmail.com>
…umer/offsets/OffsetEncoding.java

Co-authored-by: Antony Stubbs <antony.stubbs@gmail.com>
…umer/offsets/EncodedOffsetPair.java

Co-authored-by: Antony Stubbs <antony.stubbs@gmail.com>
@nachomdo
Copy link
Contributor Author

@astubbs Thanks for the review! I will apply the suggested feedback today so we can close this 😊

@nachomdo nachomdo requested a review from astubbs December 22, 2022 16:54
@astubbs astubbs merged commit f8d263b into confluentinc:master Dec 29, 2022
@nachomdo nachomdo deleted the bugs/fix-magic-number-ks branch January 3, 2023 15:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

When unexpected magic bytes occur, improve the feedback in the error shown to the user
2 participants