Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fixes #409: Adds support for compacted topics and commit offset resetting #425

Merged
merged 35 commits into from
Oct 12, 2022
Merged

fixes #409: Adds support for compacted topics and commit offset resetting #425

merged 35 commits into from
Oct 12, 2022

Conversation

astubbs
Copy link
Contributor

@astubbs astubbs commented Sep 30, 2022 •
edited
Loading

Fixes:

Checklist

  • Changelog

@astubbs
START: fixes confluentinc#409: Truncate state on offset high reset
5e8bfb6 
Track expected offsets returned from broker, and truncate where needed.
@astubbs astubbs linked an issue Sep 30, 2022 that may be closed by this pull request
Parallel Consumer keeps committing the old offset after OFFSET_OUT_OF_RANGE in auto.offset.reset = latest #409
Closed
@astubbs astubbs mentioned this pull request Sep 30, 2022
Closed
astubbs
astubbs commented Sep 30, 2022
View reviewed changes
Copy link
Contributor Author

@astubbs astubbs left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

....

...consumer-core/src/main/java/io/confluent/parallelconsumer/offsets/OffsetMapCodecManager.java Show resolved Hide resolved
parallel-consumer-core/src/main/java/io/confluent/parallelconsumer/state/PartitionState.java Outdated Show resolved Hide resolved
parallel-consumer-core/src/main/java/io/confluent/parallelconsumer/state/PartitionState.java Outdated Show resolved Hide resolved
...llel-consumer-core/src/test/java/io/confluent/parallelconsumer/state/PartitionStateTest.java Outdated Show resolved Hide resolved
@astubbs astubbs requested a review from rkolesnev September 30, 2022 15:44
@astubbs
Copy link
Contributor Author

astubbs commented Sep 30, 2022

This needs some more careful thinking through, the solution proposed won't work. I thought it needed to be checked continuously, but actually it may indeed only be upon first initial poll - as it's only the committed offset that's affected, not the offset map - I think.

@astubbs astubbs changed the title fixes #409: Truncate state on offset high reset fixes #409: Truncate state on offset higher or lower than committed and prune missing records Oct 4, 2022
@astubbs
Copy link
Contributor Author

astubbs commented Oct 11, 2022

This needs some more careful thinking through, the solution proposed won't work. I thought it needed to be checked continuously, but actually it may indeed only be upon first initial poll - as it's only the committed offset that's affected, not the offset map - I think.

done.

@astubbs
review
66fe6f5
@astubbs
Merge remote-tracking branch 'origin/master' into fixes/state-truncat…
e977ebe 
…ion-reset-latest

# Conflicts:
#	CHANGELOG.adoc
#	README.adoc
#	parallel-consumer-core/src/main/java/io/confluent/parallelconsumer/offsets/BitSetEncoder.java
@astubbs astubbs changed the title fixes #409: Truncate state on offset higher or lower than committed and prune missing records Truncate state when bootstrap polled offset higher or lower than committed and prune missing records Oct 11, 2022
@astubbs astubbs changed the title Truncate state when bootstrap polled offset higher or lower than committed and prune missing records fixes #409: Truncate state when bootstrap polled offset higher or lower than committed and prune missing records Oct 11, 2022
@astubbs astubbs marked this pull request as ready for review October 11, 2022 13:46
@astubbs astubbs changed the title fixes #409: Truncate state when bootstrap polled offset higher or lower than committed and prune missing records fixes #409: Adds support for compacted topics and commit offset resetting (#425) Oct 12, 2022
astubbs
astubbs commented Oct 12, 2022
View reviewed changes
Copy link
Contributor Author

@astubbs astubbs left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

...

@astubbs astubbs changed the title fixes #409: Adds support for compacted topics and commit offset resetting (#425) fixes #409: Adds support for compacted topics and commit offset resetting Oct 12, 2022
astubbs
astubbs commented Oct 12, 2022
View reviewed changes
Copy link
Contributor Author

@astubbs astubbs left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@astubbs astubbs merged commit de21a35 into confluentinc:master Oct 12, 2022
@astubbs astubbs deleted the fixes/state-truncation-reset-latest branch October 12, 2022 13:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@rkolesnev rkolesnev Awaiting requested review from rkolesnev

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Parallel Consumer keeps committing the old offset after OFFSET_OUT_OF_RANGE in auto.offset.reset = latest
1 participant
@astubbs