Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update latest and stable offset if it was in stabilizing state #661

Merged
merged 4 commits into from
Dec 30, 2023
Merged

Update latest and stable offset if it was in stabilizing state #661

merged 4 commits into from
Dec 30, 2023

Conversation

akrambek
Copy link
Contributor

@akrambek akrambek commented Dec 29, 2023

Description

This is a current understanding of the issue

zilla: offset <= stable <= latest
which means offset <= stable, stable <= latest, offset <= latest
kafka: offset < stable <= high water mark
which means offset < stable, stable <= high water mark, offset < high water mark

Due to this invariant sometimes we may get into a state where the fix to send a flush when stableOffset or latestOffset has advanced (with no records) after records have been sent or caught up to the latest offset so at least one FLUSH was already sent.

Fixes #666

@akrambek akrambek marked this pull request as ready for review December 29, 2023 19:33
@jfallows jfallows merged commit aeeb302 into aklivity:develop Dec 30, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Kafka Merge is getting stall because of intermediate partition offset state
2 participants