Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Branch-2.7][Cherry-pick] Fix the reader skips compacted data which original ledger been removed. #16407

Merged
merged 3 commits into from
Jul 6, 2022

Conversation

Technoboy-
Copy link
Contributor

Cherry-pick #12522

Motivation

The compactor update the compaction cursor(mark delete) first and then update the compactionHorizon of the compacted topic. During the compaction cursor move forward, the original ledger will be removed if no other durable cursors. At the same time, if the reader is reading data from the original ledger, the reader will skip the data while the original ledger been removed, details to see #6787. So the reader might skip the compacted data since the compactionHorizon have not updated yet.

The approach is:

  1. Update the compactionHorizon before the compaction cursor move forward,
    so that the reader will not skip the original data before compactionHorizon updated.
    If the broker crashes before the new compacted Ledger ID been persistent,
    after the topic been loaded, the compaction can be trigger again and will not loss any data,
    but we will have an orphan ledger cannot be delete in the BookKeeper cluster.
  2. Remove the previous compacted Ledger after the compaction cursor move forward, make sure the new compacted Ledger ID been persistent,Otherwise, we might lost compacted ledger if broker crashes.

@Technoboy- Technoboy- self-assigned this Jul 5, 2022
@Technoboy- Technoboy- merged commit f78c63b into apache:branch-2.7 Jul 6, 2022
@Technoboy- Technoboy- deleted the cherry-pick-12522 branch August 10, 2022 05:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants