Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[v24.2.x] CORE-6835 Fix first time compacted segment upload #22924

Merged
merged 3 commits into from
Aug 19, 2024
Merged

[v24.2.x] CORE-6835 Fix first time compacted segment upload #22924

merged 3 commits into from
Aug 19, 2024

Commits on Aug 17, 2024

  1. rptest: Add new ducktape test 

    The test creates compacted topic without TS. It produces the data and
waits until the data is compacted. After that it enables TS for the
topic. The archiver has to be able to upload the data.

(cherry picked from commit 9bbe429)
    @Lazin @vbotbuildovich
    Lazin authored and vbotbuildovich committed Aug 17, 2024
    Configuration menu
    Copy the full SHA
    b915634 View commit details
    Browse the repository at this point in the history

  2. archival: Fix upload policy for compacted topics 

    The archival_policy::find_segment method has a bug which causes it to
skip compacted segments unconditionally. The code checks if the segment
that it found is compacted and if this is the case it advances the
iterator to the next segment. This operation causes ntp_archiver to skip
one segment. The uploaded segment doesn't line up with the previous one
so the validation error is triggered.

The commit also implements handling of gaps in a safe way. We're not
supposed to have gaps in the log (even in compacted logs) so the
implementation has linear complexity. It's not supposed to be used and
is good enough to hadnle unexpected unique glitch.

(cherry picked from commit de01321)
    @Lazin @vbotbuildovich
    Lazin authored and vbotbuildovich committed Aug 17, 2024
    Configuration menu
    Copy the full SHA
    a4447fa View commit details
    Browse the repository at this point in the history

  3. archival: Fix archival_policy unit-test 

    The test validates logic which was removed. Previousy, the uploader was
skipping single compacted segment and now it will upload compacted
segments. So the upload in the test should start from 0.

(cherry picked from commit e2fd980)
    @Lazin @vbotbuildovich
    Lazin authored and vbotbuildovich committed Aug 17, 2024
    Configuration menu
    Copy the full SHA
    e331502 View commit details
    Browse the repository at this point in the history