[fix](compaction) Make creating tablet idempotently to keep compaction Job idempotent #56061

wyxxxcat · 2025-09-15T07:37:33Z

What problem does this PR solve?

If rpc create_tablets is not idempotent, it will recreate tablet stats, which is tragic for compaction task idempotence

if (compaction.base_compaction_cnt() < stats.base_compaction_cnt() ||
    compaction.cumulative_compaction_cnt() < stats.cumulative_compaction_cnt()) {
    code = MetaServiceCode::STALE_TABLET_CACHE;
    SS << "could not perform compaction on expired tablet cache."
       << " req_base_compaction_cnt=" << compaction.base_compaction_cnt()
       << ", base_compaction_cnt=" << stats.base_compaction_cnt()
       << ", req_cumulative_compaction_cnt=" << compaction.cumulative_compaction_cnt()
       << ", cumulative_compaction_cnt=" << stats.cumulative_compaction_cnt();
    msg = ss.str();
    return;
}

Release note

None

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

Thearas · 2025-09-15T07:37:42Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

wyxxxcat · 2025-09-15T07:44:55Z

run buildall

cloud/src/meta-service/meta_service.cpp

wyxxxcat · 2025-09-15T10:43:05Z

run buildall

hello-stephen · 2025-09-15T11:01:40Z

Cloud UT Coverage Report

Increment line coverage 87.50% (28/32) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	84.58% (1536/1816)
Line Coverage	68.30% (27657/40493)
Region Coverage	69.03% (13626/19740)
Branch Coverage	59.16% (7282/12310)

wyxxxcat · 2025-09-15T12:34:02Z

run buildall

wyxxxcat · 2025-09-15T13:34:52Z

run buildall

doris-robot · 2025-09-15T13:55:41Z

Cloud UT Coverage Report

Increment line coverage 87.88% (29/33) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	84.01% (1534/1826)
Line Coverage	68.08% (27542/40457)
Region Coverage	68.56% (13564/19783)
Branch Coverage	58.78% (7241/12318)

cloud/src/meta-service/meta_service.cpp

cloud/src/meta-service/meta_service.h

wyxxxcat · 2025-09-15T16:34:37Z

run buildall

doris-robot · 2025-09-15T16:47:57Z

Cloud UT Coverage Report

Increment line coverage 88.24% (30/34) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	84.01% (1534/1826)
Line Coverage	68.12% (27558/40458)
Region Coverage	68.56% (13564/19783)
Branch Coverage	58.87% (7252/12318)

github-actions · 2025-09-16T06:15:26Z

PR approved by at least one committer and no changes requested.

github-actions · 2025-09-16T06:15:28Z

PR approved by anyone and no changes requested.

dataroaring

LGTM

…n Job idempotent (apache#56061) If rpc create_tablets is not idempotent, it will recreate tablet stats, which is tragic for compaction task idempotence ``` if (compaction.base_compaction_cnt() < stats.base_compaction_cnt() || compaction.cumulative_compaction_cnt() < stats.cumulative_compaction_cnt()) { code = MetaServiceCode::STALE_TABLET_CACHE; SS << "could not perform compaction on expired tablet cache." << " req_base_compaction_cnt=" << compaction.base_compaction_cnt() << ", base_compaction_cnt=" << stats.base_compaction_cnt() << ", req_cumulative_compaction_cnt=" << compaction.cumulative_compaction_cnt() << ", cumulative_compaction_cnt=" << stats.cumulative_compaction_cnt(); msg = ss.str(); return; } ```

…ep compaction Job idempotent #56061 (#56108) picked from #56061

…n Job idempotent (apache#56061) If rpc create_tablets is not idempotent, it will recreate tablet stats, which is tragic for compaction task idempotence ``` if (compaction.base_compaction_cnt() < stats.base_compaction_cnt() || compaction.cumulative_compaction_cnt() < stats.cumulative_compaction_cnt()) { code = MetaServiceCode::STALE_TABLET_CACHE; SS << "could not perform compaction on expired tablet cache." << " req_base_compaction_cnt=" << compaction.base_compaction_cnt() << ", base_compaction_cnt=" << stats.base_compaction_cnt() << ", req_cumulative_compaction_cnt=" << compaction.cumulative_compaction_cnt() << ", cumulative_compaction_cnt=" << stats.cumulative_compaction_cnt(); msg = ss.str(); return; } ```

…mpotently to keep compaction Job idempotent (apache#4688) pick: apache#56061

wyxxxcat requested review from dataroaring, gavinchou and w41ter as code owners September 15, 2025 07:37

gavinchou reviewed Sep 15, 2025

View reviewed changes

cloud/src/meta-service/meta_service.cpp Outdated Show resolved Hide resolved

gavinchou reviewed Sep 15, 2025

View reviewed changes

cloud/src/meta-service/meta_service.cpp Outdated Show resolved Hide resolved

gavinchou reviewed Sep 15, 2025

View reviewed changes

cloud/src/meta-service/meta_service.cpp Show resolved Hide resolved

wyxxxcat force-pushed the compaction_idempotent branch from fa263f4 to 7505926 Compare September 15, 2025 08:50

dataroaring added dev/3.0.x dev/3.1.x p0_b labels Sep 15, 2025

wyxxxcat force-pushed the compaction_idempotent branch 2 times, most recently from 9fbdea8 to 0a028ca Compare September 15, 2025 10:42

wyxxxcat force-pushed the compaction_idempotent branch 2 times, most recently from 7a8dd26 to c236e70 Compare September 15, 2025 12:32

wyxxxcat force-pushed the compaction_idempotent branch from c236e70 to e660827 Compare September 15, 2025 13:34

gavinchou reviewed Sep 15, 2025

View reviewed changes

cloud/src/meta-service/meta_service.cpp Show resolved Hide resolved

gavinchou reviewed Sep 15, 2025

View reviewed changes

cloud/src/meta-service/meta_service.h Show resolved Hide resolved

1

748c320

wyxxxcat force-pushed the compaction_idempotent branch from e660827 to 748c320 Compare September 15, 2025 16:34

gavinchou approved these changes Sep 16, 2025

View reviewed changes

gavinchou added the dev/4.0.x label Sep 16, 2025

github-actions bot added the approved Indicates a PR has been approved by one committer. label Sep 16, 2025

github-actions bot added the reviewed label Sep 16, 2025

dataroaring approved these changes Sep 16, 2025

View reviewed changes

dataroaring merged commit bef8f36 into apache:master Sep 16, 2025
33 of 35 checks passed

github-actions bot added dev/3.0.x-conflict dev/3.1.x-conflict labels Sep 16, 2025

This was referenced Sep 16, 2025

[branch-3.0] [fix](compaction) Make creating tablet idempotently to keep compaction Job idempotent #56061 #56106

Closed

branch-3.1: [fix](compaction) Make creating tablet idempotently to keep compaction Job idempotent #56061 #56108

Merged

morrySnow pushed a commit that referenced this pull request Sep 17, 2025

branch-3.1: [fix](compaction) Make creating tablet idempotently to ke…

b57b405

…ep compaction Job idempotent #56061 (#56108) picked from #56061

morrySnow added dev/3.1.1-merged and removed dev/3.1.x dev/3.1.x-conflict labels Sep 17, 2025

morrySnow mentioned this pull request Sep 22, 2025

Release Note 3.1.1 #56289

Closed

yiguolei pushed a commit to yiguolei/incubator-doris that referenced this pull request Sep 28, 2025

branch-selectdb-doris-3.0: [fix](compaction) Make creating tablet ide…

a8b0f93

…mpotently to keep compaction Job idempotent (apache#4688) pick: apache#56061

yiguolei removed the dev/4.0.x label Sep 29, 2025

[fix](compaction) Make creating tablet idempotently to keep compaction Job idempotent #56061

[fix](compaction) Make creating tablet idempotently to keep compaction Job idempotent #56061

Uh oh!

Conversation

wyxxxcat commented Sep 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

Thearas commented Sep 15, 2025

Uh oh!

wyxxxcat commented Sep 15, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

wyxxxcat commented Sep 15, 2025

Uh oh!

hello-stephen commented Sep 15, 2025

Cloud UT Coverage Report

Uh oh!

wyxxxcat commented Sep 15, 2025

Uh oh!

wyxxxcat commented Sep 15, 2025

Uh oh!

doris-robot commented Sep 15, 2025

Cloud UT Coverage Report

Uh oh!

Uh oh!

Uh oh!

wyxxxcat commented Sep 15, 2025

Uh oh!

doris-robot commented Sep 15, 2025

Cloud UT Coverage Report

Uh oh!

github-actions bot commented Sep 16, 2025

Uh oh!

github-actions bot commented Sep 16, 2025

Uh oh!

dataroaring left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

wyxxxcat commented Sep 15, 2025 •

edited

Loading