Skip to content

[fix](cloud-mow) Fix the issue of inaccurate estimation of txn size when updating delete bitmap #46969

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
dataroaring merged 4 commits into from
Jan 16, 2025
Merged

[fix](cloud-mow) Fix the issue of inaccurate estimation of txn size when updating delete bitmap #46969

dataroaring merged 4 commits into from
Jan 16, 2025

Conversation

@hust-hhb
Copy link
Contributor

@hust-hhb hust-hhb commented Jan 14, 2025 •
edited
Loading

fdb_txn_size is not key size+value size, actually txn size is the triple of key size plus value size at least for writing, so use txn->approximate_bytes() to calculating txn size is more accurate. However, txn->approximate_bytes() is not 100% accurate either, in my test, when txn->approximate_bytes() bingger than 8.3MB, it will meet Transaction exceeds byte limit error.

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@Thearas
Copy link
Contributor

Thearas commented Jan 14, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@hust-hhb
Copy link
Contributor Author

hust-hhb commented Jan 14, 2025

run buildall

zhannngchen
zhannngchen reviewed Jan 14, 2025
View reviewed changes
zhannngchen
zhannngchen reviewed Jan 14, 2025
View reviewed changes
zhannngchen
zhannngchen reviewed Jan 14, 2025
View reviewed changes
@hust-hhb
Copy link
Contributor Author

hust-hhb commented Jan 14, 2025

run buildall

zhannngchen
zhannngchen approved these changes Jan 15, 2025
View reviewed changes
Copy link
Contributor

@zhannngchen zhannngchen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jan 15, 2025
@github-actions
Copy link
Contributor

github-actions bot commented Jan 15, 2025

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented Jan 15, 2025

PR approved by anyone and no changes requested.

dataroaring
dataroaring approved these changes Jan 15, 2025
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dataroaring dataroaring merged commit 91d3a52 into apache:master Jan 16, 2025
31 of 32 checks passed
@dataroaring dataroaring added the usercase Important user case type label label Jan 16, 2025
github-actions bot pushed a commit that referenced this pull request Jan 16, 2025
@hust-hhb
[fix](cloud-mow) Fix the issue of inaccurate estimation of txn size w…
9162a85 
…hen updating delete bitmap (#46969)

fdb_txn_size is not key size+value size, actually txn size is the triple
of key size plus value size at least for writing, so use
txn->approximate_bytes() to calculating txn size is more accurate.
However, txn->approximate_bytes() is not 100% accurate either, in my
test, when txn->approximate_bytes() bingger than 8.3MB, it will meet
Transaction exceeds byte limit error.
@github-actions github-actions bot mentioned this pull request Jan 16, 2025
Merged
zhannngchen pushed a commit that referenced this pull request Jan 17, 2025
@github-actions @hust-hhb @CalvinKirs
branch-3.0: [fix](cloud-mow) Fix the issue of inaccurate estimation o…
415aa23 
…f txn size when updating delete bitmap #46969 (#47046)

Cherry-picked from #46969

Co-authored-by: huanghaibin <huanghaibin@selectdb.com>
Co-authored-by: Calvin Kirs <guoqiang@selectdb.com>
@hust-hhb hust-hhb mentioned this pull request Feb 11, 2025
Merged
16 tasks
dataroaring pushed a commit that referenced this pull request Feb 11, 2025
@hust-hhb
[test](cloud-mow)Add update big delete bitmap case (#47097)
cbcebb1 
This case is used to test the scenario of updating a large number of
delete bitmap when doing compaction, if delete bitmap size over the fdb
limit(10MB), updating delete bitmap will fail, pr #46969 fix this
problem, before the fix, this case will fail with error msg "Transaction
exceeds byte limit" on metaservice.
github-actions bot pushed a commit that referenced this pull request Feb 11, 2025
@hust-hhb
[test](cloud-mow)Add update big delete bitmap case (#47097)
7933e5f 
This case is used to test the scenario of updating a large number of
delete bitmap when doing compaction, if delete bitmap size over the fdb
limit(10MB), updating delete bitmap will fail, pr #46969 fix this
problem, before the fix, this case will fail with error msg "Transaction
exceeds byte limit" on metaservice.
@gavinchou gavinchou mentioned this pull request Feb 18, 2025
Closed
lzyy2024 pushed a commit to lzyy2024/doris that referenced this pull request Feb 21, 2025
@hust-hhb @lzyy2024
[fix](cloud-mow) Fix the issue of inaccurate estimation of txn size w…
004d22c 
…hen updating delete bitmap (apache#46969)

fdb_txn_size is not key size+value size, actually txn size is the triple
of key size plus value size at least for writing, so use
txn->approximate_bytes() to calculating txn size is more accurate.
However, txn->approximate_bytes() is not 100% accurate either, in my
test, when txn->approximate_bytes() bingger than 8.3MB, it will meet
Transaction exceeds byte limit error.
lzyy2024 pushed a commit to lzyy2024/doris that referenced this pull request Feb 21, 2025
@hust-hhb @lzyy2024
[test](cloud-mow)Add update big delete bitmap case (apache#47097)
4ede535 
This case is used to test the scenario of updating a large number of
delete bitmap when doing compaction, if delete bitmap size over the fdb
limit(10MB), updating delete bitmap will fail, pr apache#46969 fix this
problem, before the fix, this case will fail with error msg "Transaction
exceeds byte limit" on metaservice.
koarz pushed a commit to koarz/doris that referenced this pull request Jun 4, 2025
@hust-hhb @koarz
[test](cloud-mow)Add update big delete bitmap case (apache#47097)
1b065e2 
This case is used to test the scenario of updating a large number of
delete bitmap when doing compaction, if delete bitmap size over the fdb
limit(10MB), updating delete bitmap will fail, pr apache#46969 fix this
problem, before the fix, this case will fail with error msg "Transaction
exceeds byte limit" on metaservice.
suxiaogang223 pushed a commit to suxiaogang223/doris that referenced this pull request Jul 15, 2025
@hust-hhb
branch-3.0: [fix](cloud-mow) Fix the issue of inaccurate estimation o…
a4ad5f1 
…f txn size when updating delete bitmap apache#46969 (apache#3920)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@w41ter w41ter w41ter approved these changes

@zhannngchen zhannngchen zhannngchen approved these changes

@dataroaring dataroaring dataroaring approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/3.0.4-merged reviewed usercase Important user case type label

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@hust-hhb @Thearas @w41ter @zhannngchen @dataroaring