Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[fix](rowset meta) Fix rowset meta size relation #41022

Open
wants to merge 6 commits into
base: master
Choose a base branch
from
Open

[fix](rowset meta) Fix rowset meta size relation #41022

wants to merge 6 commits into from

Conversation

Yukang-Lian
Copy link
Collaborator

Proposed changes

Issue #40879 step 2

make rowset total size = rowset data size + rowset index size

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@Yukang-Lian
Copy link
Collaborator Author

Tests will be added to verify the correction of this PR.

@Yukang-Lian
Copy link
Collaborator Author

run buildall

1 similar comment
@Yukang-Lian
Copy link
Collaborator Author

run buildall

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Sep 19, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

gavinchou
gavinchou requested changes Sep 19, 2024
View reviewed changes
Copy link
Collaborator

@gavinchou gavinchou left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pls add regression test to ensure the correctness from this PR on

cloud/src/meta-service/meta_service_txn.cpp Show resolved Hide resolved
@github-actions github-actions bot removed approved Indicates a PR has been approved by one committer. labels Sep 19, 2024
@Yukang-Lian Yukang-Lian changed the title [Refactor](rowset meta) Refactor rowset meta size releation [Refactor](rowset meta) Refactor rowset meta size relation Sep 19, 2024
@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 37.32% (9590/25699)
Line Coverage: 28.70% (79264/276147)
Region Coverage: 28.17% (41030/145669)
Branch Coverage: 24.79% (20905/84318)
Coverage Report: http://coverage.selectdb-in.cc/coverage/eb9d0914c4a0d7001bce4b79a05c08475afece22_eb9d0914c4a0d7001bce4b79a05c08475afece22/report/index.html

@zhannngchen
Copy link
Contributor

pls add regression test to ensure the correctness from this PR on

+1

@Yukang-Lian
Copy link
Collaborator Author

run buildall

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 37.26% (9605/25775)
Line Coverage: 28.68% (79521/277234)
Region Coverage: 28.13% (41117/146193)
Branch Coverage: 24.77% (20960/84632)
Coverage Report: http://coverage.selectdb-in.cc/coverage/d2e24449e52101031adb42897de52eb536c64d77_d2e24449e52101031adb42897de52eb536c64d77/report/index.html

@Yukang-Lian
Copy link
Collaborator Author

run buildall

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 37.28% (9607/25773)
Line Coverage: 28.70% (79534/277151)
Region Coverage: 28.14% (41128/146133)
Branch Coverage: 24.78% (20964/84594)
Coverage Report: http://coverage.selectdb-in.cc/coverage/fd0aa240aee27f50c09fbe530d76cf0a8b564088_fd0aa240aee27f50c09fbe530d76cf0a8b564088/report/index.html

@Yukang-Lian
Copy link
Collaborator Author

run p0

@gavinchou gavinchou changed the title [Refactor](rowset meta) Refactor rowset meta size relation [fix](rowset meta) Fix rowset meta size relation Sep 25, 2024
gavinchou
gavinchou previously approved these changes Sep 25, 2024
View reviewed changes
w41ter
w41ter previously approved these changes Sep 25, 2024
View reviewed changes
@Yukang-Lian Yukang-Lian dismissed stale reviews from w41ter and gavinchou via a011c26 September 26, 2024 09:43
@Yukang-Lian
Copy link
Collaborator Author

run buildall

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 37.31% (9635/25823)
Line Coverage: 28.70% (79717/277712)
Region Coverage: 28.14% (41224/146514)
Branch Coverage: 24.76% (20993/84776)
Coverage Report: http://coverage.selectdb-in.cc/coverage/6738ab2113c9d357e6ad15b998fcc45b9b65155b_6738ab2113c9d357e6ad15b998fcc45b9b65155b/report/index.html

gavinchou
gavinchou previously approved these changes Sep 27, 2024
View reviewed changes
@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Sep 27, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@Yukang-Lian
Copy link
Collaborator Author

run buildall

@github-actions github-actions bot removed the approved Indicates a PR has been approved by one committer. label Sep 27, 2024
@Yukang-Lian
Copy link
Collaborator Author

run cloudut

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 37.30% (9627/25812)
Line Coverage: 28.69% (79672/277674)
Region Coverage: 28.13% (41198/146458)
Branch Coverage: 24.75% (20983/84772)
Coverage Report: http://coverage.selectdb-in.cc/coverage/89bd9a2b12cb24550d74750754c829c5a6f4706c_89bd9a2b12cb24550d74750754c829c5a6f4706c/report/index.html

csun5285
csun5285 suggested changes Sep 29, 2024
View reviewed changes
be/src/olap/rowset/beta_rowset_writer.cpp
@@ -830,7 +830,8 @@ Status BaseBetaRowsetWriter::_build_rowset_meta(RowsetMeta* rowset_meta, bool ch

rowset_meta->set_num_segments(segment_num);
rowset_meta->set_num_rows(num_rows_written + _num_rows_written);
rowset_meta->set_total_disk_size(total_data_size + _total_data_size);
rowset_meta->set_total_disk_size(total_data_size + _total_data_size + total_index_size +
Copy link
Contributor

@csun5285 csun5285 Sep 29, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you want to use the formula total size = column data size + index size, you need to modify the data size and index size statistics in the segment creator, vertical segment writer, and vertical beta rowset writer.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Currently, data size = segment file size + inverted index file size, total size = data size

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@csun5285 csun5285 csun5285 requested changes

@dataroaring dataroaring dataroaring left review comments

@bobhan1 bobhan1 bobhan1 approved these changes

@gavinchou gavinchou gavinchou left review comments

@w41ter w41ter w41ter left review comments

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
8 participants
@Yukang-Lian @doris-robot @zhannngchen @gavinchou @w41ter @bobhan1 @csun5285 @dataroaring