[Refactor] (inverted index) Refactor Inverted index file writer #41625
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Thank you for your contribution to Apache Doris. Since 2024-03-18, the Document has been moved to doris-website. |
csun5285
force-pushed
the
inverted_index_file_writer
branch
from
30b25d7 to
c4a3510
Compare
|
run buildall |
|
clang-tidy review says "All clean, LGTM! 👍" |
2 similar comments
|
clang-tidy review says "All clean, LGTM! 👍" |
|
clang-tidy review says "All clean, LGTM! 👍" |
|
run buildall |
|
TeamCity be ut coverage result: |
TPC-H: Total hot run time: 41171 ms |
TPC-DS: Total hot run time: 192952 ms |
ClickBench: Total hot run time: 31.97 s |
|
run buildall |
|
clang-tidy review says "All clean, LGTM! 👍" |
|
TeamCity be ut coverage result: |
TPC-H: Total hot run time: 40780 ms |
TPC-DS: Total hot run time: 192369 ms |
ClickBench: Total hot run time: 32.09 s |
|
run buildall |
|
clang-tidy review says "All clean, LGTM! 👍" |
|
TeamCity be ut coverage result: |
TPC-H: Total hot run time: 40815 ms |
TPC-DS: Total hot run time: 191870 ms |
ClickBench: Total hot run time: 32.17 s |
airborne12
reviewed
Oct 10, 2024
csun5285
force-pushed
the
inverted_index_file_writer
branch
from
6bf48d0 to
4f8864b
Compare
|
run buildall |
|
clang-tidy review says "All clean, LGTM! 👍" |
|
TeamCity be ut coverage result: |
|
clang-tidy review says "All clean, LGTM! 👍" |
|
run buildall |
|
PR approved by at least one committer and no changes requested. |
|
PR approved by anyone and no changes requested. |
16 tasks
airborne12
pushed a commit
that referenced
this pull request
Nov 4, 2024
### What problem does this PR solve? <!-- You need to clearly describe your PR in this part: 1. What problem was fixed (it's best to include specific error reporting information). How it was fixed. 2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be. 3. What features were added. Why this function was added. 4. Which codes were refactored and why this part of the code was refactored. 5. Which functions were optimized and what is the difference before and after the optimization. The description of the PR needs to enable reviewers to quickly and clearly understand the logic of the code modification. --> Introduced by #41625 <!-- If there are related issues, please fill in the issue number. - If you want the issue to be closed after the PR is merged, please use "close #12345". Otherwise, use "ref #12345" --> <!-- If this PR is followup a preivous PR, for example, fix the bug that introduced by a related PR, link the PR here --> Related PR: #41625 Problem Summary: In #41625, the index compaction process was modified. During index compaction, the output rowset has not yet been built, so the output rowset cannot be accessed at this point. ### Check List (For Committer) - Test <!-- At least one of them must be included. --> - [x] Regression test - [ ] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No colde files have been changed. - [ ] Other reason <!-- Add your reason? --> - Behavior changed: - [x] No. - [ ] Yes. <!-- Explain the behavior change --> - Does this need documentation? - [x] No. - [ ] Yes. <!-- Add document PR link here. eg: apache/doris-website#1214 --> - Release note <!-- bugfix, feat, behavior changed need a release note --> <!-- Add one line release note for this PR. --> None ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into -->
This was referenced Nov 4, 2024
[Fix](segment compaction) fix error using of inverted index file writer in segment compaction
#43114
Merged
airborne12
added a commit
that referenced
this pull request
Nov 4, 2024
…er in segment compaction (#43114) ### What problem does this PR solve? <!-- You need to clearly describe your PR in this part: 1. What problem was fixed (it's best to include specific error reporting information). How it was fixed. 2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be. 3. What features were added. Why this function was added. 4. Which codes were refactored and why this part of the code was refactored. 5. Which functions were optimized and what is the difference before and after the optimization. The description of the PR needs to enable reviewers to quickly and clearly understand the logic of the code modification. --> <!-- If there are related issues, please fill in the issue number. - If you want the issue to be closed after the PR is merged, please use "close #12345". Otherwise, use "ref #12345" --> <!-- If this PR is followup a preivous PR, for example, fix the bug that introduced by a related PR, link the PR here --> Related PR: #41625 Problem Summary: Fix BUAF problem, stack like this ``` stack trace: *** 0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at /home/zcp/repo_center/doris_master/doris/be/src/common/signal_handler.h:421 1# 0x00007F5370ABD520 in /lib/x86_64-linux-gnu/libc.so.6 2# pthread_kill at ./nptl/pthread_kill.c:89 3# raise at ../sysdeps/posix/raise.c:27 4# abort at ./stdlib/abort.c:81 5# _gnu_cxx::_verbose_terminate_handler() [clone .cold] at ../../../../libstdc+-v3/libsupc+/vterminate.cc:75 6# _cxxabiv1::_terminate(void ()) at ../../../../libstdc+-v3/libsupc+/eh_terminate.cc:48 7# 0x000055D9E015B681 in /mnt/hdd01/ci/master-deploy/be/lib/doris_be 8# 0x000055D9E015B7D4 in /mnt/hdd01/ci/master-deploy/be/lib/doris_be 9# 0x000055D9E015BBC6 in /mnt/hdd01/ci/master-deploy/be/lib/doris_be 10# void fmt::v7::detail::buffer::append(char const*, char const*) in /mnt/hdd01/ci/master-deploy/be/lib/doris_be 11# char const* fmt::v7::detail::parse_replacement_field, char, fmt::v7::basic_format_context, char> >&>(char const*, char const*, fmt::v7::detail::format_handler, char, fmt::v7::basic_format_context, char> >&) in /mnt/hdd01/ci/master-deploy/be/lib/doris_be 12# void fmt::v7::detail::vformat_to(fmt::v7::detail::buffer&, fmt::v7::basic_string_view, fmt::v7::basic_format_args::type>, fmt::v7::type_identity::type> >, fmt::v7::detail::locale_ref) in /mnt/hdd01/ci/master-deploy/be/lib/doris_be 13# fmt::v7::detail::vformat[abi:cxx11](fmt::v7::basic_string_view, fmt::v7::format_args) in /mnt/hdd01/ci/master-deploy/be/lib/doris_be 14# doris::segment_v2::InvertedIndexDescriptor::get_temporary_index_path[abi:cxx11](std::basic_string_view >, std::basic_string_view >, long, long, std::basic_string_view >) at /home/zcp/repo_center/doris_master/doris/be/src/olap/rowset/segment_v2/inverted_index_desc.cpp:35 15# doris::segment_v2::InvertedIndexFileWriter::open(doris::TabletIndex const*) at /home/zcp/repo_center/doris_master/doris/be/src/olap/rowset/segment_v2/inverted_index_file_writer.cpp:45 16# doris::segment_v2::InvertedIndexColumnWriterImpl<(doris::FieldType)7>::init_bkd_index() at /home/zcp/repo_center/doris_master/doris/be/src/olap/rowset/segment_v2/inverted_index_writer.cpp:146 17# doris::segment_v2::InvertedIndexColumnWriterImpl<(doris::FieldType)7>::init() at /home/zcp/repo_center/doris_master/doris/be/src/olap/rowset/segment_v2/inverted_index_writer.cpp:116 18# doris::segment_v2::InvertedIndexColumnWriter::create(doris::Field const*, std::unique_ptr >, doris::segment_v2::InvertedIndexFileWriter, doris::TabletIndex const*) at /home/zcp/repo_center/doris_master/doris/be/src/olap/rowset/segment_v2/inverted_index_writer.cpp:698 19# doris::segment_v2::ScalarColumnWriter::init() at /home/zcp/repo_center/doris_master/doris/be/src/olap/rowset/segment_v2/column_writer.cpp:483 20# doris::segment_v2::SegmentWriter::_create_column_writer(unsigned int, doris::TabletColumn const&, std::shared_ptr const&) at /home/zcp/repo_center/doris_master/doris/be/src/olap/rowset/segment_v2/segment_writer.cpp:258 21# doris::segment_v2::SegmentWriter::_create_writers(std::shared_ptr const&, std::vector > const&) at /home/zcp/repo_center/doris_master/doris/be/src/olap/rowset/segment_v2/segment_writer.cpp:307 22# doris::segment_v2::SegmentWriter::init(std::vector > const&, bool) at /home/zcp/repo_center/doris_master/doris/be/src/olap/rowset/segment_v2/segment_writer.cpp:276 23# doris::SegcompactionWorker::_do_compact_segments(std::shared_ptr, std::allocator > > >) in /mnt/hdd01/ci/master-deploy/be/lib/doris_be 24# doris::SegcompactionWorker::compact_segments(std::shared_ptr, std::allocator > > >) at /home/zcp/repo_center/doris_master/doris/be/src/olap/rowset/segcompaction.cpp:354 25# doris::StorageEngine::_handle_seg_compaction(std::shared_ptr, std::shared_ptr, std::allocator > > >, unsigned long) at /home/zcp/repo_center/doris_master/doris/be/src/olap/olap_server.cpp:1121 26# std::_Function_handler, std::shared_ptr, std::allocator > > >)::$_0>::_M_invoke(std::_Any_data const&) at /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:291 27# doris::ThreadPool::dispatch_thread() in /mnt/hdd01/ci/master-deploy/be/lib/doris_be 28# doris::Thread::supervise_thread(void*) at /home/zcp/repo_center/doris_master/doris/be/src/util/thread.cpp:499 29# start_thread at ./nptl/pthread_create.c:442 30# 0x00007F5370BA1850 at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:83 172.20.50.120 last coredump sql: last SQL query not found ``` ### Check List (For Committer) - Test <!-- At least one of them must be included. --> - [ ] Regression test - [x] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No colde files have been changed. - [ ] Other reason <!-- Add your reason? --> - Behavior changed: - [x] No. - [ ] Yes. <!-- Explain the behavior change --> - Does this need documentation? - [x] No. - [ ] Yes. <!-- Add document PR link here. eg: apache/doris-website#1214 --> - Release note <!-- bugfix, feat, behavior changed need a release note --> <!-- Add one line release note for this PR. --> None ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into -->
airborne12
added a commit
that referenced
this pull request
Nov 4, 2024
### What problem does this PR solve? <!-- You need to clearly describe your PR in this part: 1. What problem was fixed (it's best to include specific error reporting information). How it was fixed. 2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be. 3. What features were added. Why this function was added. 4. Which codes were refactored and why this part of the code was refactored. 5. Which functions were optimized and what is the difference before and after the optimization. The description of the PR needs to enable reviewers to quickly and clearly understand the logic of the code modification. --> <!-- If there are related issues, please fill in the issue number. - If you want the issue to be closed after the PR is merged, please use "close #12345". Otherwise, use "ref #12345" --> <!-- If this PR is followup a preivous PR, for example, fix the bug that introduced by a related PR, link the PR here --> Related PR: #41625 Problem Summary: Refine case of index compaction, DO NOT need so much data. ### Check List (For Committer) - Test <!-- At least one of them must be included. --> - [x] Regression test - [ ] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No colde files have been changed. - [ ] Other reason <!-- Add your reason? --> - Behavior changed: - [x] No. - [ ] Yes. <!-- Explain the behavior change --> - Does this need documentation? - [x] No. - [ ] Yes. <!-- Add document PR link here. eg: apache/doris-website#1214 --> - Release note <!-- bugfix, feat, behavior changed need a release note --> <!-- Add one line release note for this PR. --> None ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into -->
16 tasks
csun5285
added a commit
to csun5285/doris
that referenced
this pull request
Nov 8, 2024
…he#41625) ## Proposed changes 1. After the normal segment is flushed, the `close_inverted_index` is directly called to write the final composite file. 2. During compaction, in the first step, the `segment writer `writes the `bkd index` while writing normal data. In the second step, the` index compaction` writes the `string index`. In the third step, `close_inverted_index` is uniformly called for all indexes to write the final files. 3. The rowset writer uses `InvertedIndexFileCollection` to store all inverted index file writers, ensuring their lifecycle exists throughout the entire writing or compaction process. 4. When the rowset writer generates the final rowset through `build(rowset)`, it can retrieve the index file sizes from the `InvertedIndexFileCollection` and record them in the rowset meta.
This was referenced Nov 8, 2024
Merged
16 tasks
airborne12
pushed a commit
that referenced
this pull request
Nov 11, 2024
csun5285
added a commit
to csun5285/doris
that referenced
this pull request
Nov 11, 2024
### What problem does this PR solve? <!-- You need to clearly describe your PR in this part: 1. What problem was fixed (it's best to include specific error reporting information). How it was fixed. 2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be. 3. What features were added. Why this function was added. 4. Which codes were refactored and why this part of the code was refactored. 5. Which functions were optimized and what is the difference before and after the optimization. The description of the PR needs to enable reviewers to quickly and clearly understand the logic of the code modification. --> Introduced by apache#41625 <!-- If there are related issues, please fill in the issue number. - If you want the issue to be closed after the PR is merged, please use "close apache#12345". Otherwise, use "ref apache#12345" --> <!-- If this PR is followup a preivous PR, for example, fix the bug that introduced by a related PR, link the PR here --> Related PR: apache#41625 Problem Summary: In apache#41625, the index compaction process was modified. During index compaction, the output rowset has not yet been built, so the output rowset cannot be accessed at this point. ### Check List (For Committer) - Test <!-- At least one of them must be included. --> - [x] Regression test - [ ] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No colde files have been changed. - [ ] Other reason <!-- Add your reason? --> - Behavior changed: - [x] No. - [ ] Yes. <!-- Explain the behavior change --> - Does this need documentation? - [x] No. - [ ] Yes. <!-- Add document PR link here. eg: apache/doris-website#1214 --> - Release note <!-- bugfix, feat, behavior changed need a release note --> <!-- Add one line release note for this PR. --> None ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into -->
airborne12
added a commit
to airborne12/apache-doris
that referenced
this pull request
Nov 11, 2024
…er in segment compaction (apache#43114) ### What problem does this PR solve? <!-- You need to clearly describe your PR in this part: 1. What problem was fixed (it's best to include specific error reporting information). How it was fixed. 2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be. 3. What features were added. Why this function was added. 4. Which codes were refactored and why this part of the code was refactored. 5. Which functions were optimized and what is the difference before and after the optimization. The description of the PR needs to enable reviewers to quickly and clearly understand the logic of the code modification. --> <!-- If there are related issues, please fill in the issue number. - If you want the issue to be closed after the PR is merged, please use "close apache#12345". Otherwise, use "ref apache#12345" --> <!-- If this PR is followup a preivous PR, for example, fix the bug that introduced by a related PR, link the PR here --> Related PR: apache#41625 Problem Summary: Fix BUAF problem, stack like this ``` stack trace: *** 0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at /home/zcp/repo_center/doris_master/doris/be/src/common/signal_handler.h:421 1# 0x00007F5370ABD520 in /lib/x86_64-linux-gnu/libc.so.6 2# pthread_kill at ./nptl/pthread_kill.c:89 3# raise at ../sysdeps/posix/raise.c:27 4# abort at ./stdlib/abort.c:81 5# _gnu_cxx::_verbose_terminate_handler() [clone .cold] at ../../../../libstdc+-v3/libsupc+/vterminate.cc:75 6# _cxxabiv1::_terminate(void ()) at ../../../../libstdc+-v3/libsupc+/eh_terminate.cc:48 7# 0x000055D9E015B681 in /mnt/hdd01/ci/master-deploy/be/lib/doris_be 8# 0x000055D9E015B7D4 in /mnt/hdd01/ci/master-deploy/be/lib/doris_be 9# 0x000055D9E015BBC6 in /mnt/hdd01/ci/master-deploy/be/lib/doris_be 10# void fmt::v7::detail::buffer::append(char const*, char const*) in /mnt/hdd01/ci/master-deploy/be/lib/doris_be 11# char const* fmt::v7::detail::parse_replacement_field, char, fmt::v7::basic_format_context, char> >&>(char const*, char const*, fmt::v7::detail::format_handler, char, fmt::v7::basic_format_context, char> >&) in /mnt/hdd01/ci/master-deploy/be/lib/doris_be 12# void fmt::v7::detail::vformat_to(fmt::v7::detail::buffer&, fmt::v7::basic_string_view, fmt::v7::basic_format_args::type>, fmt::v7::type_identity::type> >, fmt::v7::detail::locale_ref) in /mnt/hdd01/ci/master-deploy/be/lib/doris_be 13# fmt::v7::detail::vformat[abi:cxx11](fmt::v7::basic_string_view, fmt::v7::format_args) in /mnt/hdd01/ci/master-deploy/be/lib/doris_be 14# doris::segment_v2::InvertedIndexDescriptor::get_temporary_index_path[abi:cxx11](std::basic_string_view >, std::basic_string_view >, long, long, std::basic_string_view >) at /home/zcp/repo_center/doris_master/doris/be/src/olap/rowset/segment_v2/inverted_index_desc.cpp:35 15# doris::segment_v2::InvertedIndexFileWriter::open(doris::TabletIndex const*) at /home/zcp/repo_center/doris_master/doris/be/src/olap/rowset/segment_v2/inverted_index_file_writer.cpp:45 16# doris::segment_v2::InvertedIndexColumnWriterImpl<(doris::FieldType)7>::init_bkd_index() at /home/zcp/repo_center/doris_master/doris/be/src/olap/rowset/segment_v2/inverted_index_writer.cpp:146 17# doris::segment_v2::InvertedIndexColumnWriterImpl<(doris::FieldType)7>::init() at /home/zcp/repo_center/doris_master/doris/be/src/olap/rowset/segment_v2/inverted_index_writer.cpp:116 18# doris::segment_v2::InvertedIndexColumnWriter::create(doris::Field const*, std::unique_ptr >, doris::segment_v2::InvertedIndexFileWriter, doris::TabletIndex const*) at /home/zcp/repo_center/doris_master/doris/be/src/olap/rowset/segment_v2/inverted_index_writer.cpp:698 19# doris::segment_v2::ScalarColumnWriter::init() at /home/zcp/repo_center/doris_master/doris/be/src/olap/rowset/segment_v2/column_writer.cpp:483 20# doris::segment_v2::SegmentWriter::_create_column_writer(unsigned int, doris::TabletColumn const&, std::shared_ptr const&) at /home/zcp/repo_center/doris_master/doris/be/src/olap/rowset/segment_v2/segment_writer.cpp:258 21# doris::segment_v2::SegmentWriter::_create_writers(std::shared_ptr const&, std::vector > const&) at /home/zcp/repo_center/doris_master/doris/be/src/olap/rowset/segment_v2/segment_writer.cpp:307 22# doris::segment_v2::SegmentWriter::init(std::vector > const&, bool) at /home/zcp/repo_center/doris_master/doris/be/src/olap/rowset/segment_v2/segment_writer.cpp:276 23# doris::SegcompactionWorker::_do_compact_segments(std::shared_ptr, std::allocator > > >) in /mnt/hdd01/ci/master-deploy/be/lib/doris_be 24# doris::SegcompactionWorker::compact_segments(std::shared_ptr, std::allocator > > >) at /home/zcp/repo_center/doris_master/doris/be/src/olap/rowset/segcompaction.cpp:354 25# doris::StorageEngine::_handle_seg_compaction(std::shared_ptr, std::shared_ptr, std::allocator > > >, unsigned long) at /home/zcp/repo_center/doris_master/doris/be/src/olap/olap_server.cpp:1121 26# std::_Function_handler, std::shared_ptr, std::allocator > > >)::$_0>::_M_invoke(std::_Any_data const&) at /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:291 27# doris::ThreadPool::dispatch_thread() in /mnt/hdd01/ci/master-deploy/be/lib/doris_be 28# doris::Thread::supervise_thread(void*) at /home/zcp/repo_center/doris_master/doris/be/src/util/thread.cpp:499 29# start_thread at ./nptl/pthread_create.c:442 30# 0x00007F5370BA1850 at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:83 172.20.50.120 last coredump sql: last SQL query not found ``` ### Check List (For Committer) - Test <!-- At least one of them must be included. --> - [ ] Regression test - [x] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No colde files have been changed. - [ ] Other reason <!-- Add your reason? --> - Behavior changed: - [x] No. - [ ] Yes. <!-- Explain the behavior change --> - Does this need documentation? - [x] No. - [ ] Yes. <!-- Add document PR link here. eg: apache/doris-website#1214 --> - Release note <!-- bugfix, feat, behavior changed need a release note --> <!-- Add one line release note for this PR. --> None ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into -->
airborne12
added a commit
to airborne12/apache-doris
that referenced
this pull request
Nov 11, 2024
…e in segment compaction (apache#43477) Related PR: apache#41625 Problem Summary: Fix coredump when doing segcompaction after refactor of inverted index file writer. Co-authored-by: airborne12 <jiangkai@selectdb.com>
This was referenced Nov 14, 2024
csun5285
pushed a commit
to csun5285/doris
that referenced
this pull request
Nov 21, 2024
### What problem does this PR solve? <!-- You need to clearly describe your PR in this part: 1. What problem was fixed (it's best to include specific error reporting information). How it was fixed. 2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be. 3. What features were added. Why this function was added. 4. Which codes were refactored and why this part of the code was refactored. 5. Which functions were optimized and what is the difference before and after the optimization. The description of the PR needs to enable reviewers to quickly and clearly understand the logic of the code modification. --> <!-- If there are related issues, please fill in the issue number. - If you want the issue to be closed after the PR is merged, please use "close apache#12345". Otherwise, use "ref apache#12345" --> <!-- If this PR is followup a preivous PR, for example, fix the bug that introduced by a related PR, link the PR here --> Related PR: apache#41625 Problem Summary: Refine case of index compaction, DO NOT need so much data. ### Check List (For Committer) - Test <!-- At least one of them must be included. --> - [x] Regression test - [ ] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No colde files have been changed. - [ ] Other reason <!-- Add your reason? --> - Behavior changed: - [x] No. - [ ] Yes. <!-- Explain the behavior change --> - Does this need documentation? - [x] No. - [ ] Yes. <!-- Add document PR link here. eg: apache/doris-website#1214 --> - Release note <!-- bugfix, feat, behavior changed need a release note --> <!-- Add one line release note for this PR. --> None ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into -->
16 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Proposed changes
close_inverted_indexis directly called to write the final composite file.segment writerwrites thebkd indexwhile writing normal data. In the second step, theindex compactionwrites thestring index. In the third step,close_inverted_indexis uniformly called for all indexes to write the final files.InvertedIndexFileCollectionto store all inverted index file writers, ensuring their lifecycle exists throughout the entire writing or compaction process.build(rowset), it can retrieve the index file sizes from theInvertedIndexFileCollectionand record them in the rowset meta.