[Fix](segment iterator) fix shrink non-char column coredump #36275
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Thank you for your contribution to Apache Doris. Since 2024-03-18, the Document has been moved to doris-website. |
|
run buildall |
|
clang-tidy review says "All clean, LGTM! 👍" |
TPC-H: Total hot run time: 40006 ms |
|
TeamCity be ut coverage result: |
TPC-DS: Total hot run time: 171375 ms |
ClickBench: Total hot run time: 29.86 s |
|
clang-tidy review says "All clean, LGTM! 👍" |
|
run buildall |
|
clang-tidy review says "All clean, LGTM! 👍" |
|
clang-tidy review says "All clean, LGTM! 👍" |
|
TeamCity be ut coverage result: |
TPC-H: Total hot run time: 39627 ms |
TPC-DS: Total hot run time: 174743 ms |
ClickBench: Total hot run time: 30.76 s |
|
run buildall |
|
TeamCity be ut coverage result: |
TPC-H: Total hot run time: 40144 ms |
TPC-DS: Total hot run time: 171531 ms |
ClickBench: Total hot run time: 30.52 s |
xiaokang
reviewed
Jun 16, 2024
| @@ -2264,6 +2264,10 @@ Status SegmentIterator::_next_batch_internal(vectorized::Block* block) { | |||
| } | |||
| _current_return_columns.resize(_schema->columns().size()); | |||
| _converted_column_ids.resize(_schema->columns().size(), 0); | |||
| if (_char_type_idx.empty() && _char_type_idx_no_0.empty()) { | |||
There was a problem hiding this comment.
add comment to explain why move _vec_init_char_column_id() to here.
eldenmoon
reviewed
Jun 17, 2024
| _char_type_idx_no_0.emplace_back(i); | ||
| // The additional deleted filter condition will be in the materialized column at the end of the block. | ||
| // After _output_column_by_sel_idx, it will be erased, so we do not need to shrink it. | ||
| if (i < block->columns()) { |
There was a problem hiding this comment.
why add this if branch? _has_char_type for delete predicate column should always be false?
There was a problem hiding this comment.
_char_type_idx should not include delete predicate column
|
PR approved by anyone and no changes requested. |
github-actions
bot
added
the
approved
|
PR approved by at least one committer and no changes requested. |
airborne12
added a commit
to airborne12/apache-doris
that referenced
this pull request
Jun 18, 2024
…6275) If we execute a delete predicate in an inverted index table and then query something from it, it will cause a core dump in shrink_char_type_column_suffix_zero. This occurs because the delete predicate column ID in _char_type_idx is incorrectly shifted to the result column inserted by the inverted index result. coredump stack like: ``` COULD NOT CREATE A LOGGINGFILE 20240604-145331!F20240604 14:53:31.991016 24178 column.h:134] Cannot get_shrinked_column() column Const(UInt8) *** Check failure stack trace: *** F20240604 14:53:31.991436 24158 column.h:134] Cannot get_shrinked_column() column Const(UInt8) *** Check failure stack trace: *** F20240604 14:53:31.991436 24158 column.h:134] Cannot get_shrinked_column() column Const(UInt8)F20240604 14:53:31.991645 24160 column.h:134] Cannot get_shrinked_column() column Const(UInt8) at /home/zcp/repo_center/doris_release/doris/be/src/vec/core/block.cpp:1126 at /home/zcp/repo_center/doris_release/doris/be/src/olap/rowset/segment_v2/segment_iterator.cpp:2408 at /home/zcp/repo_center/doris_release/doris/be/src/olap/rowset/segment_v2/segment_iterator.cpp:2090 at /home/zcp/repo_center/doris_release/doris/be/src/olap/rowset/beta_rowset_reader.cpp:342 at /home/zcp/repo_center/doris_release/doris/be/src/vec/olap/vcollect_iterator.h:256 at /home/zcp/repo_center/doris_release/doris/be/src/vec/olap/vcollect_iterator.cpp:514 at /home/zcp/repo_center/doris_release/doris/be/src/vec/olap/vcollect_iterator.cpp:493 at /home/zcp/repo_center/doris_release/doris/be/src/vec/olap/vcollect_iterator.cpp:692 at /home/zcp/repo_center/doris_release/doris/be/src/vec/olap/vcollect_iterator.cpp:186 at /home/zcp/repo_center/doris_release/doris/be/src/vec/olap/block_reader.cpp:156 at /home/zcp/repo_center/doris_release/doris/be/src/vec/olap/block_reader.cpp:228 --Type <RET> for more, q to quit, c to continue without paging--c at /home/zcp/repo_center/doris_release/doris/be/src/vec/exec/scan/new_olap_scanner.cpp:227 ```
airborne12
added a commit
to airborne12/apache-doris
that referenced
this pull request
Jun 18, 2024
…6275) If we execute a delete predicate in an inverted index table and then query something from it, it will cause a core dump in shrink_char_type_column_suffix_zero. This occurs because the delete predicate column ID in _char_type_idx is incorrectly shifted to the result column inserted by the inverted index result. coredump stack like: ``` COULD NOT CREATE A LOGGINGFILE 20240604-145331!F20240604 14:53:31.991016 24178 column.h:134] Cannot get_shrinked_column() column Const(UInt8) *** Check failure stack trace: *** F20240604 14:53:31.991436 24158 column.h:134] Cannot get_shrinked_column() column Const(UInt8) *** Check failure stack trace: *** F20240604 14:53:31.991436 24158 column.h:134] Cannot get_shrinked_column() column Const(UInt8)F20240604 14:53:31.991645 24160 column.h:134] Cannot get_shrinked_column() column Const(UInt8) #0 0x00007fa0bf784387 in ?? () from /mnt/disk2/tengjianping/local/ldb_toolchain/lib/libc.so.6 #1 0x0000557dc8e617a0 in google::LogMessage::Flush() () #2 0x0000557dc8e65599 in google::LogMessageFatal::~LogMessageFatal() () apache#3 0x0000557dbe4ec939 in doris::vectorized::IColumn::get_shrinked_column (this=0x7fa094ac16a0) at /home/zcp/repo_center/doris_release/doris/be/src/vec/columns/column.h:134 apache#4 0x0000557dc236d8a2 in doris::vectorized::Block::shrink_char_type_column_suffix_zero (this=0x7f9a372a4b70, char_type_idx=...) at /home/zcp/repo_center/doris_release/doris/be/src/vec/core/block.cpp:1126 apache#5 0x0000557dbee9ea95 in doris::segment_v2::SegmentIterator::_next_batch_internal (this=<optimized out>, block=0x7f9a372a4b70) at /home/zcp/repo_center/doris_release/doris/be/src/olap/rowset/segment_v2/segment_iterator.cpp:2408 apache#6 0x0000557dbee9cc5a in doris::segment_v2::SegmentIterator::next_batch(doris::vectorized::Block*)::$_0::operator()() const (this=<optimized out>) at /home/zcp/repo_center/doris_release/doris/be/src/olap/rowset/segment_v2/segment_iterator.cpp:2090 apache#7 doris::segment_v2::SegmentIterator::next_batch (this=0x1c5d, block=0x6) at /home/zcp/repo_center/doris_release/doris/be/src/olap/rowset/segment_v2/segment_iterator.cpp:2089 apache#8 0x0000557dbeebbb44 in doris::BetaRowsetReader::next_block (this=0x7f9a3a041600, block=0x7f9a372a4b70) at /home/zcp/repo_center/doris_release/doris/be/src/olap/rowset/beta_rowset_reader.cpp:342 apache#9 0x0000557dc8590ddd in doris::vectorized::VCollectIterator::Level0Iterator::_refresh (this=0x7f9a372a1be0) at /home/zcp/repo_center/doris_release/doris/be/src/vec/olap/vcollect_iterator.h:256 apache#10 doris::vectorized::VCollectIterator::Level0Iterator::refresh_current_row (this=0x7f9a372a1be0) at /home/zcp/repo_center/doris_release/doris/be/src/vec/olap/vcollect_iterator.cpp:514 apache#11 0x0000557dc8591055 in doris::vectorized::VCollectIterator::Level0Iterator::ensure_first_row_ref (this=0x1c5d) at /home/zcp/repo_center/doris_release/doris/be/src/vec/olap/vcollect_iterator.cpp:493 apache#12 0x0000557dc85934f2 in doris::vectorized::VCollectIterator::Level1Iterator::ensure_first_row_ref (this=0x7fa090e36c00) at /home/zcp/repo_center/doris_release/doris/be/src/vec/olap/vcollect_iterator.cpp:692 apache#13 0x0000557dc858ddc9 in doris::vectorized::VCollectIterator::build_heap (this=0x7f9d80976dc0, rs_readers=...) at /home/zcp/repo_center/doris_release/doris/be/src/vec/olap/vcollect_iterator.cpp:186 apache#14 0x0000557dc857e808 in doris::vectorized::BlockReader::_init_collect_iter (this=this@entry=0x7f9d80976800, read_params=...) at /home/zcp/repo_center/doris_release/doris/be/src/vec/olap/block_reader.cpp:156 apache#15 0x0000557dc857f64d in doris::vectorized::BlockReader::init (this=<optimized out>, read_params=...) at /home/zcp/repo_center/doris_release/doris/be/src/vec/olap/block_reader.cpp:228 apache#16 0x0000557dc370dc2c in doris::vectorized::NewOlapScanner::open (this=0x7f9a2aa68210, state=<optimized out>) --Type <RET> for more, q to quit, c to continue without paging--c at /home/zcp/repo_center/doris_release/doris/be/src/vec/exec/scan/new_olap_scanner.cpp:227 apache#17 0x0000557dc3718d89 in doris::vectorized::ScannerScheduler::_scanner_scan (ctx=std::shared_ptr<doris::vectorized::ScannerContext> (use count 4, weak count 1) = {...}, scan_task=std::shared_ptr<doris::vectorized::ScanTask> (use count 2, weak count 0) = {...}) at /home/zcp/repo_center/doris_release/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:242 apache#18 0x0000557dc3719b77 in doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1::operator()() const (this=0x0) at /home/zcp/repo_center/doris_release/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:158 apache#19 std::__invoke_impl<void, doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1&>(std::__invoke_other, doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1&) (__f=...) at /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61 apache#20 std::__invoke_r<void, doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1&>(doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1&) (__fn=...) at /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:111 apache#21 std::_Function_handler<void (), doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1>::_M_invoke(std::_Any_data const&) (__functor=...) at /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:291 apache#22 0x0000557dbf3c7788 in doris::ThreadPool::dispatch_thread (this=0x7f9d0bbdc400) at /home/zcp/repo_center/doris_release/doris/be/src/util/threadpool.cpp:543 apache#23 0x0000557dbf3bcb41 in std::function<void ()>::operator()() const (this=0x1471) at /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:560 apache#24 doris::Thread::supervise_thread (arg=0x7f9d0bbdf4c0) at /home/zcp/repo_center/doris_release/doris/be/src/util/thread.cpp:498 apache#25 0x00007fa0bee1dea5 in sem_open () from /mnt/disk2/tengjianping/local/ldb_toolchain/lib/libpthread.so.0 ```
airborne12
added a commit
that referenced
this pull request
Jun 18, 2024
) ## Proposed changes Pick from (#36275)
xiaokang
pushed a commit
that referenced
this pull request
Jun 18, 2024
dataroaring
pushed a commit
that referenced
this pull request
Jun 21, 2024
If we execute a delete predicate in an inverted index table and then query something from it, it will cause a core dump in shrink_char_type_column_suffix_zero. This occurs because the delete predicate column ID in _char_type_idx is incorrectly shifted to the result column inserted by the inverted index result. coredump stack like: ``` COULD NOT CREATE A LOGGINGFILE 20240604-145331!F20240604 14:53:31.991016 24178 column.h:134] Cannot get_shrinked_column() column Const(UInt8) *** Check failure stack trace: *** F20240604 14:53:31.991436 24158 column.h:134] Cannot get_shrinked_column() column Const(UInt8) *** Check failure stack trace: *** F20240604 14:53:31.991436 24158 column.h:134] Cannot get_shrinked_column() column Const(UInt8)F20240604 14:53:31.991645 24160 column.h:134] Cannot get_shrinked_column() column Const(UInt8) #0 0x00007fa0bf784387 in ?? () from /mnt/disk2/tengjianping/local/ldb_toolchain/lib/libc.so.6 #1 0x0000557dc8e617a0 in google::LogMessage::Flush() () #2 0x0000557dc8e65599 in google::LogMessageFatal::~LogMessageFatal() () #3 0x0000557dbe4ec939 in doris::vectorized::IColumn::get_shrinked_column (this=0x7fa094ac16a0) at /home/zcp/repo_center/doris_release/doris/be/src/vec/columns/column.h:134 #4 0x0000557dc236d8a2 in doris::vectorized::Block::shrink_char_type_column_suffix_zero (this=0x7f9a372a4b70, char_type_idx=...) at /home/zcp/repo_center/doris_release/doris/be/src/vec/core/block.cpp:1126 #5 0x0000557dbee9ea95 in doris::segment_v2::SegmentIterator::_next_batch_internal (this=<optimized out>, block=0x7f9a372a4b70) at /home/zcp/repo_center/doris_release/doris/be/src/olap/rowset/segment_v2/segment_iterator.cpp:2408 #6 0x0000557dbee9cc5a in doris::segment_v2::SegmentIterator::next_batch(doris::vectorized::Block*)::$_0::operator()() const (this=<optimized out>) at /home/zcp/repo_center/doris_release/doris/be/src/olap/rowset/segment_v2/segment_iterator.cpp:2090 #7 doris::segment_v2::SegmentIterator::next_batch (this=0x1c5d, block=0x6) at /home/zcp/repo_center/doris_release/doris/be/src/olap/rowset/segment_v2/segment_iterator.cpp:2089 #8 0x0000557dbeebbb44 in doris::BetaRowsetReader::next_block (this=0x7f9a3a041600, block=0x7f9a372a4b70) at /home/zcp/repo_center/doris_release/doris/be/src/olap/rowset/beta_rowset_reader.cpp:342 #9 0x0000557dc8590ddd in doris::vectorized::VCollectIterator::Level0Iterator::_refresh (this=0x7f9a372a1be0) at /home/zcp/repo_center/doris_release/doris/be/src/vec/olap/vcollect_iterator.h:256 #10 doris::vectorized::VCollectIterator::Level0Iterator::refresh_current_row (this=0x7f9a372a1be0) at /home/zcp/repo_center/doris_release/doris/be/src/vec/olap/vcollect_iterator.cpp:514 #11 0x0000557dc8591055 in doris::vectorized::VCollectIterator::Level0Iterator::ensure_first_row_ref (this=0x1c5d) at /home/zcp/repo_center/doris_release/doris/be/src/vec/olap/vcollect_iterator.cpp:493 #12 0x0000557dc85934f2 in doris::vectorized::VCollectIterator::Level1Iterator::ensure_first_row_ref (this=0x7fa090e36c00) at /home/zcp/repo_center/doris_release/doris/be/src/vec/olap/vcollect_iterator.cpp:692 #13 0x0000557dc858ddc9 in doris::vectorized::VCollectIterator::build_heap (this=0x7f9d80976dc0, rs_readers=...) at /home/zcp/repo_center/doris_release/doris/be/src/vec/olap/vcollect_iterator.cpp:186 #14 0x0000557dc857e808 in doris::vectorized::BlockReader::_init_collect_iter (this=this@entry=0x7f9d80976800, read_params=...) at /home/zcp/repo_center/doris_release/doris/be/src/vec/olap/block_reader.cpp:156 #15 0x0000557dc857f64d in doris::vectorized::BlockReader::init (this=<optimized out>, read_params=...) at /home/zcp/repo_center/doris_release/doris/be/src/vec/olap/block_reader.cpp:228 #16 0x0000557dc370dc2c in doris::vectorized::NewOlapScanner::open (this=0x7f9a2aa68210, state=<optimized out>) --Type <RET> for more, q to quit, c to continue without paging--c at /home/zcp/repo_center/doris_release/doris/be/src/vec/exec/scan/new_olap_scanner.cpp:227 #17 0x0000557dc3718d89 in doris::vectorized::ScannerScheduler::_scanner_scan (ctx=std::shared_ptr<doris::vectorized::ScannerContext> (use count 4, weak count 1) = {...}, scan_task=std::shared_ptr<doris::vectorized::ScanTask> (use count 2, weak count 0) = {...}) at /home/zcp/repo_center/doris_release/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:242 #18 0x0000557dc3719b77 in doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1::operator()() const (this=0x0) at /home/zcp/repo_center/doris_release/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:158 #19 std::__invoke_impl<void, doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1&>(std::__invoke_other, doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1&) (__f=...) at /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61 #20 std::__invoke_r<void, doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1&>(doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1&) (__fn=...) at /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:111 #21 std::_Function_handler<void (), doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1>::_M_invoke(std::_Any_data const&) (__functor=...) at /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:291 #22 0x0000557dbf3c7788 in doris::ThreadPool::dispatch_thread (this=0x7f9d0bbdc400) at /home/zcp/repo_center/doris_release/doris/be/src/util/threadpool.cpp:543 #23 0x0000557dbf3bcb41 in std::function<void ()>::operator()() const (this=0x1471) at /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:560 #24 doris::Thread::supervise_thread (arg=0x7f9d0bbdf4c0) at /home/zcp/repo_center/doris_release/doris/be/src/util/thread.cpp:498 #25 0x00007fa0bee1dea5 in sem_open () from /mnt/disk2/tengjianping/local/ldb_toolchain/lib/libpthread.so.0 ```
mongo360
pushed a commit
to mongo360/doris
that referenced
this pull request
Aug 16, 2024
…che#36466) ## Proposed changes Pick from (apache#36275)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Proposed changes
Issue Number: close #xxx
If we execute a delete predicate in an inverted index table and then query something from it, it will cause a core dump in shrink_char_type_column_suffix_zero. This occurs because the delete predicate column ID in _char_type_idx is incorrectly shifted to the result column inserted by the inverted index result.
coredump stack like: