[bug](shared scan) Fix use-after-free when enable pipeline shared scanning #26199
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
clang-tidy review says "All clean, LGTM! 👍" |
1 similar comment
|
clang-tidy review says "All clean, LGTM! 👍" |
|
PR approved by anyone and no changes requested. |
|
run buildall |
|
TeamCity be ut coverage result: |
|
(From new machine)TeamCity pipeline, clickbench performance test result: |
github-actions
bot
added
the
approved
|
PR approved by at least one committer and no changes requested. |
Gabriel39
added a commit
to Gabriel39/incubator-doris
that referenced
this pull request
Nov 2, 2023
…nning (apache#26199) When enable shared scan, all scanners will be created by one instance. When the main instance reach eos and quit, all states of it will be released. But other instances are still possible to get block from those scanners. So we must assure scanners will not be dependent on any states of the main instance after it quit.
xiaokang
pushed a commit
that referenced
this pull request
Nov 2, 2023
…nning (#26199) (#26269) When enable shared scan, all scanners will be created by one instance. When the main instance reach eos and quit, all states of it will be released. But other instances are still possible to get block from those scanners. So we must assure scanners will not be dependent on any states of the main instance after it quit.
dutyu
pushed a commit
to dutyu/doris
that referenced
this pull request
Nov 4, 2023
…nning (apache#26199) When enable shared scan, all scanners will be created by one instance. When the main instance reach eos and quit, all states of it will be released. But other instances are still possible to get block from those scanners. So we must assure scanners will not be dependent on any states of the main instance after it quit.
seawinde
pushed a commit
to seawinde/doris
that referenced
this pull request
Nov 13, 2023
…nning (apache#26199) When enable shared scan, all scanners will be created by one instance. When the main instance reach eos and quit, all states of it will be released. But other instances are still possible to get block from those scanners. So we must assure scanners will not be dependent on any states of the main instance after it quit.
wsjz
pushed a commit
to wsjz/incubator-doris
that referenced
this pull request
Nov 19, 2023
…nning (apache#26199) When enable shared scan, all scanners will be created by one instance. When the main instance reach eos and quit, all states of it will be released. But other instances are still possible to get block from those scanners. So we must assure scanners will not be dependent on any states of the main instance after it quit.
gnehil
pushed a commit
to gnehil/doris
that referenced
this pull request
Dec 4, 2023
…nning (apache#26199) (apache#26269) When enable shared scan, all scanners will be created by one instance. When the main instance reach eos and quit, all states of it will be released. But other instances are still possible to get block from those scanners. So we must assure scanners will not be dependent on any states of the main instance after it quit.
XuJianxu
pushed a commit
to XuJianxu/doris
that referenced
this pull request
Dec 14, 2023
…nning (apache#26199) When enable shared scan, all scanners will be created by one instance. When the main instance reach eos and quit, all states of it will be released. But other instances are still possible to get block from those scanners. So we must assure scanners will not be dependent on any states of the main instance after it quit.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Proposed changes
When enable shared scan, all scanners will be created by one instance. When the main instance reach eos and quit, all states of it will be released. But other instances are still possible to get block from those scanners. So we must assure scanners will not be dependent on any states of the main instance after it quit.
core stack:
==2495324==ERROR: AddressSanitizer: heap-use-after-free on address 0x60d00023c478 at pc 0x5602f11666ec bp 0x7f294a6deed0 sp 0x7f294a6deec8
READ of size 8 at 0x60d00023c478 thread T272 (WithoutGroupTas)
#0 0x5602f11666eb in __gnu_cxx::__normal_iterator>>::__normal_iterator(doris::SlotDescriptor* const* const&) /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_iterator.h:1008:20
#1 0x5602f113ceb4 in std::vector>::begin() const /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_vector.h:821:16
#2 0x56030ef15350 in doris::vectorized::ScannerContext::validate_block_schema(doris::vectorized::Block*) /home/zcp/repo_center/doris_master/doris/be/src/vec/exec/scan/scanner_context.cpp:264:21
#3 0x56030f30113e in doris::pipeline::PipScannerContext::get_block_from_queue(doris::RuntimeState*, std::unique_ptr>, bool, int, bool) /home/zcp/repo_center/doris_master/doris/be/src/vec/exec/scan/pip_scanner_context.h:87:13
#4 0x56030f03e279 in doris::vectorized::VScanNode::get_next(doris::RuntimeState*, doris::vectorized::Block*, bool*) /home/zcp/repo_center/doris_master/doris/be/src/vec/exec/scan/vscan_node.cpp:259:5
#5 0x5602f4385d15 in doris::ExecNode::pull(doris::RuntimeState*, doris::vectorized::Block*, bool*) /home/zcp/repo_center/doris_master/doris/be/src/exec/exec_node.h:127:16
#6 0x56032153c14e in doris::Status std::__invoke_impl(std::__invoke_memfun_deref, doris::Status (doris::ExecNode::&)(doris::RuntimeState, doris::vectorized::Block*, bool*), doris::vectorized::VScanNode*&, doris::RuntimeState*&&, doris::vectorized::Block*&&, bool*&&) /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:74:14
#7 0x56032153be91 in std::__invoke_result::type std::__invoke(doris::Status (doris::ExecNode::&)(doris::RuntimeState, doris::vectorized::Block*, bool*), doris::vectorized::VScanNode*&, doris::RuntimeState*&&, doris::vectorized::Block*&&, bool*&&) /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:96:14
#8 0x56032153bdb8 in doris::Status std::_Bind, std::_Placeholder<2>, std::_Placeholder<3>))(doris::RuntimeState*, doris::vectorized::Block*, bool*)>::__call(std::tuple&&, std::_Index_tuple<0ul, 1ul, 2ul, 3ul>) /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/functional:420:11
#9 0x56032153bb39 in doris::Status std::_Bind, std::_Placeholder<2>, std::_Placeholder<3>))(doris::RuntimeState*, doris::vectorized::Block*, bool*)>::operator()(doris::RuntimeState*&&, doris::vectorized::Block*&&, bool*&&) /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/functional:503:17
#10 0x56032153b9d9 in doris::Status std::__invoke_impl, std::_Placeholder<2>, std::_Placeholder<3>))(doris::RuntimeState*, doris::vectorized::Block*, bool*)>&, doris::RuntimeState*, doris::vectorized::Block*, bool*>(std::__invoke_other, std::_Bind, std::_Placeholder<2>, std::_Placeholder<3>))(doris::RuntimeState*, doris::vectorized::Block*, bool*)>&, doris::RuntimeState*&&, doris::vectorized::Block*&&, bool*&&) /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61:14
#11 0x56032153b919 in std::enable_if, std::_Placeholder<2>, std::_Placeholder<3>))(doris::RuntimeState*, doris::vectorized::Block*, bool*)>&, doris::RuntimeState*, doris::vectorized::Block*, bool*>, doris::Status>::type std::__invoke_r, std::_Placeholder<2>, std::_Placeholder<3>))(doris::RuntimeState*, doris::vectorized::Block*, bool*)>&, doris::RuntimeState*, doris::vectorized::Block*, bool*>(std::_Bind, std::_Placeholder<2>, std::_Placeholder<3>))(doris::RuntimeState*, doris::vectorized::Block*, bool*)>&, doris::RuntimeState*&&, doris::vectorized::Block*&&, bool*&&) /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:114:9
#12 0x56032153b479 in std::_Function_handler, std::_Placeholder<2>, std::_Placeholder<3>))(doris::RuntimeState*, doris::vectorized::Block*, bool*)>>::_M_invoke(std::_Any_data const&, doris::RuntimeState*&&, doris::vectorized::Block*&&, bool*&&) /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:291:9
#13 0x5602f4385be2 in std::function::operator()(doris::RuntimeState*, doris::vectorized::Block*, bool*) const /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:560:9
#14 0x5602f4377264 in doris::ExecNode::get_next_after_projects(doris::RuntimeState*, doris::vectorized::Block*, bool*, std::function const&, bool) /home/zcp/repo_center/doris_master/doris/be/src/exec/exec_node.cpp:580:23
#15 0x560320fbdfee in doris::pipeline::SourceOperator::get_block(doris::RuntimeState*, doris::vectorized::Block*, doris::pipeline::SourceState&) /home/zcp/repo_center/doris_master/doris/be/src/pipeline/exec/operator.h:432:9
#16 0x560321eb1d80 in doris::pipeline::PipelineTask::execute(bool*) /home/zcp/repo_center/doris_master/doris/be/src/pipeline/pipeline_task.cpp:282:13
#17 0x56032208bf4e in doris::pipeline::TaskScheduler::_do_work(unsigned long) /home/zcp/repo_center/doris_master/doris/be/src/pipeline/task_scheduler.cpp:262:28
#18 0x56032209ccba in void std::__invoke_impl(std::__invoke_memfun_deref, void (doris::pipeline::TaskScheduler::&)(unsigned long), doris::pipeline::TaskScheduler&, unsigned long&) /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:74:14
#19 0x56032209cad6 in std::__invoke_result::type std::__invoke(void (doris::pipeline::TaskScheduler::&)(unsigned long), doris::pipeline::TaskScheduler&, unsigned long&) /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:96:14
#20 0x56032209ca35 in void std::_Bind::__call(std::tuple<>&&, std::_Index_tuple<0ul, 1ul>) /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/functional:420:11
#21 0x56032209c88f in void std::_Bind::operator()() /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/functional:503:17
#22 0x56032209c796 in void std::__invoke_impl&>(std::__invoke_other, std::_Bind&) /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61:14
#23 0x56032209c708 in std::enable_if&>, void>::type std::__invoke_r&>(std::_Bind&) /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:111:2
#24 0x56032209c31e in std::_Function_handler>::_M_invoke(std::_Any_data const&) /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:291:9
#25 0x5602f104edd6 in std::function::operator()() const /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:560:9
#26 0x5602f4e3be8a in doris::FunctionRunnable::run() /home/zcp/repo_center/doris_master/doris/be/src/util/threadpool.cpp:48:27
#27 0x5602f4e258ce in doris::ThreadPool::dispatch_thread() /home/zcp/repo_center/doris_master/doris/be/src/util/threadpool.cpp:543:24
#28 0x5602f4e50805 in void std::__invoke_impl(std::__invoke_memfun_deref, void (doris::ThreadPool::&)(), doris::ThreadPool&) /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:74:14
#29 0x5602f4e506ae in std::__invoke_result::type std::__invoke(void (doris::ThreadPool::&)(), doris::ThreadPool&) /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:96:14
#30 0x5602f4e50626 in void std::_Bind::__call(std::tuple<>&&, std::_Index_tuple<0ul>) /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/functional:420:11
#31 0x5602f4e504bf in void std::_Bind::operator()() /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/functional:503:17
#32 0x5602f4e503c6 in void std::__invoke_impl&>(std::__invoke_other, std::_Bind&) /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61:14
#33 0x5602f4e50338 in std::enable_if&>, void>::type std::__invoke_r&>(std::_Bind&) /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:111:2
#34 0x5602f4e4fefe in std::_Function_handler>::_M_invoke(std::_Any_data const&) /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:291:9
#35 0x5602f104edd6 in std::function::operator()() const /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:560:9
#36 0x5602f4dede99 in doris::Thread::supervise_thread(void*) /home/zcp/repo_center/doris_master/doris/be/src/util/thread.cpp:494:5
#37 0x7f2b2033e608 in start_thread /build/glibc-SzIz7B/glibc-2.31/nptl/pthread_create.c:477:8
#38 0x7f2b205eb132 in __clone /build/glibc-SzIz7B/glibc-2.31/misc/../sysdeps/unix/sysv/linux/x86_64/clone.S:95
0x60d00023c478 is located 40 bytes inside of 144-byte region [0x60d00023c450,0x60d00023c4e0)
freed by thread T390 (FragmentInstanc) here:
#0 0x5602f0e86d9d in operator delete(void*) (/mnt/hdd01/ci/master-deploy/be/lib/doris_be+0x230dcd9d) (BuildId: ce9045b5bbcc85c1)
#1 0x5602f4050255 in doris::TupleDescriptor* doris::ObjectPool::add(doris::TupleDescriptor*)::'lambda'(void*)::operator()(void*) const /home/zcp/repo_center/doris_master/doris/be/src/common/object_pool.h:40:59
#2 0x5602f40501be in doris::TupleDescriptor* doris::ObjectPool::add(doris::TupleDescriptor*)::'lambda'(void*)::__invoke(void*) /home/zcp/repo_center/doris_master/doris/be/src/common/object_pool.h:40:43
#3 0x5602f114df62 in doris::ObjectPool::clear() /home/zcp/repo_center/doris_master/doris/be/src/common/object_pool.h:57:13
#4 0x5602f46744db in doris::RuntimeState::
RuntimeState() /home/zcp/repo_center/doris_master/doris/be/src/runtime/runtime_state.cpp:222:16__shared_ptr() /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/shared_ptr_base.h:1149:31#5 0x5602f3da264c in std::default_delete::operator()(doris::RuntimeState*) const /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/unique_ptr.h:85:2
#6 0x5602f3e01ead in std::__uniq_ptr_impl>::reset(doris::RuntimeState*) /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/unique_ptr.h:182:4
#7 0x5602f44750ae in std::unique_ptr>::reset(doris::RuntimeState*) /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/unique_ptr.h:456:7
#8 0x5603215739c0 in doris::pipeline::PipelineFragmentContext::
PipelineFragmentContext() /home/zcp/repo_center/doris_master/doris/be/src/pipeline/pipeline_fragment_context.cpp:147:24__shared_count() /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/shared_ptr_base.h:702:11#9 0x5602f41a4eae in void std::destroy_at(doris::pipeline::PipelineFragmentContext*) /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_construct.h:88:15
#10 0x5602f41a4dc9 in void std::allocator_traits>::destroy(std::allocator&, doris::pipeline::PipelineFragmentContext*) /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/alloc_traits.h:533:4
#11 0x5602f41a26a7 in std::_Sp_counted_ptr_inplace, (__gnu_cxx::_Lock_policy)2>::_M_dispose() /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/shared_ptr_base.h:528:2
#12 0x5602f0ea0f61 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/shared_ptr_base.h:168:6
#13 0x5602f0ea0c99 in std::__shared_count<(__gnu_cxx::_Lock_policy)2>::
#14 0x5602f41a050a in std::__shared_ptr::
#15 0x5602f4173104 in std::shared_ptr::
shared_ptr() /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/shared_ptr.h:122:11$_0() /home/zcp/repo_center/doris_master/doris/be/src/runtime/fragment_mgr.cpp:186:51#16 0x5602f412b16f in doris::FragmentMgr::trigger_pipeline_context_report(doris::ReportStatusRequest, std::shared_ptr&&)::$_0::
#17 0x5602f4153c52 in std::_Function_base::_Base_manager&&)::$_0>::_M_destroy(std::_Any_data&, std::integral_constant) /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:174:4
#18 0x5602f4153af9 in std::_Function_base::_Base_manager&&)::$_0>::_M_manager(std::_Any_data&, std::_Any_data const&, std::_Manager_operation) /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:200:8
#19 0x5602f415366b in std::_Function_handler&&)::$_0>::_M_manager(std::_Any_data&, std::_Any_data const&, std::_Manager_operation) /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:283:6
#20 0x5602f0f21dd2 in std::_Function_base::~_Function_base() /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:245:2
#21 0x5602f0f79e94 in std::function::
function() /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:111:11__shared_ptr() /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/shared_ptr_base.h:1149:31#22 0x5602f4e3bee8 in doris::FunctionRunnable::
FunctionRunnable() /home/zcp/repo_center/doris_master/doris/be/src/util/threadpool.cpp:44:7__shared_count() /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/shared_ptr_base.h:702:11#23 0x5602f4e3c062 in void std::destroy_at(doris::FunctionRunnable*) /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_construct.h:88:15
#24 0x5602f4e3bf79 in void std::allocator_traits>::destroy(std::allocator&, doris::FunctionRunnable*) /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/alloc_traits.h:533:4
#25 0x5602f4e3b917 in std::_Sp_counted_ptr_inplace, (__gnu_cxx::_Lock_policy)2>::_M_dispose() /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/shared_ptr_base.h:528:2
#26 0x5602f0ea0f61 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/shared_ptr_base.h:168:6
#27 0x5602f0ea0c99 in std::__shared_count<(__gnu_cxx::_Lock_policy)2>::
#28 0x5602f110beda in std::__shared_ptr::
#29 0x5602f4e31fea in std::__shared_ptr::reset() /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/shared_ptr_base.h:1267:9
Further comments
If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...