[bug](shared scan) Fix use-after-free when enable pipeline shared scanning

[Gabriel39]

Proposed changes

When enable shared scan, all scanners will be created by one instance. When the main instance reach eos and quit, all states of it will be released. But other instances are still possible to get block from those scanners. So we must assure scanners will not be dependent on any states of the main instance after it quit.

core stack:
==2495324==ERROR: AddressSanitizer: heap-use-after-free on address 0x60d00023c478 at pc 0x5602f11666ec bp 0x7f294a6deed0 sp 0x7f294a6deec8
READ of size 8 at 0x60d00023c478 thread T272 (WithoutGroupTas)
#0 0x5602f11666eb in __gnu_cxx::__normal_iterator>>::__normal_iterator(doris::SlotDescriptor* const* const&) /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_iterator.h:1008:20
#1 0x5602f113ceb4 in std::vector>::begin() const /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_vector.h:821:16
#2 0x56030ef15350 in doris::vectorized::ScannerContext::validate_block_schema(doris::vectorized::Block*) /home/zcp/repo_center/doris_master/doris/be/src/vec/exec/scan/scanner_context.cpp:264:21
#3 0x56030f30113e in doris::pipeline::PipScannerContext::get_block_from_queue(doris::RuntimeState*, std::unique_ptr>, bool, int, bool) /home/zcp/repo_center/doris_master/doris/be/src/vec/exec/scan/pip_scanner_context.h:87:13
#4 0x56030f03e279 in doris::vectorized::VScanNode::get_next(doris::RuntimeState*, doris::vectorized::Block*, bool*) /home/zcp/repo_center/doris_master/doris/be/src/vec/exec/scan/vscan_node.cpp:259:5
#5 0x5602f4385d15 in doris::ExecNode::pull(doris::RuntimeState*, doris::vectorized::Block*, bool*) /home/zcp/repo_center/doris_master/doris/be/src/exec/exec_node.h:127:16
#6 0x56032153c14e in doris::Status std::__invoke_impl(std::__invoke_memfun_deref, doris::Status (doris::ExecNode::&)(doris::RuntimeState, doris::vectorized::Block*, bool*), doris::vectorized::VScanNode*&, doris::RuntimeState*&&, doris::vectorized::Block*&&, bool*&&) /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:74:14
#7 0x56032153be91 in std::__invoke_result::type std::__invoke(doris::Status (doris::ExecNode::&)(doris::RuntimeState, doris::vectorized::Block*, bool*), doris::vectorized::VScanNode*&, doris::RuntimeState*&&, doris::vectorized::Block*&&, bool*&&) /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:96:14
#8 0x56032153bdb8 in doris::Status std::_Bind, std::_Placeholder<2>, std::_Placeholder<3>))(doris::RuntimeState*, doris::vectorized::Block*, bool*)>::__call(std::tuple&&, std::_Index_tuple<0ul, 1ul, 2ul, 3ul>) /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/functional:420:11
#9 0x56032153bb39 in doris::Status std::_Bind, std::_Placeholder<2>, std::_Placeholder<3>))(doris::RuntimeState*, doris::vectorized::Block*, bool*)>::operator()(doris::RuntimeState*&&, doris::vectorized::Block*&&, bool*&&) /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/functional:503:17
#10 0x56032153b9d9 in doris::Status std::__invoke_impl, std::_Placeholder<2>, std::_Placeholder<3>))(doris::RuntimeState*, doris::vectorized::Block*, bool*)>&, doris::RuntimeState*, doris::vectorized::Block*, bool*>(std::__invoke_other, std::_Bind, std::_Placeholder<2>, std::_Placeholder<3>))(doris::RuntimeState*, doris::vectorized::Block*, bool*)>&, doris::RuntimeState*&&, doris::vectorized::Block*&&, bool*&&) /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61:14
#11 0x56032153b919 in std::enable_if, std::_Placeholder<2>, std::_Placeholder<3>))(doris::RuntimeState*, doris::vectorized::Block*, bool*)>&, doris::RuntimeState*, doris::vectorized::Block*, bool*>, doris::Status>::type std::__invoke_r, std::_Placeholder<2>, std::_Placeholder<3>))(doris::RuntimeState*, doris::vectorized::Block*, bool*)>&, doris::RuntimeState*, doris::vectorized::Block*, bool*>(std::_Bind, std::_Placeholder<2>, std::_Placeholder<3>))(doris::RuntimeState*, doris::vectorized::Block*, bool*)>&, doris::RuntimeState*&&, doris::vectorized::Block*&&, bool*&&) /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:114:9
#12 0x56032153b479 in std::_Function_handler, std::_Placeholder<2>, std::_Placeholder<3>))(doris::RuntimeState*, doris::vectorized::Block*, bool*)>>::_M_invoke(std::_Any_data const&, doris::RuntimeState*&&, doris::vectorized::Block*&&, bool*&&) /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:291:9
#13 0x5602f4385be2 in std::function::operator()(doris::RuntimeState*, doris::vectorized::Block*, bool*) const /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:560:9
#14 0x5602f4377264 in doris::ExecNode::get_next_after_projects(doris::RuntimeState*, doris::vectorized::Block*, bool*, std::function const&, bool) /home/zcp/repo_center/doris_master/doris/be/src/exec/exec_node.cpp:580:23
#15 0x560320fbdfee in doris::pipeline::SourceOperator::get_block(doris::RuntimeState*, doris::vectorized::Block*, doris::pipeline::SourceState&) /home/zcp/repo_center/doris_master/doris/be/src/pipeline/exec/operator.h:432:9
#16 0x560321eb1d80 in doris::pipeline::PipelineTask::execute(bool*) /home/zcp/repo_center/doris_master/doris/be/src/pipeline/pipeline_task.cpp:282:13
#17 0x56032208bf4e in doris::pipeline::TaskScheduler::_do_work(unsigned long) /home/zcp/repo_center/doris_master/doris/be/src/pipeline/task_scheduler.cpp:262:28
#18 0x56032209ccba in void std::__invoke_impl(std::__invoke_memfun_deref, void (doris::pipeline::TaskScheduler::&)(unsigned long), doris::pipeline::TaskScheduler&, unsigned long&) /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:74:14
#19 0x56032209cad6 in std::__invoke_result::type std::__invoke(void (doris::pipeline::TaskScheduler::&)(unsigned long), doris::pipeline::TaskScheduler&, unsigned long&) /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:96:14
#20 0x56032209ca35 in void std::_Bind::__call(std::tuple<>&&, std::_Index_tuple<0ul, 1ul>) /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/functional:420:11
#21 0x56032209c88f in void std::_Bind::operator()() /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/functional:503:17
#22 0x56032209c796 in void std::__invoke_impl&>(std::__invoke_other, std::_Bind&) /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61:14
#23 0x56032209c708 in std::enable_if&>, void>::type std::__invoke_r&>(std::_Bind&) /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:111:2
#24 0x56032209c31e in std::_Function_handler>::_M_invoke(std::_Any_data const&) /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:291:9
#25 0x5602f104edd6 in std::function::operator()() const /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:560:9
#26 0x5602f4e3be8a in doris::FunctionRunnable::run() /home/zcp/repo_center/doris_master/doris/be/src/util/threadpool.cpp:48:27
#27 0x5602f4e258ce in doris::ThreadPool::dispatch_thread() /home/zcp/repo_center/doris_master/doris/be/src/util/threadpool.cpp:543:24
#28 0x5602f4e50805 in void std::__invoke_impl(std::__invoke_memfun_deref, void (doris::ThreadPool::&)(), doris::ThreadPool&) /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:74:14
#29 0x5602f4e506ae in std::__invoke_result::type std::__invoke(void (doris::ThreadPool::&)(), doris::ThreadPool&) /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:96:14
#30 0x5602f4e50626 in void std::_Bind::__call(std::tuple<>&&, std::_Index_tuple<0ul>) /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/functional:420:11
#31 0x5602f4e504bf in void std::_Bind::operator()() /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/functional:503:17
#32 0x5602f4e503c6 in void std::__invoke_impl&>(std::__invoke_other, std::_Bind&) /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61:14
#33 0x5602f4e50338 in std::enable_if&>, void>::type std::__invoke_r&>(std::_Bind&) /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:111:2
#34 0x5602f4e4fefe in std::_Function_handler>::_M_invoke(std::_Any_data const&) /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:291:9
#35 0x5602f104edd6 in std::function::operator()() const /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:560:9
#36 0x5602f4dede99 in doris::Thread::supervise_thread(void*) /home/zcp/repo_center/doris_master/doris/be/src/util/thread.cpp:494:5
#37 0x7f2b2033e608 in start_thread /build/glibc-SzIz7B/glibc-2.31/nptl/pthread_create.c:477:8
#38 0x7f2b205eb132 in __clone /build/glibc-SzIz7B/glibc-2.31/misc/../sysdeps/unix/sysv/linux/x86_64/clone.S:95

0x60d00023c478 is located 40 bytes inside of 144-byte region [0x60d00023c450,0x60d00023c4e0)
freed by thread T390 (FragmentInstanc) here:
#0 0x5602f0e86d9d in operator delete(void*) (/mnt/hdd01/ci/master-deploy/be/lib/doris_be+0x230dcd9d) (BuildId: ce9045b5bbcc85c1)
#1 0x5602f4050255 in doris::TupleDescriptor* doris::ObjectPool::add(doris::TupleDescriptor*)::'lambda'(void*)::operator()(void*) const /home/zcp/repo_center/doris_master/doris/be/src/common/object_pool.h:40:59
#2 0x5602f40501be in doris::TupleDescriptor* doris::ObjectPool::add(doris::TupleDescriptor*)::'lambda'(void*)::__invoke(void*) /home/zcp/repo_center/doris_master/doris/be/src/common/object_pool.h:40:43
#3 0x5602f114df62 in doris::ObjectPool::clear() /home/zcp/repo_center/doris_master/doris/be/src/common/object_pool.h:57:13
#4 0x5602f46744db in doris::RuntimeState::RuntimeState() /home/zcp/repo_center/doris_master/doris/be/src/runtime/runtime_state.cpp:222:16
#5 0x5602f3da264c in std::default_delete::operator()(doris::RuntimeState*) const /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/unique_ptr.h:85:2
#6 0x5602f3e01ead in std::__uniq_ptr_impl>::reset(doris::RuntimeState*) /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/unique_ptr.h:182:4
#7 0x5602f44750ae in std::unique_ptr>::reset(doris::RuntimeState*) /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/unique_ptr.h:456:7
#8 0x5603215739c0 in doris::pipeline::PipelineFragmentContext::PipelineFragmentContext() /home/zcp/repo_center/doris_master/doris/be/src/pipeline/pipeline_fragment_context.cpp:147:24
#9 0x5602f41a4eae in void std::destroy_at(doris::pipeline::PipelineFragmentContext*) /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_construct.h:88:15
#10 0x5602f41a4dc9 in void std::allocator_traits>::destroy(std::allocator&, doris::pipeline::PipelineFragmentContext*) /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/alloc_traits.h:533:4
#11 0x5602f41a26a7 in std::_Sp_counted_ptr_inplace, (__gnu_cxx::_Lock_policy)2>::_M_dispose() /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/shared_ptr_base.h:528:2
#12 0x5602f0ea0f61 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/shared_ptr_base.h:168:6
#13 0x5602f0ea0c99 in std::__shared_count<(__gnu_cxx::_Lock_policy)2>::__shared_count() /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/shared_ptr_base.h:702:11
#14 0x5602f41a050a in std::__shared_ptr::__shared_ptr() /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/shared_ptr_base.h:1149:31
#15 0x5602f4173104 in std::shared_ptr::shared_ptr() /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/shared_ptr.h:122:11
#16 0x5602f412b16f in doris::FragmentMgr::trigger_pipeline_context_report(doris::ReportStatusRequest, std::shared_ptr&&)::$_0::$_0() /home/zcp/repo_center/doris_master/doris/be/src/runtime/fragment_mgr.cpp:186:51
#17 0x5602f4153c52 in std::_Function_base::_Base_manager&&)::$_0>::_M_destroy(std::_Any_data&, std::integral_constant) /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:174:4
#18 0x5602f4153af9 in std::_Function_base::_Base_manager&&)::$_0>::_M_manager(std::_Any_data&, std::_Any_data const&, std::_Manager_operation) /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:200:8
#19 0x5602f415366b in std::_Function_handler&&)::$_0>::_M_manager(std::_Any_data&, std::_Any_data const&, std::_Manager_operation) /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:283:6
#20 0x5602f0f21dd2 in std::_Function_base::~_Function_base() /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:245:2
#21 0x5602f0f79e94 in std::function::function() /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:111:11
#22 0x5602f4e3bee8 in doris::FunctionRunnable::FunctionRunnable() /home/zcp/repo_center/doris_master/doris/be/src/util/threadpool.cpp:44:7
#23 0x5602f4e3c062 in void std::destroy_at(doris::FunctionRunnable*) /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_construct.h:88:15
#24 0x5602f4e3bf79 in void std::allocator_traits>::destroy(std::allocator&, doris::FunctionRunnable*) /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/alloc_traits.h:533:4
#25 0x5602f4e3b917 in std::_Sp_counted_ptr_inplace, (__gnu_cxx::_Lock_policy)2>::_M_dispose() /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/shared_ptr_base.h:528:2
#26 0x5602f0ea0f61 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/shared_ptr_base.h:168:6
#27 0x5602f0ea0c99 in std::__shared_count<(__gnu_cxx::_Lock_policy)2>::__shared_count() /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/shared_ptr_base.h:702:11
#28 0x5602f110beda in std::__shared_ptr::__shared_ptr() /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/shared_ptr_base.h:1149:31
#29 0x5602f4e31fea in std::__shared_ptr::reset() /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/shared_ptr_base.h:1267:9

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

[github-actions]

clang-tidy review says "All clean, LGTM! 👍"

[github-actions]

clang-tidy review says "All clean, LGTM! 👍"

[mrhhsg]

LGTM

[github-actions]

PR approved by anyone and no changes requested.

[Gabriel39]

run buildall

[doris-robot]

TeamCity be ut coverage result:
Function Coverage: 37.34% (8468/22681)
Line Coverage: 29.66% (68551/231159)
Region Coverage: 28.23% (35600/126112)
Branch Coverage: 25.10% (18227/72622)
Coverage Report: http://coverage.selectdb-in.cc/coverage/d5ba59cf169aa569ea6c181cb9c62e623a037088_d5ba59cf169aa569ea6c181cb9c62e623a037088/report/index.html

[doris-robot]

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 48.54 seconds
stream load tsv: 556 seconds loaded 74807831229 Bytes, about 128 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.9 seconds inserted 10000000 Rows, about 346K ops/s
storage size: 17162261907 Bytes

[HappenLee]

LGTM

[github-actions]

PR approved by at least one committer and no changes requested.