[Bug] remove_all_remote_rowset core

[yixiutt]

Search before asking

• I had searched in the issues and found no similar issues.

Version

master

What's Wrong?

#0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1 0x00007fb9e97fc859 in __GI_abort () at abort.c:79
#2 0x000055cd04c70ec9 in google::DumpStackTraceAndExit () at src/utilities.cc:160
#3 0x000055cd04c664dd in google::LogMessage::Fail () at src/logging.cc:1650
#4 0x000055cd04c68a19 in google::LogMessage::SendToLog (this=0x7fb95f1f3050) at src/logging.cc:1607
#5 0x000055cd04c66046 in google::LogMessage::Flush (this=0x7fb95f1f3050) at src/logging.cc:1476
#6 0x000055cd04c69089 in google::LogMessageFatal::~LogMessageFatal (this=, __in_chrg=) at src/logging.cc:2226
#7 0x000055ccfe87174b in doris::Tablet::remove_all_remote_rowsets (this=0x61f001ba5090)
at /home/zcp/repo_center/doris_master/be/src/olap/tablet.cpp:1791
#8 0x000055ccfe75ccc4 in doris::TaskWorkerPool::_drop_tablet_worker_thread_callback (this=0x614000055c40)
at /home/zcp/repo_center/doris_master/be/src/agent/task_worker_pool.cpp:448
#9 0x000055ccfe7a4970 in std::__invoke_impl<void, void (doris::TaskWorkerPool::&)(), doris::TaskWorkerPool&> (
__f=@0x603002ab0110: (void (doris::TaskWorkerPool::)(doris::TaskWorkerPool * const)) 0x55ccfe75c350 doris::TaskWorkerPool::_drop_tablet_worker_thread_callback(), __t=@0x603002ab0120: 0x614000055c40) at /var/local/ldb_toolchain/include/c++/11/bits/invoke.h:74
#10 0x000055ccfe7a4109 in std::__invoke_r<void, void (doris::TaskWorkerPool::&)(), doris::TaskWorkerPool&> (
__fn=@0x603002ab0110: (void (doris::TaskWorkerPool::)(doris::TaskWorkerPool * const)) 0x55ccfe75c350 doris::TaskWorkerPool::_drop_tablet_worker_thread_callback()) at /var/local/ldb_toolchain/include/c++/11/bits/invoke.h:111
#11 0x000055ccfe7a3070 in std::Bind_result<void, void (doris::TaskWorkerPool::(doris::TaskWorkerPool))()>::_call<void, , 0ul>(std::tuple<>&&, std::_Index_tuple<0ul>) (this=0x603002ab0110, __args=...) at /var/local/ldb_toolchain/include/c++/11/functional:570
#12 0x000055ccfe7a0adb in std::_Bind_result<void, void (doris::TaskWorkerPool::(doris::TaskWorkerPool))()>::operator()<>() (
this=0x603002ab0110) at /var/local/ldb_toolchain/include/c++/11/functional:629
#13 0x000055ccfe79e622 in std::_invoke_impl<void, std::_Bind_result<void, void (doris::TaskWorkerPool::(doris::TaskWorkerPool))()>&>(std::invoke_other, std::_Bind_result<void, void (doris::TaskWorkerPool::(doris::TaskWorkerPool))()>&) (_f=...)
at /var/local/ldb_toolchain/include/c++/11/bits/invoke.h:61
#14 0x000055ccfe799c49 in std::_invoke_r<void, std::_Bind_result<void, void (doris::TaskWorkerPool::(doris::TaskWorkerPool))()>&>(std::_Bind_result<void, void (doris::TaskWorkerPool::(doris::TaskWorkerPool))()>&) (_fn=...) at /var/local/ldb_toolchain/include/c++/11/bits/invoke.h:111
#15 0x000055ccfe793dfa in std::Function_handler<void (), std::_Bind_result<void, void (doris::TaskWorkerPool::(doris::TaskWorkerPool))()> >::_M_invoke(std::_Any_data const&) (_functor=...) at /var/local/ldb_toolchain/include/c++/11/bits/std_function.h:291
#16 0x000055ccff4c1800 in std::function<void ()>::operator()() const (this=0x606000f7c358)
at /var/local/ldb_toolchain/include/c++/11/bits/std_function.h:560
#17 0x000055ccffcee274 in doris::FunctionRunnable::run (this=0x606000f7c350) at /home/zcp/repo_center/doris_master/be/src/util/threadpool.cpp:45
#18 0x000055ccffce9697 in doris::ThreadPool::dispatch_thread (this=0x61e00000e480)
at /home/zcp/repo_center/doris_master/be/src/util/threadpool.cpp:540
-Type for more, q to quit, c to continue without paging-c
#19 0x000055ccffd0a844 in std::_invoke_impl<void, void (doris::ThreadPool::&)(), doris::ThreadPool&> (_f=@0x603002ab0080: (void (doris::ThreadPool::)(doris::ThreadPool * const)) 0x55ccffce8360 doris::ThreadPool::dispatch_thread(), __t=@0x603002ab0090: 0x61e00000e480) at /var/local/ldb_toolchain/include/c++/11/bits/invoke.h:74
#20 0x000055ccffd0a0e3 in std::_invoke<void (doris::ThreadPool::&)(), doris::ThreadPool&> (_fn=@0x603002ab0080: (void (doris::ThreadPool::)(doris::ThreadPool * const)) 0x55ccffce8360 doris::ThreadPool::dispatch_thread()) at /var/local/ldb_toolchain/include/c++/11/bits/invoke.h:96
#21 0x000055ccffd09482 in std::Bind<void (doris::ThreadPool::(doris::ThreadPool))()>::_call<void, , 0ul>(std::tuple<>&&, std::_Index_tuple<0ul>) (this=0x603002ab0080, __args=...) at /var/local/ldb_toolchain/include/c++/11/functional:420
#22 0x000055ccffd07f85 in std::_Bind<void (doris::ThreadPool::(doris::ThreadPool))()>::operator()<, void>() (this=0x603002ab0080) at /var/local/ldb_toolchain/include/c++/11/functional:503
#23 0x000055ccffd04aec in std::_invoke_impl<void, std::_Bind<void (doris::ThreadPool::(doris::ThreadPool))()>&>(std::invoke_other, std::_Bind<void (doris::ThreadPool::(doris::ThreadPool))()>&) (_f=...) at /var/local/ldb_toolchain/include/c++/11/bits/invoke.h:61
#24 0x000055ccffd02048 in std::_invoke_r<void, std::_Bind<void (doris::ThreadPool::(doris::ThreadPool))()>&>(std::_Bind<void (doris::ThreadPool::(doris::ThreadPool))()>&) (_fn=...) at /var/local/ldb_toolchain/include/c++/11/bits/invoke.h:111
#25 0x000055ccffcfd34b in std::Function_handler<void (), std::_Bind<void (doris::ThreadPool::(doris::ThreadPool))()> >::_M_invoke(std::_Any_data const&) (_functor=...) at /var/local/ldb_toolchain/include/c++/11/bits/std_function.h:291
#26 0x000055ccff4c1800 in std::function<void ()>::operator()() const (this=0x611000dd0818) at /var/local/ldb_toolchain/include/c++/11/bits/std_function.h:560
#27 0x000055ccffcc918c in doris::Thread::supervise_thread (arg=0x611000dd0800) at /home/zcp/repo_center/doris_master/be/src/util/thread.cpp:408
#28 0x00007fb9e97bf609 in start_thread (arg=) at pthread_create.c:477
#29 0x00007fb9e98f9133 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

What You Expected?

fix

How to Reproduce?

No response

Anything Else?

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[xiaokang]

I also encountered this problem. More information is provided here.

• related code

Status Tablet::remove_all_remote_rowsets() {
DCHECK(_state == TABLET_SHUTDOWN);
if (storage_policy_id() == 0) {
return Status::OK();
}
auto tablet_gc_key = REMOTE_TABLET_GC_PREFIX + std::to_string(tablet_id());
auto policy = get_storage_policy(storage_policy_id());
return _data_dir->get_meta()->put(META_COLUMN_FAMILY_INDEX, tablet_gc_key,
std::to_string(policy->resource_id));
}

• gdb information

`
(gdb) f 0
#0 0x00005562d4370a4f in doris::Tablet::remove_all_remote_rowsets (this=0x619000d98a90)
at /mnt/disk2/xiaokang/src/doris/be/src/olap/tablet.cpp:1920
1920 std::to_string(policy->resource_id));
(gdb) p policy
$1 = std::shared_ptrdoris::StoragePolicy (empty) = {
get() = 0x0 # policy is a shared_ptr contains nullptr

(gdb) p _data_dir
$2 = (doris::DataDir *) 0x616000013580 # _data_dir seems to be normal

(gdb) p ((doris::TabletMeta)0x6110003d79c0)._storage_policy_id
$7 = 152677

(gdb) p s_storage_policy_mgr.map
$6 = std::unordered_map with 0 elements # it's a problem
`