release 3.2 storage crash during balance data test

[kikimo]

Please check the FAQ documentation before raising an issue

Describe the bug (required)

the crash stack:

[New LWP 6225]
[New LWP 19]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
--Type <RET> for more, q to quit, c to continue without paging--
Core was generated by `/root/src/nebula-comm-3.2/build/bin/nebula-storaged --flagfile /root/nebula-cha'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  __GI___pthread_mutex_lock (mutex=0x632d616c756266fe) at ../nptl/pthread_mutex_lock.c:67
67	../nptl/pthread_mutex_lock.c: No such file or directory.
[Current thread is 1 (Thread 0x7f0e4d1ff700 (LWP 79))]
(gdb) bt
#0  __GI___pthread_mutex_lock (mutex=0x632d616c756266fe) at ../nptl/pthread_mutex_lock.c:67
#1  0x0000000001c40541 in __gthread_mutex_lock (__mutex=0x632d616c756266fe) at /usr/include/x86_64-linux-gnu/c++/9/bits/gthr-default.h:749
#2  std::mutex::lock (this=0x632d616c756266fe) at /usr/include/c++/9/bits/std_mutex.h:100
#3  std::lock_guard<std::mutex>::lock_guard (__m=..., this=<synthetic pointer>) at /usr/include/c++/9/bits/std_mutex.h:159
#4  nebula::raftex::RaftPart::isLearner (this=0x632d616c7562656e) at /data/src/nebula-comm-3.2/src/kvstore/raftex/RaftPart.h:116
#5  nebula::kvstore::NebulaStore::removeSpace (this=0x7f0e71c66400, spaceId=1, isListener=<optimized out>)
    at /data/src/nebula-comm-3.2/src/kvstore/NebulaStore.cpp:509
#6  0x00000000013bc015 in nebula::meta::MetaClient::diff (this=0x7f0e71c98000, oldCache=..., newCache=...)
    at /data/src/nebula-comm-3.2/src/clients/meta/MetaClient.cpp:1027
#7  0x00000000013d1213 in nebula::meta::MetaClient::loadData (this=0x7f0e71c98000) at /data/src/nebula-comm-3.2/src/clients/meta/MetaClient.cpp:420
#8  0x00000000013d1fb8 in nebula::meta::MetaClient::heartBeatThreadFunc (this=0x7f0e71c98000)
    at /data/src/nebula-comm-3.2/src/clients/meta/MetaClient.cpp:198
#9  0x000000000146da95 in std::function<void ()>::operator()() const (this=<optimized out>) at /usr/include/c++/9/bits/std_function.h:683
#10 nebula::thread::GenericWorker::addDelayTask<void (nebula::meta::MetaClient::*)(), nebula::meta::MetaClient*>(unsigned long, void (nebula::meta::MetaClient::*&&)(), nebula::meta::MetaClient*&&)::{lambda()#1}::operator()() const (this=0x7f0e4c429700)
    at /data/src/nebula-comm-3.2/src/common/thread/GenericWorker.h:217
#11 0x0000000001b73d8f in std::function<void ()>::operator()() const (this=0x7f0e5221d248) at /usr/include/c++/9/bits/std_function.h:683
#12 nebula::thread::GenericWorker::<lambda(int, int16_t, void*)>::operator() (__closure=0x0, fd=<optimized out>, arg=0x7f0e5221d230)
    at /data/src/nebula-comm-3.2/src/common/thread/GenericWorker.cpp:128
#13 nebula::thread::GenericWorker::<lambda(int, int16_t, void*)>::_FUN(int, int16_t, void *) ()
    at /data/src/nebula-comm-3.2/src/common/thread/GenericWorker.cpp:132
#14 0x00000000026551b5 in ?? ()
#15 0x000000000265581f in event_base_loop ()
#16 0x0000000001b743aa in std::__invoke_impl<void, void (*)(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::function<void ()> const&), std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::_Bind<void (nebula::thread::GenericWorker::*(nebula::thread::GenericWorker*))()> >(std::__invoke_other, void (*&&)(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::function<void ()> const&), std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&&, std::_Bind<void (nebula::thread::GenericWorker::*(nebula::thread::GenericWorker*))()>&&) (__f=
    @0x7f0e4e020900: 0x1b36580 <nebula::thread::NamedThread::hook(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::function<void ()> const&)>) at /usr/include/c++/9/bits/std_function.h:87
#17 std::__invoke<void (*)(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::function<void ()> const&), std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::_Bind<void (nebula::thread::GenericWorker::*(nebula::thread::GenericWorker*))()> >(void (*&&)(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::function<void ()> const&), std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&&, std::_Bind<void (nebula::thread::GenericWorker::*(nebula::thread::GenericWorker*))()>&&) (
    __fn=@0x7f0e4e020900: 0x1b36580 <nebula::thread::NamedThread::hook(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&,--Type <RET> for more, q to quit, c to continue without paging--
 std::function<void ()> const&)>) at /usr/include/c++/9/bits/invoke.h:95
#18 std::thread::_Invoker<std::tuple<void (*)(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::function<void ()> const&), std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::_Bind<void (nebula::thread::GenericWorker::*(nebula::thread::GenericWorker*))()> > >::_M_invoke<0ul, 1ul, 2ul>(std::_Index_tuple<0ul, 1ul, 2ul>) (this=0x7f0e4e0208c8) at /usr/include/c++/9/thread:244
#19 std::thread::_Invoker<std::tuple<void (*)(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::function<void ()> const&), std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::_Bind<void (nebula::thread::GenericWorker::*(nebula::thread::GenericWorker*))()> > >::operator()() (this=0x7f0e4e0208c8) at /usr/include/c++/9/thread:251
#20 std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (*)(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::function<void ()> const&), std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::_Bind<void (nebula::thread::GenericWorker::*(nebula::thread::GenericWorker*))()> > > >::_M_run() (this=0x7f0e4e0208c0) at /usr/include/c++/9/thread:195
#21 0x0000000002b95b54 in execute_native_thread_routine ()
#22 0x00007f0e72247609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#23 0x00007f0e7216e293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Your Environments (required)

• OS: uname -a
• Compiler: g++ --version or clang++ --version
• CPU: lscpu
• Commit id (e.g. a3ffc7d8) bb2e684

How To Reproduce(required)

Steps to reproduce the behavior:

1. Step 1
2. Step 2
3. Step 3

Expected behavior

Additional context

[Sophie-Xie]

Discussed it with muyiyong offline: Balance data is a temp feature, we won't fix it in open source edition. And enterprise edition is fine.