Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

storage service crashed when stopping service #583

Closed
monadbobo opened this issue Jul 4, 2019 · 1 comment
Closed

storage service crashed when stopping service #583

monadbobo opened this issue Jul 4, 2019 · 1 comment
Assignees
@monadbobo
Milestone
R201910_beta

Comments

@monadbobo
Copy link
Contributor

I compile the nebula with addresssanitizer.

When I started, the query engine service crashed due to some exceptions. At this point, I stopped the storage service and then caused the storage service to crash. The following was the output of addresssanitizer.

==17116==ERROR: AddressSanitizer: heap-use-after-free on address 0x60400011f060 at pc 0x000001805b39 bp 0x7fcbfc2af400 sp 0x7fcbfc2af3f0
READ of size 8 at 0x60400011f060 thread T8
#0 0x1805b38 in nebula::meta::MetaClient::heartbeat() src/meta/client/MetaClient.cpp:957
#1 0x17f0850 in nebula::meta::MetaClient::heartBeatThreadFunc() src/meta/client/MetaClient.cpp:67
#2 0x1926f9d in void std::__invoke_impl<void, void (nebula::meta::MetaClient::&)(), nebula::meta::MetaClient&>(std::__invoke_memfun_deref, void (nebula::meta::MetaClient::&)(), nebula::meta::MetaClient&) (/data/pagefault/tools/nebula/bin/nebula-storaged+0x1926f9d)
#3 0x1910f61 in std::__invoke_result<void (nebula::meta::MetaClient::&)(), nebula::meta::MetaClient&>::type std::__invoke<void (nebula::meta::MetaClient::&)(), nebula::meta::MetaClient&>(void (nebula::meta::MetaClient::&)(), nebula::meta::MetaClient&) /usr/include/c++/8/bits/invoke.h:95
#4 0x18e9e60 in void std::_Bind<void (nebula::meta::MetaClient::(nebula::meta::MetaClient))()>::__call<void, , 0ul>(std::tuple<>&&, std::_Index_tuple<0ul>) (/data/pagefault/tools/nebula/bin/nebula-storaged+0x18e9e60)
#5 0x18bb3fd in void std::_Bind<void (nebula::meta::MetaClient::(nebula::meta::MetaClient))()>::operator()<, void>() (/data/pagefault/tools/nebula/bin/nebula-storaged+0x18bb3fd)
#6 0x189dcbf in std::_Function_handler<void (), std::_Bind<void (nebula::meta::MetaClient::(nebula::meta::MetaClient))()> >::_M_invoke(std::_Any_data const&) /usr/include/c++/8/bits/std_function.h:297
#7 0x1f9f25f in std::function<void ()>::operator()() const /usr/include/c++/8/bits/std_function.h:687
#8 0x21560f1 in operator() src/common/thread/GenericWorker.cpp:124
#9 0x215621a in _FUN src/common/thread/GenericWorker.cpp:128
#10 0x2704780 in event_process_active_single_queue (/data/pagefault/tools/nebula/bin/nebula-storaged+0x2704780)
#11 0x2704ed6 in event_base_loop (/data/pagefault/tools/nebula/bin/nebula-storaged+0x2704ed6)
#12 0x2155b92 in nebula::thread::GenericWorker::loop() src/common/thread/GenericWorker.cpp:86
#13 0x215ed33 in void std::__invoke_impl<void, void (nebula::thread::GenericWorker::&)(), nebula::thread::GenericWorker&>(std::__invoke_memfun_deref, void (nebula::thread::GenericWorker::&)(), nebula::thread::GenericWorker&) /usr/include/c++/8/bits/invoke.h:73
#14 0x215eb8f in std::__invoke_result<void (nebula::thread::GenericWorker::&)(), nebula::thread::GenericWorker&>::type std::__invoke<void (nebula::thread::GenericWorker::&)(), nebula::thread::GenericWorker&>(void (nebula::thread::GenericWorker::&)(), nebula::thread::GenericWorker&) (/data/pagefault/tools/
nebula/bin/nebula-storaged+0x215eb8f)
#15 0x215e9ae in void std::_Bind<void (nebula::thread::GenericWorker::(nebula::thread::GenericWorker))()>::__call<void, , 0ul>(std::tuple<>&&, std::_Index_tuple<0ul>) /usr/include/c++/8/functional:400
#16 0x215e44f in void std::_Bind<void (nebula::thread::GenericWorker::(nebula::thread::GenericWorker))()>::operator()<, void>() /usr/include/c++/8/functional:484
#17 0x215debd in std::_Function_handler<void (), std::_Bind<void (nebula::thread::GenericWorker::(nebula::thread::GenericWorker))()> >::_M_invoke(std::_Any_data const&) /usr/include/c++/8/bits/std_function.h:297
#18 0x1f9f25f in std::function<void ()>::operator()() const /usr/include/c++/8/bits/std_function.h:687
#19 0x209f4cb in nebula::thread::NamedThread::hook(std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, std::function<void ()> const&) src/common/thread/NamedThread.h:65
#20 0x215c271 in void std::__invoke_impl<void, void ()(std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, std::function<void ()> const&), std::__cxx11::basic_string<char, std::char_traits, std::allocator >, std::_Bind<void (nebula::thread::GenericWorker::(nebul
a::thread::GenericWorker*))()> >(std::__invoke_other, void (&&)(std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, std::function<void ()> const&), std::__cxx11::basic_string<char, std::char_traits, std::allocator >&&, std::_Bind<void (nebula::thread::GenericWorker::
(nebula::thread::GenericWorker))()>&&) /usr/include/c++/8/bits/invoke.h:60
#21 0x215a165 in std::__invoke_result<void ()(std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, std::function<void ()> const&), std::__cxx11::basic_string<char, std::char_traits, std::allocator >, std::_Bind<void (nebula::thread::GenericWorker::(nebula::thread
::GenericWorker))()> >::type std::__invoke<void ()(std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, std::function<void ()> const&), std::__cxx11::basic_string<char, std::char_traits, std::allocator >, std::_Bind<void (nebula::thread::GenericWorker::(nebula::thre
ad::GenericWorker*))()> >(void (&&)(std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, std::function<void ()> const&), std::__cxx11::basic_string<char, std::char_traits, std::allocator >&&, std::_Bind<void (nebula::thread::GenericWorker::(nebula::thread::GenericWor
ker*))()>&&) /usr/include/c++/8/bits/invoke.h:95
#22 0x215ef08 in decltype (__invoke((_S_declval<0ul>)(), (_S_declval<1ul>)(), (_S_declval<2ul>)())) std::thread::_Invoker<std::tuple<void ()(std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, std::function<void ()> const&), std::__cxx11::basic_string<char, std::char_traits<
char>, std::allocator >, std::_Bind<void (nebula::thread::GenericWorker::(nebula::thread::GenericWorker*))()> > >::_M_invoke<0ul, 1ul, 2ul>(std::_Index_tuple<0ul, 1ul, 2ul>) /usr/include/c++/8/thread:244
#23 0x215ee5f in std::thread::_Invoker<std::tuple<void ()(std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, std::function<void ()> const&), std::__cxx11::basic_string<char, std::char_traits, std::allocator >, std::_Bind<void (nebula::thread::GenericWorker::(ne
bula::thread::GenericWorker*))()> > >::operator()() /usr/include/c++/8/thread:253
#24 0x215edd7 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<void ()(std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, std::function<void ()> const&), std::__cxx11::basic_string<char, std::char_traits, std::allocator >, std::_Bind<void (nebula::th
read::GenericWorker::(nebula::thread::GenericWorker*))()> > > >::_M_run() /usr/include/c++/8/thread:196
#25 0x28017c2 in execute_native_thread_routine (/data/pagefault/tools/nebula/bin/nebula-storaged+0x28017c2)
#26 0x7fcc07b2458d in start_thread /usr/src/debug/glibc-2.28/nptl/pthread_create.c:486
#27 0x7fcc07a53512 in clone (/lib64/libc.so.6+0xfd512)

0x60400011f060 is located 16 bytes inside of 40-byte region [0x60400011f050,0x60400011f078)
freed by thread T0 here:
#0 0x7fcc08302a50 in operator delete(void*, unsigned long) (/lib64/libasan.so.5+0xf2a50)
#1 0x17a93cc in nebula::kvstore::MetaServerBasedPartManager::~MetaServerBasedPartManager() src/kvstore/PartManager.cpp:48
#2 0x148d8ce in std::default_deletenebula::kvstore::PartManager::operator()(nebula::kvstore::PartManager*) const (/data/pagefault/tools/nebula/bin/nebula-storaged+0x148d8ce)
#3 0x1487dbc in std::unique_ptr<nebula::kvstore::PartManager, std::default_deletenebula::kvstore::PartManager >::~unique_ptr() /usr/include/c++/8/bits/unique_ptr.h:274
#4 0x17cfdd8 in nebula::kvstore::NebulaStore::~NebulaStore() src/kvstore/NebulaStore.h:46
#5 0x17cfe3f in nebula::kvstore::NebulaStore::~NebulaStore() src/kvstore/NebulaStore.h:46
#6 0x148e1ea in std::default_deletenebula::kvstore::KVStore::operator()(nebula::kvstore::KVStore*) const (/data/pagefault/tools/nebula/bin/nebula-storaged+0x148e1ea)
#7 0x14883b2 in std::unique_ptr<nebula::kvstore::KVStore, std::default_deletenebula::kvstore::KVStore >::~unique_ptr() (/data/pagefault/tools/nebula/bin/nebula-storaged+0x14883b2)
#8 0x1478b94 in main src/daemons/StorageDaemon.cpp:208
#9 0x7fcc0797a412 in __libc_start_main ../csu/libc-start.c:308
#10 0x1474d0d in _start (/data/pagefault/tools/nebula/bin/nebula-storaged+0x1474d0d)

previously allocated by thread T0 here:
#0 0x7fcc08301470 in operator new(unsigned long) (/lib64/libasan.so.5+0xf1470)
#1 0x1488844 in std::_MakeUniqnebula::kvstore::MetaServerBasedPartManager::__single_object std::make_unique<nebula::kvstore::MetaServerBasedPartManager, std::pair<int, int>&, nebula::meta::MetaClient*&>(std::pair<int, int>&, nebula::meta::MetaClient*&) (/data/pagefault/tools/nebula/bin/nebula-storaged+0x14888
44)
#2 0x1475435 in getStoreInstance(std::pair<int, int>, std::vector<std::__cxx11::basic_string<char, std::char_traits, std::allocator >, std::allocator<std::__cxx11::basic_string<char, std::char_traits, std::allocator > > >, std::shared_ptrfolly::IOThreadPoolExecutor, std::shared_ptr, nebula::meta::MetaClient*, nebula::meta::SchemaManager*) src/daemons/StorageDaemon.cpp:64
#3 0x1477d50 in main src/daemons/StorageDaemon.cpp:173
#4 0x7fcc0797a412 in __libc_start_main ../csu/libc-start.c:308
#5 0x1474d0d in _start (/data/pagefault/tools/nebula/bin/nebula-storaged+0x1474d0d)
@monadbobo monadbobo self-assigned this Jul 4, 2019
monadbobo added a commit to monadbobo/nebula that referenced this issue Jul 4, 2019
@monadbobo
Fixed a bug where the storage service might crash (vesoft-inc#583)
09bbae3 
    When the storage service deamon is started,
    the nebula::meta::MetaClient will use the MetaServerBasedPartManager object(MetaClient::registerListener),
    but the MetaServerBasedPartManager object will be released before nebula::meta::MetaClient(through KVStore object),
    so web need to manually release nebula::meta::MetaClient here before release KVStore object.
monadbobo added a commit to monadbobo/nebula that referenced this issue Jul 4, 2019
@monadbobo
Fixed a bug where the storage service might crash (vesoft-inc#583)
1c346f6 
  When the storage service deamon is started, the nebula::meta::MetaClient
  will use the MetaServerBasedPartManager object(MetaClient::registerListener),
  but the MetaServerBasedPartManager object will be released before
  nebula::meta::MetaClient(through KVStore object), so we need to manually
  release nebula::meta::MetaClient here before release KVStore object.
monadbobo added a commit to monadbobo/nebula that referenced this issue Jul 8, 2019
@monadbobo
Fixed a bug where the storage service might crash (vesoft-inc#583)
efee045 
  When the storage service deamon is started, the nebula::meta::MetaClient
  will use the MetaServerBasedPartManager object(MetaClient::registerListener),
  but the MetaServerBasedPartManager object will be released before
  nebula::meta::MetaClient(through KVStore object), so we need to manually
  release nebula::meta::MetaClient here before release KVStore object.
monadbobo added a commit to monadbobo/nebula that referenced this issue Jul 8, 2019
@monadbobo
Fixed a bug where the storage service might crash (vesoft-inc#583)
c0fc272 
summary:

  When the storage service deamon is started, the nebula::meta::MetaClient
  will use the MetaServerBasedPartManager object(MetaClient::registerListener),
  but the MetaServerBasedPartManager object will be released before
  nebula::meta::MetaClient(through KVStore object).

  Added the MetaClient::unRegisterListener method to stop the background thread
  and reset the listener_ object, And the unRegisterListener is called
  when the MetaServerBasedPartManager object was released.
@sherman-the-tank
Copy link
Member

I think we should enable ASAN by default

@sherman-the-tank sherman-the-tank added this to the R201910_beta milestone Jul 10, 2019
monadbobo added a commit to monadbobo/nebula that referenced this issue Jul 12, 2019
@monadbobo
Fixed a bug where the storage service might crash (vesoft-inc#583)
361ac99 
Use a read-write lock in metaclient to protect the listener_.
monadbobo added a commit to monadbobo/nebula that referenced this issue Jul 19, 2019
@monadbobo
Fixed a bug where the storage service might crash (vesoft-inc#583)
73a8243 
        Fixed possible thread competition for checking listener_ .
dutor pushed a commit that referenced this issue Jul 19, 2019
@monadbobo @dutor
Fixed a bug where the storage service might crash (#583) (#595)
9214d99 
* Fixed a bug where the storage service might crash (#583)

summary:

  When the storage service deamon is started, the nebula::meta::MetaClient
  will use the MetaServerBasedPartManager object(MetaClient::registerListener),
  but the MetaServerBasedPartManager object will be released before
  nebula::meta::MetaClient(through KVStore object).

  Added the MetaClient::unRegisterListener method to stop the background thread
  and reset the listener_ object, And the unRegisterListener is called
  when the MetaServerBasedPartManager object was released.

* Fixed a bug where the storage service might crash (#583)

Use a read-write lock in metaclient to protect the listener_.

*         Fixed a bug where the storage service might crash (#583)

        Fixed possible thread competition for checking listener_ .
tong-hao pushed a commit to tong-hao/nebula that referenced this issue Jun 1, 2021
@monadbobo @dutor
Fixed a bug where the storage service might crash (vesoft-inc#583) (v…
d4a5ccb 
…esoft-inc#595)

* Fixed a bug where the storage service might crash (vesoft-inc#583)

summary:

  When the storage service deamon is started, the nebula::meta::MetaClient
  will use the MetaServerBasedPartManager object(MetaClient::registerListener),
  but the MetaServerBasedPartManager object will be released before
  nebula::meta::MetaClient(through KVStore object).

  Added the MetaClient::unRegisterListener method to stop the background thread
  and reset the listener_ object, And the unRegisterListener is called
  when the MetaServerBasedPartManager object was released.

* Fixed a bug where the storage service might crash (vesoft-inc#583)

Use a read-write lock in metaclient to protect the listener_.

*         Fixed a bug where the storage service might crash (vesoft-inc#583)

        Fixed possible thread competition for checking listener_ .
yixinglu pushed a commit to yixinglu/nebula that referenced this issue Mar 21, 2022
@zhaohaifei
fix bug (vesoft-inc#583)
69a53e7
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@monadbobo monadbobo

Labels
None yet
Projects
None yet
Milestone
R201910_beta
Development

No branches or pull requests
2 participants
@monadbobo @sherman-the-tank