Skip to content
This repository has been archived by the owner on Oct 28, 2021. It is now read-only.

Use-after-free when closing eth #3332

Closed
chfast opened this issue Oct 5, 2016 · 5 comments
Closed

Use-after-free when closing eth #3332

chfast opened this issue Oct 5, 2016 · 5 comments
Labels
bug help wanted

Comments

@chfast
Copy link
Member

chfast commented Oct 5, 2016 

=================================================================
==6164==ERROR: AddressSanitizer: heap-use-after-free on address 0x625000059bc8 at pc 0x7fd141367c15 bp 0x7fd12a7219a0 sp 0x7fd12a721990
READ of size 1 at 0x625000059bc8 thread T14 (p2p)
    #0 0x7fd141367c14 in std::_Function_handler<void (boost::system::error_code const&), dev::p2p::DeadlineOps::reap()::{lambda(boost::system::error_code const&)#1}>::_M_invoke(std::_Any_data const&, boost::system::error_code const&) (/home/chfast/Projects/ethereum/build/cpp-ethereum/asan/libp2p/libp2p.so+0x199c14)
    #1 0x7fd1412f8266 in boost::asio::detail::wait_handler<std::function<void (boost::system::error_code const&)> >::do_complete(boost::asio::detail::task_io_service*, boost::asio::detail::task_io_service_operation*, boost::system::error_code const&, unsigned long) (/home/chfast/Projects/ethereum/build/cpp-ethereum/asan/libp2p/libp2p.so+0x12a266)
    #2 0x7fd14132200f in dev::p2p::Host::doneWorking() (/home/chfast/Projects/ethereum/build/cpp-ethereum/asan/libp2p/libp2p.so+0x15400f)
    #3 0x7fd140f41c64 in dev::Worker::startWorking()::{lambda()#1}::operator()() const [clone .constprop.236] (/home/chfast/Projects/ethereum/build/cpp-ethereum/asan/libdevcore/libdevcore.so+0x8cc64)
    #4 0x7fd1403a2c7f  (/usr/lib/x86_64-linux-gnu/libstdc++.so.6+0xb8c7f)
    #5 0x7fd141e7d6f9 in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x76f9)
    #6 0x7fd13fb08b5c in clone (/lib/x86_64-linux-gnu/libc.so.6+0x106b5c)

0x625000059bc8 is located 8904 bytes inside of 8912-byte region [0x625000057900,0x625000059bd0)
freed by thread T14 (p2p) here:
    #0 0x7fd14280fb2a in operator delete(void*) (/usr/lib/x86_64-linux-gnu/libasan.so.2+0x99b2a)
    #1 0x7fd14133b2ee in dev::p2p::Host::run(boost::system::error_code const&) (/home/chfast/Projects/ethereum/build/cpp-ethereum/asan/libp2p/libp2p.so+0x16d2ee)
    #2 0x7fd14133cfd9 in boost::asio::detail::wait_handler<dev::p2p::Host::run(boost::system::error_code const&)::{lambda(boost::system::error_code const&)#3}>::do_complete(boost::asio::detail::task_io_service*, boost::asio::detail::task_io_service_operation*, boost::system::error_code const&, unsigned long) (/home/chfast/Projects/ethereum/build/cpp-ethereum/asan/libp2p/libp2p.so+0x16efd9)
    #3 0x7fd1412b1981 in boost::asio::detail::task_io_service::run(boost::system::error_code&) (/home/chfast/Projects/ethereum/build/cpp-ethereum/asan/libp2p/libp2p.so+0xe3981)
    #4 0x7fd14131fae4 in dev::p2p::Host::doWork() (/home/chfast/Projects/ethereum/build/cpp-ethereum/asan/libp2p/libp2p.so+0x151ae4)
    #5 0x7fd140f417b4 in dev::Worker::startWorking()::{lambda()#1}::operator()() const [clone .constprop.236] (/home/chfast/Projects/ethereum/build/cpp-ethereum/asan/libdevcore/libdevcore.so+0x8c7b4)
    #6 0x7fd1403a2c7f  (/usr/lib/x86_64-linux-gnu/libstdc++.so.6+0xb8c7f)

previously allocated by thread T14 (p2p) here:
    #0 0x7fd14280f532 in operator new(unsigned long) (/usr/lib/x86_64-linux-gnu/libasan.so.2+0x99532)
    #1 0x7fd14133d7ee in dev::p2p::Host::startedWorking() (/home/chfast/Projects/ethereum/build/cpp-ethereum/asan/libp2p/libp2p.so+0x16f7ee)
    #2 0x7fd140f41cc1 in dev::Worker::startWorking()::{lambda()#1}::operator()() const [clone .constprop.236] (/home/chfast/Projects/ethereum/build/cpp-ethereum/asan/libdevcore/libdevcore.so+0x8ccc1)
    #3 0x7fd1403a2c7f  (/usr/lib/x86_64-linux-gnu/libstdc++.so.6+0xb8c7f)

Thread T14 (p2p) created by T0 here:
    #0 0x7fd1427ac253 in pthread_create (/usr/lib/x86_64-linux-gnu/libasan.so.2+0x36253)
    #1 0x7fd1403a2dc2 in std::thread::_M_start_thread(std::shared_ptr<std::thread::_Impl_base>, void (*)()) (/usr/lib/x86_64-linux-gnu/libstdc++.so.6+0xb8dc2)

SUMMARY: AddressSanitizer: heap-use-after-free ??:0 std::_Function_handler<void (boost::system::error_code const&), dev::p2p::DeadlineOps::reap()::{lambda(boost::system::error_code const&)#1}>::_M_invoke(std::_Any_data const&, boost::system::error_code const&)
Shadow bytes around the buggy address:
  0x0c4a80003320: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0c4a80003330: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0c4a80003340: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0c4a80003350: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0c4a80003360: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
=>0x0c4a80003370: fd fd fd fd fd fd fd fd fd[fd]fa fa fa fa fa fa
  0x0c4a80003380: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c4a80003390: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c4a800033a0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c4a800033b0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c4a800033c0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Heap right redzone:      fb
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack partial redzone:   f4
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
==6164==ABORTING
@chfast chfast added the bug label Oct 5, 2016
@chfast
Copy link
Member Author

chfast commented Feb 28, 2017

Might be related to #3586

@kmokeddem kmokeddem mentioned this issue Mar 17, 2017
Open
@chfast
Copy link
Member Author

chfast commented Aug 14, 2018

Still reproducible.

@chfast
Copy link
Member Author

chfast commented Aug 14, 2018 

=================================================================
==12559==ERROR: AddressSanitizer: heap-use-after-free on address 0x625000bec3e0 at pc 0x56019e499255 bp 0x7f7d3e263920 sp 0x7f7d3e263910
READ of size 1 at 0x625000bec3e0 thread T15 (p2p)
    #0 0x56019e499254 in std::__atomic_base<bool>::load(std::memory_order) const /usr/include/c++/8/bits/atomic_base.h:396
    #1 0x56019e499254 in std::atomic<bool>::operator bool() const /usr/include/c++/8/atomic:86
    #2 0x56019e499254 in operator() /home/chfast/Projects/ethereum/aleth/libp2p/Common.cpp:164
    #3 0x56019e499254 in _M_invoke /usr/include/c++/8/bits/std_function.h:297
    #4 0x56019e4ad61b in boost::asio::detail::wait_handler<std::function<void (boost::system::error_code const&)> >::do_complete(boost::asio::detail::task_io_service*, boost::asio::detail::task_io_service_operation*, boost::system::error_code const&, unsigned long) /usr/include/c++/8/bits/std_function.h:687
    #5 0x56019e4f87d0 in boost::asio::detail::task_io_service_operation::complete(boost::asio::detail::task_io_service&, boost::system::error_code const&, unsigned long) /home/chfast/.hunter/_Base/aa1facc/6dee614/852a159/Install/include/boost/asio/detail/task_io_service_operation.hpp:38
    #6 0x56019e4f87d0 in boost::asio::detail::task_io_service::do_poll_one(boost::asio::detail::scoped_lock<boost::asio::detail::posix_mutex>&, boost::asio::detail::task_io_service_thread_info&, boost::system::error_code const&) /home/chfast/.hunter/_Base/aa1facc/6dee614/852a159/Install/include/boost/asio/detail/impl/task_io_service.ipp:436
    #7 0x56019e4f87d0 in boost::asio::detail::task_io_service::poll(boost::system::error_code&) /home/chfast/.hunter/_Base/aa1facc/6dee614/852a159/Install/include/boost/asio/detail/impl/task_io_service.ipp:198
    #8 0x56019e4b0f3c in boost::asio::io_service::poll() /home/chfast/.hunter/_Base/aa1facc/6dee614/852a159/Install/include/boost/asio/impl/io_service.ipp:85
    #9 0x56019e4b64c0 in dev::p2p::Host::doneWorking() /home/chfast/Projects/ethereum/aleth/libp2p/Host.cpp:181
    #10 0x56019e31c145 in operator() /home/chfast/Projects/ethereum/aleth/libdevcore/Worker.cpp:63
    #11 0x56019e31c46c in __invoke_impl<void, dev::Worker::startWorking()::<lambda()> > /usr/include/c++/8/bits/invoke.h:60
    #12 0x56019e31c46c in __invoke<dev::Worker::startWorking()::<lambda()> > /usr/include/c++/8/bits/invoke.h:95
    #13 0x56019e31c46c in _M_invoke<0> /usr/include/c++/8/thread:234
    #14 0x56019e31c46c in operator() /usr/include/c++/8/thread:243
    #15 0x56019e31c46c in _M_run /usr/include/c++/8/thread:186
    #16 0x7f7d82eb4732  (/usr/lib/x86_64-linux-gnu/libstdc++.so.6+0xbe732)
    #17 0x7f7d8318b6da in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x76da)
    #18 0x7f7d8257088e in __clone (/lib/x86_64-linux-gnu/libc.so.6+0x12188e)

0x625000bec3e0 is located 8928 bytes inside of 8936-byte region [0x625000bea100,0x625000bec3e8)
freed by thread T15 (p2p) here:
    #0 0x7f7d838f5348 in operator delete(void*) (/usr/lib/x86_64-linux-gnu/libasan.so.5+0xf1348)
    #1 0x56019e4e9529 in std::_Sp_counted_ptr_inplace<dev::p2p::NodeTable, std::allocator<dev::p2p::NodeTable>, (__gnu_cxx::_Lock_policy)2>::_M_destroy() /usr/include/c++/8/ext/new_allocator.h:125
    #2 0x56019e4ca7b2 in std::__shared_count<(__gnu_cxx::_Lock_policy)2>::~__shared_count() /usr/include/c++/8/bits/shared_ptr_base.h:712
    #3 0x56019e4ca7b2 in std::__shared_ptr<dev::p2p::NodeTable, (__gnu_cxx::_Lock_policy)2>::~__shared_ptr() /usr/include/c++/8/bits/shared_ptr_base.h:1151
    #4 0x56019e4ca7b2 in std::__shared_ptr<dev::p2p::NodeTable, (__gnu_cxx::_Lock_policy)2>::reset() /usr/include/c++/8/bits/shared_ptr_base.h:1269
    #5 0x56019e4ca7b2 in dev::p2p::Host::run(boost::system::error_code const&) /home/chfast/Projects/ethereum/aleth/libp2p/Host.cpp:674
    #6 0x56019e4cb194 in do_complete /home/chfast/Projects/ethereum/aleth/libp2p/Host.cpp:743
    #7 0x56019e4f74bf in boost::asio::detail::task_io_service_operation::complete(boost::asio::detail::task_io_service&, boost::system::error_code const&, unsigned long) /home/chfast/.hunter/_Base/aa1facc/6dee614/852a159/Install/include/boost/asio/detail/task_io_service_operation.hpp:38
    #8 0x56019e4f74bf in boost::asio::detail::task_io_service::do_run_one(boost::asio::detail::scoped_lock<boost::asio::detail::posix_mutex>&, boost::asio::detail::task_io_service_thread_info&, boost::system::error_code const&) /home/chfast/.hunter/_Base/aa1facc/6dee614/852a159/Install/include/boost/asio/detail/impl/task_io_service.ipp:372
    #9 0x56019e4f74bf in boost::asio::detail::task_io_service::run(boost::system::error_code&) /home/chfast/.hunter/_Base/aa1facc/6dee614/852a159/Install/include/boost/asio/detail/impl/task_io_service.ipp:149
    #10 0x56019e4be3af in boost::asio::io_service::run() /home/chfast/.hunter/_Base/aa1facc/6dee614/852a159/Install/include/boost/asio/impl/io_service.ipp:59
    #11 0x56019e4be3af in dev::p2p::Host::doWork() /home/chfast/Projects/ethereum/aleth/libp2p/Host.cpp:798
    #12 0x56019e3190a3 in dev::Worker::workLoop() /home/chfast/Projects/ethereum/aleth/libdevcore/Worker.cpp:140
    #13 0x56019e31b774 in operator() /home/chfast/Projects/ethereum/aleth/libdevcore/Worker.cpp:62
    #14 0x56019e31c46c in __invoke_impl<void, dev::Worker::startWorking()::<lambda()> > /usr/include/c++/8/bits/invoke.h:60
    #15 0x56019e31c46c in __invoke<dev::Worker::startWorking()::<lambda()> > /usr/include/c++/8/bits/invoke.h:95
    #16 0x56019e31c46c in _M_invoke<0> /usr/include/c++/8/thread:234
    #17 0x56019e31c46c in operator() /usr/include/c++/8/thread:243
    #18 0x56019e31c46c in _M_run /usr/include/c++/8/thread:186
    #19 0x7f7d82eb4732  (/usr/lib/x86_64-linux-gnu/libstdc++.so.6+0xbe732)

previously allocated by thread T15 (p2p) here:
    #0 0x7f7d838f4470 in operator new(unsigned long) (/usr/lib/x86_64-linux-gnu/libasan.so.5+0xf0470)
    #1 0x56019e4d6232 in std::shared_ptr<dev::p2p::NodeTable> std::make_shared<dev::p2p::NodeTable, boost::asio::io_service&, dev::KeyPair&, dev::p2p::NodeIPEndpoint, bool&>(boost::asio::io_service&, dev::KeyPair&, dev::p2p::NodeIPEndpoint&&, bool&) /usr/include/c++/8/ext/new_allocator.h:111
    #2 0x56019e4d6232 in dev::p2p::Host::startedWorking() /home/chfast/Projects/ethereum/aleth/libp2p/Host.cpp:782
    #3 0x56019e31c14c in operator() /home/chfast/Projects/ethereum/aleth/libdevcore/Worker.cpp:61
    #4 0x56019e31c46c in __invoke_impl<void, dev::Worker::startWorking()::<lambda()> > /usr/include/c++/8/bits/invoke.h:60
    #5 0x56019e31c46c in __invoke<dev::Worker::startWorking()::<lambda()> > /usr/include/c++/8/bits/invoke.h:95
    #6 0x56019e31c46c in _M_invoke<0> /usr/include/c++/8/thread:234
    #7 0x56019e31c46c in operator() /usr/include/c++/8/thread:243
    #8 0x56019e31c46c in _M_run /usr/include/c++/8/thread:186
    #9 0x7f7d82eb4732  (/usr/lib/x86_64-linux-gnu/libstdc++.so.6+0xbe732)

Thread T15 (p2p) created by T0 here:
    #0 0x7f7d8384f043 in pthread_create (/usr/lib/x86_64-linux-gnu/libasan.so.5+0x4b043)
    #1 0x7f7d82eb4a18 in std::thread::_M_start_thread(std::unique_ptr<std::thread::_State, std::default_delete<std::thread::_State> >, void (*)()) (/usr/lib/x86_64-linux-gnu/libstdc++.so.6+0xbea18)
    #2 0x56019e4bce5c in dev::p2p::Host::start() /home/chfast/Projects/ethereum/aleth/libp2p/Host.cpp:129
    #3 0x56019d9ebacb in dev::WebThreeDirect::startNetwork() /home/chfast/Projects/ethereum/aleth/libwebthree/WebThree.h:203
    #4 0x56019d9ebacb in main /home/chfast/Projects/ethereum/aleth/aleth/main.cpp:1034
    #5 0x7f7d82470b96 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x21b96)

SUMMARY: AddressSanitizer: heap-use-after-free /usr/include/c++/8/bits/atomic_base.h:396 in std::__atomic_base<bool>::load(std::memory_order) const
Shadow bytes around the buggy address:
  0x0c4a80175820: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0c4a80175830: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0c4a80175840: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0c4a80175850: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0c4a80175860: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
=>0x0c4a80175870: fd fd fd fd fd fd fd fd fd fd fd fd[fd]fa fa fa
  0x0c4a80175880: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c4a80175890: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c4a801758a0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c4a801758b0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c4a801758c0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
==12559==ABORTING

@halfalicious
Copy link
Contributor

@chfast : Another case of a wait handler being executed after the node table has been deleted due to the host polling the io service...I think this is addressed by #5454, thoughts?

@chfast
Copy link
Member Author

chfast commented Feb 8, 2019

I cannot reproduce it any more.

@chfast chfast closed this as completed Feb 8, 2019
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
bug help wanted
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@chfast @halfalicious