Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

deadlock on locking HBBFT State after receiving Agreements #7

Closed
SurfingNerd opened this issue Mar 9, 2023 · 8 comments
Closed

deadlock on locking HBBFT State after receiving Agreements #7

SurfingNerd opened this issue Mar 9, 2023 · 8 comments

Comments

@SurfingNerd
Copy link
Collaborator

SurfingNerd commented Mar 9, 2023 •
edited
Loading

The reason for not producing blocks is an infinite lock on the hbbft worker thread that tries to write the hbbft state:

  1. State Write: ethcore::engines::hbbft::hbbft_engine::HoneyBadgerBFT::start_hbbft_epoch
  2. State Write ethcore::engines::hbbft::hbbft_engine::HoneyBadgerBFT::process_hb_message
  3. State Write: ethcore::engines::hbbft::hbbft_engine::HoneyBadgerBFT::replay_cached_messages

This bug is new and might be connected to the integration of either peer management or hbbft message tracking

Woker HBBFT 0 (This is the Engine Main Loop)
syscall (@syscall:12)
parking_lot_core::thread_parker::imp::ThreadParker::futex_wait (/home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/parking_lot_core-0.8.3/src/thread_parker/linux.rs:112)
<parking_lot_core::thread_parker::imp::ThreadParker as parking_lot_core::thread_parker::ThreadParkerT>::park (/home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/parking_lot_core-0.8.3/src/thread_parker/linux.rs:66)
parking_lot_core::parking_lot::park::{{closure}} (/home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/parking_lot_core-0.8.3/src/parking_lot.rs:611)
parking_lot_core::parking_lot::with_thread_data (/home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/parking_lot_core-0.8.3/src/parking_lot.rs:183)
parking_lot_core::parking_lot::park (/home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/parking_lot_core-0.8.3/src/parking_lot.rs:576)
parking_lot::raw_rwlock::RawRwLock::lock_common (/home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/parking_lot-0.11.1/src/raw_rwlock.rs:1110)
parking_lot::raw_rwlock::RawRwLock::lock_exclusive_slow (/home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/parking_lot-0.11.1/src/raw_rwlock.rs:628)
<parking_lot::raw_rwlock::RawRwLock as lock_api::rwlock::RawRwLock>::lock_exclusive (/home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/parking_lot-0.11.1/src/raw_rwlock.rs:74)
lock_api::rwlock::RwLock<R,T>::write (/home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/lock_api-0.4.3/src/rwlock.rs:461)
ethcore::engines::hbbft::hbbft_engine::HoneyBadgerBFT::start_hbbft_epoch (/home/sn/dmd/diamond-node/crates/ethcore/src/engines/hbbft/hbbft_engine.rs:724)
<ethcore::engines::hbbft::hbbft_engine::TransitionHandler as ethcore_io::IoHandler<()>>::timeout (/home/sn/dmd/diamond-node/crates/ethcore/src/engines/hbbft/hbbft_engine.rs:210)
ethcore_io::worker::Worker::do_work (/home/sn/dmd/diamond-node/crates/runtime/io/src/worker.rs:135)
ethcore_io::worker::Worker::new::{{closure}}::{{closure}} (/home/sn/dmd/diamond-node/crates/runtime/io/src/worker.rs:98)
futures::future::loop_fn::loop_fn (/home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.29/src/future/loop_fn.rs:79)
ethcore_io::worker::Worker::new::{{closure}} (/home/sn/dmd/diamond-node/crates/runtime/io/src/worker.rs:86)
std::sys_common::backtrace::__rust_begin_short_backtrace (@std::sys_common::backtrace::__rust_begin_short_backtrace:17)
std::thread::Builder::spawn_unchecked_::{{closure}}::{{closure}} (@core::ops::function::FnOnce::call_once{{vtable.shim}}:70)
<core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once (@core::ops::function::FnOnce::call_once{{vtable.shim}}:58)
std::panicking::try::do_call (@core::ops::function::FnOnce::call_once{{vtable.shim}}:58)

a second thread that tries to write on the state is

Worker Client 1
syscall (@syscall:12)
parking_lot_core::thread_parker::imp::ThreadParker::futex_wait (/home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/parking_lot_core-0.8.3/src/thread_parker/linux.rs:112)
<parking_lot_core::thread_parker::imp::ThreadParker as parking_lot_core::thread_parker::ThreadParkerT>::park (/home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/parking_lot_core-0.8.3/src/thread_parker/linux.rs:66)
parking_lot_core::parking_lot::park::{{closure}} (/home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/parking_lot_core-0.8.3/src/parking_lot.rs:611)
parking_lot_core::parking_lot::with_thread_data (/home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/parking_lot_core-0.8.3/src/parking_lot.rs:183)
parking_lot_core::parking_lot::park (/home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/parking_lot_core-0.8.3/src/parking_lot.rs:576)
parking_lot::raw_rwlock::RawRwLock::lock_common (/home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/parking_lot-0.11.1/src/raw_rwlock.rs:1110)
parking_lot::raw_rwlock::RawRwLock::lock_exclusive_slow (/home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/parking_lot-0.11.1/src/raw_rwlock.rs:628)
<parking_lot::raw_rwlock::RawRwLock as lock_api::rwlock::RawRwLock>::lock_exclusive (/home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/parking_lot-0.11.1/src/raw_rwlock.rs:74)
lock_api::rwlock::RwLock<R,T>::write (/home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/lock_api-0.4.3/src/rwlock.rs:461)
ethcore::engines::hbbft::hbbft_engine::HoneyBadgerBFT::process_hb_message (/home/sn/dmd/diamond-node/crates/ethcore/src/engines/hbbft/hbbft_engine.rs:524)
<ethcore::engines::hbbft::hbbft_engine::HoneyBadgerBFT as ethcore::engines::Engine<ethcore::machine::impls::EthereumMachine>>::handle_message (/home/sn/dmd/diamond-node/crates/ethcore/src/engines/hbbft/hbbft_engine.rs:1305)
<ethcore::client::client::Client as ethcore::client::traits::IoClient>::queue_consensus_message::{{closure}} (/home/sn/dmd/diamond-node/crates/ethcore/src/client/client.rs:2976)
ethcore::client::client::IoChannelQueue::queue::{{closure}} (/home/sn/dmd/diamond-node/crates/ethcore/src/client/client.rs:3520)
<ethcore_service::service::ClientIoHandler as ethcore_io::IoHandler<ethcore::client::io_message::ClientIoMessage>>::message (/home/sn/dmd/diamond-node/crates/ethcore/service/src/service.rs:229)
ethcore_io::worker::Worker::do_work (/home/sn/dmd/diamond-node/crates/runtime/io/src/worker.rs:139)
ethcore_io::worker::Worker::new::{{closure}}::{{closure}} (/home/sn/dmd/diamond-node/crates/runtime/io/src/worker.rs:98)
<futures::future::loop_fn::LoopFn<A,F> as futures::future::Future>::poll (/home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.29/src/future/loop_fn.rs:95)
futures::task_impl::Spawn<T>::poll_future_notify::{{closure}} (/home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.29/src/task_impl/mod.rs:329)
futures::task_impl::Spawn<T>::enter::{{closure}} (/home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.29/src/task_impl/mod.rs:399)

Worker Client 2
syscall (@syscall:12)
parking_lot_core::thread_parker::imp::ThreadParker::futex_wait (/home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/parking_lot_core-0.8.3/src/thread_parker/linux.rs:112)
<parking_lot_core::thread_parker::imp::ThreadParker as parking_lot_core::thread_parker::ThreadParkerT>::park (/home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/parking_lot_core-0.8.3/src/thread_parker/linux.rs:66)
parking_lot_core::parking_lot::park::{{closure}} (/home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/parking_lot_core-0.8.3/src/parking_lot.rs:611)
parking_lot_core::parking_lot::with_thread_data (/home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/parking_lot_core-0.8.3/src/parking_lot.rs:183)
parking_lot_core::parking_lot::park (/home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/parking_lot_core-0.8.3/src/parking_lot.rs:576)
parking_lot::raw_rwlock::RawRwLock::lock_common (/home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/parking_lot-0.11.1/src/raw_rwlock.rs:1110)
parking_lot::raw_rwlock::RawRwLock::lock_exclusive_slow (/home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/parking_lot-0.11.1/src/raw_rwlock.rs:628)
<parking_lot::raw_rwlock::RawRwLock as lock_api::rwlock::RawRwLock>::lock_exclusive (/home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/parking_lot-0.11.1/src/raw_rwlock.rs:74)
lock_api::rwlock::RwLock<R,T>::write (/home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/lock_api-0.4.3/src/rwlock.rs:461)
ethcore::engines::hbbft::hbbft_engine::HoneyBadgerBFT::join_hbbft_epoch (/home/sn/dmd/diamond-node/crates/ethcore/src/engines/hbbft/hbbft_engine.rs:708)
ethcore::engines::hbbft::hbbft_engine::HoneyBadgerBFT::process_hb_message (/home/sn/dmd/diamond-node/crates/ethcore/src/engines/hbbft/hbbft_engine.rs:546)
<ethcore::engines::hbbft::hbbft_engine::HoneyBadgerBFT as ethcore::engines::Engine<ethcore::machine::impls::EthereumMachine>>::handle_message (/home/sn/dmd/diamond-node/crates/ethcore/src/engines/hbbft/hbbft_engine.rs:1305)
<ethcore::client::client::Client as ethcore::client::traits::IoClient>::queue_consensus_message::{{closure}} (/home/sn/dmd/diamond-node/crates/ethcore/src/client/client.rs:2976)
ethcore::client::client::IoChannelQueue::queue::{{closure}} (/home/sn/dmd/diamond-node/crates/ethcore/src/client/client.rs:3520)
<ethcore_service::service::ClientIoHandler as ethcore_io::IoHandler<ethcore::client::io_message::ClientIoMessage>>::message (/home/sn/dmd/diamond-node/crates/ethcore/service/src/service.rs:229)
ethcore_io::worker::Worker::do_work (/home/sn/dmd/diamond-node/crates/runtime/io/src/worker.rs:139)
ethcore_io::worker::Worker::new::{{closure}}::{{closure}} (/home/sn/dmd/diamond-node/crates/runtime/io/src/worker.rs:98)
<futures::future::loop_fn::LoopFn<A,F> as futures::future::Future>::poll (/home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.29/src/future/loop_fn.rs:95)
futures::task_impl::Spawn<T>::poll_future_notify::{{closure}} (/home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.29/src/task_impl/mod.rs:329)
@SurfingNerd
Copy link
Collaborator Author

checked in a fix with deb449e,
the deadlock happened because of a double write lock.
currently i don't know why this problem triggered now for the first time.
Maybe the code changes changed the relative execution times, so the error never occured.

There is a deadlock detection tool:
https://amanieu.github.io/parking_lot/parking_lot/deadlock/index.html

after fixing this doublelock, we can now create a Seal, but we seem to have the same Issue with locking of seals.

@SurfingNerd
Copy link
Collaborator Author

an update of parking_lot to latest version v0.12.1 results into a vesion conflict by our old friend parity-crypto:

 failed to select a version for `parking_lot_core`.
    ... required by package `parking_lot v0.12.1`
    ... which satisfies dependency `parking_lot = "^0.12.1"` of package `blooms-db v0.1.0 (/home/sn/dmd/dmd-meta/diamond-node/crates/db/blooms-db)`
    ... which satisfies path dependency `blooms-db` (locked to 0.1.0) of package `diamond-node v3.3.5-hbbft-0.8.5 (/home/sn/dmd/dmd-meta/diamond-node)`
versions that meet the requirements `^0.9.0` are: 0.9.7, 0.9.6, 0.9.5, 0.9.4, 0.9.3, 0.9.2, 0.9.1, 0.9.0

all possible versions conflict with previously selected packages.

  previously selected package `parking_lot_core v0.9.7`
    ... which satisfies dependency `parking_lot_core = "^0.9.0"` of package `parking_lot v0.12.1`
    ... which satisfies dependency `parking_lot = "^0.12.1"` of package `blooms-db v0.1.0 (/home/sn/dmd/dmd-meta/diamond-node/crates/db/blooms-db)`
    ... which satisfies path dependency `blooms-db` (locked to 0.1.0) of package `diamond-node v3.3.5-hbbft-0.8.5 (/home/sn/dmd/dmd-meta/diamond-node)`

  previously selected package `parity-crypto v0.6.2`
    ... which satisfies dependency `parity-crypto = "^0.6.2"` of package `common-types v0.1.0 (/home/sn/dmd/dmd-meta/diamond-node/crates/ethcore/types)`
    ... which satisfies path dependency `common-types` (locked to 0.1.0) of package `diamond-node v3.3.5-hbbft-0.8.5 (/home/sn/dmd/dmd-meta/diamond-node)`

failed to select a version for `parking_lot_core` which could resolve this conflict

@SurfingNerd
Copy link
Collaborator Author

with latest changes it looks like that we manage to achieve the decryption share,
but cannot generate a seal out of it.
could not find any further deadlocks

@SurfingNerd
Copy link
Collaborator Author

SurfingNerd commented Mar 16, 2023 •
edited
Loading

probably a reentrant lock has been identified for update-sealing.
so i removed those long holding locks with checkin: 2f2f92a




2023-03-16 10:15:33 UTC  WARN diamond_node     0:     0x555f09838f2c - backtrace::backtrace::libunwind::trace::h64cea6bb0e2da25e
                               at /home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.56/src/backtrace/libunwind.rs:90:5
                           backtrace::backtrace::trace_unsynchronized::hbadc31359dc71a0d
                               at /home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.56/src/backtrace/mod.rs:66:5
                           backtrace::backtrace::trace::hdd36da1833347942
                               at /home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.56/src/backtrace/mod.rs:53:14
                           backtrace::capture::Backtrace::create::hbabe7b7ff540ec1b
                               at /home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.56/src/capture.rs:176:9
                           backtrace::capture::Backtrace::new::hc18be4f6ec18f39d
                               at /home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.56/src/capture.rs:140:22
   1:     0x555f0982d33c - parking_lot_core::parking_lot::deadlock_impl::on_unpark::hc64b111c0a867e5c
                               at /home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/parking_lot_core-0.8.3/src/parking_lot.rs:1179:32
   2:     0x555f0880c8a5 - parking_lot_core::parking_lot::deadlock::on_unpark::h1490c8b38d698178
                               at /home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/parking_lot_core-0.8.3/src/parking_lot.rs:1112:9
                           parking_lot_core::parking_lot::park::{{closure}}::h973c6d3779b4a31c
                               at /home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/parking_lot_core-0.8.3/src/parking_lot.rs:613:17
                           parking_lot_core::parking_lot::with_thread_data::h5c3548625547c0c2
                               at /home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/parking_lot_core-0.8.3/src/parking_lot.rs:183:5
                           parking_lot_core::parking_lot::park::h2118f7dd06fa574e
                               at /home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/parking_lot_core-0.8.3/src/parking_lot.rs:576:5
                           parking_lot::raw_rwlock::RawRwLock::lock_common::hd1be605fff67d99a
                               at /home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/parking_lot-0.11.1/src/raw_rwlock.rs:1110:17
                           parking_lot::raw_rwlock::RawRwLock::lock_exclusive_slow::h8462f8b273eb1b2e
                               at /home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/parking_lot-0.11.1/src/raw_rwlock.rs:628:26
   3:     0x555f0926f75c - <parking_lot::raw_rwlock::RawRwLock as lock_api::rwlock::RawRwLock>::lock_exclusive::hc684cdd184939d33
                               at /home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/parking_lot-0.11.1/src/raw_rwlock.rs:74:26
                           lock_api::rwlock::RwLock<R,T>::write::ha5df1c44d38f555a
                               at /home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/lock_api-0.4.3/src/rwlock.rs:461:9
                           <ethcore::engines::hbbft::hbbft_engine::HoneyBadgerBFT as ethcore::engines::Engine<ethcore::machine::impls::EthereumMachine>>::generate_seal::h41b4427b3641b42c
                               at /home/sn/dmd/diamond-node/crates/ethcore/src/engines/hbbft/hbbft_engine.rs:1331:13
   4:     0x555f08fad510 - ethcore::miner::miner::Miner::seal_and_import_block_internally::h46a6ca24b8467dac
                               at /home/sn/dmd/diamond-node/crates/ethcore/src/miner/miner.rs:900:15
   5:     0x555f08fafc8c - <ethcore::miner::miner::Miner as ethcore::miner::MinerService>::update_sealing::h061604355c1ddf38
                               at /home/sn/dmd/diamond-node/crates/ethcore/src/miner/miner.rs:1560:20
   6:     0x555f092697af - ethcore::engines::hbbft::hbbft_engine::HoneyBadgerBFT::process_seal_step::h5b6344c4e308fb96
                               at /home/sn/dmd/diamond-node/crates/ethcore/src/engines/hbbft/hbbft_engine.rs:677:13
   7:     0x555f0926a7e9 - ethcore::engines::hbbft::hbbft_engine::HoneyBadgerBFT::process_output::h79897a1daa8506de
                               at /home/sn/dmd/diamond-node/crates/ethcore/src/engines/hbbft/hbbft_engine.rs:503:13
                           ethcore::engines::hbbft::hbbft_engine::HoneyBadgerBFT::process_step::h65ce5d0bdeeb3fbe
                               at /home/sn/dmd/diamond-node/crates/ethcore/src/engines/hbbft/hbbft_engine.rs:696:9
   8:     0x555f0926ef83 - ethcore::engines::hbbft::hbbft_engine::HoneyBadgerBFT::process_hb_message::h57b12ebec3a9b3fd
                               at /home/sn/dmd/diamond-node/crates/ethcore/src/engines/hbbft/hbbft_engine.rs:544:17
                           <ethcore::engines::hbbft::hbbft_engine::HoneyBadgerBFT as ethcore::engines::Engine<ethcore::machine::impls::EthereumMachine>>::handle_message::h8041d03b60970f12
                               at /home/sn/dmd/diamond-node/crates/ethcore/src/engines/hbbft/hbbft_engine.rs:1304:17
   9:     0x555f09020bd7 - <ethcore::client::client::Client as ethcore::client::traits::IoClient>::queue_consensus_message::{{closure}}::h32e3796723863131
                               at /home/sn/dmd/diamond-node/crates/ethcore/src/client/client.rs:2975:33
                           ethcore::client::client::IoChannelQueue::queue::{{closure}}::h26a18bd9661d6518
                               at /home/sn/dmd/diamond-node/crates/ethcore/src/client/client.rs:3519:13
  10:     0x555f08c3b7d1 - <ethcore_service::service::ClientIoHandler as ethcore_io::IoHandler<ethcore::client::io_message::ClientIoMessage>>::message::h0f1dc1dd6fde9004
                               at /home/sn/dmd/diamond-node/crates/ethcore/service/src/service.rs:229:17
  11:     0x555f08c44858 - ethcore_io::worker::Worker::do_work::h961397f7586553a1
                               at /home/sn/dmd/diamond-node/crates/runtime/io/src/worker.rs:139:17
  12:     0x555f08c44e7f - ethcore_io::worker::Worker::new::{{closure}}::{{closure}}::h407607a04ba924be
                               at /home/sn/dmd/diamond-node/crates/runtime/io/src/worker.rs:98:41
  13:     0x555f08c4711d - <futures::future::loop_fn::LoopFn<A,F> as futures::future::Future>::poll::h62b746df975f13f0
                               at /home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.29/src/future/loop_fn.rs:95:52
                           futures::task_impl::Spawn<T>::poll_future_notify::{{closure}}::h6b9247accd9e2367
                               at /home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.29/src/task_impl/mod.rs:329:45
                           futures::task_impl::Spawn<T>::enter::{{closure}}::h3b83f6e9b438e694
                               at /home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.29/src/task_impl/mod.rs:399:27
                           futures::task_impl::std::set::h5ba53cbcea28b319
                               at /home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.29/src/task_impl/std/mod.rs:83:13
  14:     0x555f08c4595b - futures::task_impl::Spawn<T>::enter::hea54243bf658ac0e
                               at /home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.29/src/task_impl/mod.rs:399:9
                           futures::task_impl::Spawn<T>::poll_fn_notify::hcf4769411c232bc6
                               at /home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.29/src/task_impl/mod.rs:291:9
                           futures::task_impl::Spawn<T>::poll_future_notify::hadbe081f42cfff17
                               at /home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.29/src/task_impl/mod.rs:329:9
                           tokio_current_thread::Entered<P>::block_on::{{closure}}::h468083b21b27f363
                               at /home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-current-thread-0.1.6/src/lib.rs:494:39
                           tokio_current_thread::Borrow<U>::enter::{{closure}}::{{closure}}::h0b213a470d2c31fd
                               at /home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-current-thread-0.1.6/src/lib.rs:788:40
                           tokio_current_thread::CurrentRunner::set_spawn::h7a023b7cd15e6adb
                               at /home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-current-thread-0.1.6/src/lib.rs:825:9
  15:     0x555f08c45b45 - tokio_current_thread::Borrow<U>::enter::{{closure}}::hba7362894d530e2c
                               at /home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-current-thread-0.1.6/src/lib.rs:788:13
                           std::thread::local::LocalKey<T>::try_with::he782ffaaab1b33fb
                               at /rustc/d5a82bbd26e1ad8b7401f6a718a9c57c96905483/library/std/src/thread/local.rs:446:16
                           std::thread::local::LocalKey<T>::with::h84f168650c8b7e0e
                               at /rustc/d5a82bbd26e1ad8b7401f6a718a9c57c96905483/library/std/src/thread/local.rs:422:9
                           tokio_current_thread::Borrow<U>::enter::h4ad1399c3779c69d
                               at /home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-current-thread-0.1.6/src/lib.rs:786:9
                           tokio_current_thread::Entered<P>::block_on::hbc6241f3d865c228
                               at /home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-current-thread-0.1.6/src/lib.rs:491:23
  16:     0x555f08c45370 - tokio::runtime::current_thread::runtime::Runtime::block_on::{{closure}}::hc5db3610e0068c2b
                               at /home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.1.22/src/runtime/current_thread/runtime.rs:200:23
                           tokio::runtime::current_thread::runtime::Runtime::enter::{{closure}}::{{closure}}::{{closure}}::{{closure}}::h3bdc9c962bc2b1c7
                               at /home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.1.22/src/runtime/current_thread/runtime.rs:241:25
                           tokio_executor::global::with_default::{{closure}}::h192678a6165ff7e5
                               at /home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-executor-0.1.10/src/global.rs:221:9
                           std::thread::local::LocalKey<T>::try_with::hcdb6422ea47cf582
                               at /rustc/d5a82bbd26e1ad8b7401f6a718a9c57c96905483/library/std/src/thread/local.rs:446:16
                           std::thread::local::LocalKey<T>::with::hfc18bfbb4518daa4
                               at /rustc/d5a82bbd26e1ad8b7401f6a718a9c57c96905483/library/std/src/thread/local.rs:422:9
  17:     0x555f08c4730d - tokio_executor::global::with_default::h0113d2869870aabf
                               at /home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-executor-0.1.10/src/global.rs:190:5
                           tokio::runtime::current_thread::runtime::Runtime::enter::{{closure}}::{{closure}}::{{closure}}::h4000dbce57152f3b
                               at /home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.1.22/src/runtime/current_thread/runtime.rs:239:21
                           tokio_timer::timer::handle::with_default::h3bf04637cb4baabc
                               at /home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-timer-0.2.13/src/timer/handle.rs:74:5
  18:     0x555f08c46e83 - tokio::runtime::current_thread::runtime::Runtime::enter::{{closure}}::{{closure}}::he5af6edfcc8b7001
                               at /home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.1.22/src/runtime/current_thread/runtime.rs:232:17
                           tokio_timer::clock::clock::with_default::h8ae2643145119567
                               at /home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-timer-0.2.13/src/clock/clock.rs:125:5
  19:     0x555f08c45813 - tokio::runtime::current_thread::runtime::Runtime::enter::{{closure}}::h127a158816ff9841
                               at /home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.1.22/src/runtime/current_thread/runtime.rs:231:13
                           tokio_reactor::with_default::h6f624c1ff1c5ee08
                               at /home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-reactor-0.1.12/src/lib.rs:220:5
  20:     0x555f08c44a70 - tokio::runtime::current_thread::runtime::Runtime::enter::hc55c4f032c0d8a4b
                               at /home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.1.22/src/runtime/current_thread/runtime.rs:230:9
  21:     0x555f08c425d3 - tokio::runtime::current_thread::runtime::Runtime::block_on::h05bacb16484ab6b2
                               at /home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.1.22/src/runtime/current_thread/runtime.rs:198:9
                           tokio::runtime::current_thread::block_on_all::h8441ef7e2220bcf2
                               at /home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.1.22/src/runtime/current_thread/mod.rs:92:13
  22:     0x555f08c4191c - ethcore_io::worker::Worker::new::{{closure}}::h6f9f6c0e509ec370
                               at /home/sn/dmd/diamond-node/crates/runtime/io/src/worker.rs:108:38
                           std::sys_common::backtrace::__rust_begin_short_backtrace::h7317b2b8c3107b68
                               at /rustc/d5a82bbd26e1ad8b7401f6a718a9c57c96905483/library/std/src/sys_common/backtrace.rs:121:18
  23:     0x555f08c3c9d1 - std::thread::Builder::spawn_unchecked_::{{closure}}::{{closure}}::h5442fcdc9bbc97f9
                               at /rustc/d5a82bbd26e1ad8b7401f6a718a9c57c96905483/library/std/src/thread/mod.rs:550:17
                           <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once::hbfe5602085a1b7e6
                               at /rustc/d5a82bbd26e1ad8b7401f6a718a9c57c96905483/library/core/src/panic/unwind_safe.rs:271:9
                           std::panicking::try::do_call::h0a048fca16c61e90
                               at /rustc/d5a82bbd26e1ad8b7401f6a718a9c57c96905483/library/std/src/panicking.rs:483:40
                           std::panicking::try::hd84f5427e7f00c9e
                               at /rustc/d5a82bbd26e1ad8b7401f6a718a9c57c96905483/library/std/src/panicking.rs:447:19
                           std::panic::catch_unwind::h89b8f9234005858f
                               at /rustc/d5a82bbd26e1ad8b7401f6a718a9c57c96905483/library/std/src/panic.rs:137:14
                           std::thread::Builder::spawn_unchecked_::{{closure}}::h6defa27b04822577
                               at /rustc/d5a82bbd26e1ad8b7401f6a718a9c57c96905483/library/std/src/thread/mod.rs:549:30
                           core::ops::function::FnOnce::call_once{{vtable.shim}}::h834c26ef4492cf4e
                               at /rustc/d5a82bbd26e1ad8b7401f6a718a9c57c96905483/library/core/src/ops/function.rs:507:5
  24:     0x555f09b2bc93 - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once::hb77d8d72ebcf79c4
                               at /rustc/d5a82bbd26e1ad8b7401f6a718a9c57c96905483/library/alloc/src/boxed.rs:2000:9
                           <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once::hc08c3353e1568487
                               at /rustc/d5a82bbd26e1ad8b7401f6a718a9c57c96905483/library/alloc/src/boxed.rs:2000:9
                           std::sys::unix::thread::Thread::new::thread_start::h7168e596cd5e5ce6
                               at /rustc/d5a82bbd26e1ad8b7401f6a718a9c57c96905483/library/std/src/sys/unix/thread.rs:108:17
  25:     0x7f93fe1ad609 - start_thread
  26:     0x7f93fdf7d133 - clone
  27:                0x0 - <unknown>

@SurfingNerd
Copy link
Collaborator Author

after removing long holding hbbft state locks we have another deadlock scenario, that was not detected by the deadlock detection:

2f2f92a

this checkin resulted in the situation that the system is not able to acquire a write lock on hbbft state again.
sounds like a deadlock, but neither the deadlock detection triggered, nor a deadlock could be found after analysis

@SurfingNerd
Copy link
Collaborator Author

here are the connected stacktraces for this:



Thread 8:
replay_cached_messages:
Tries to get a Write Lock for hbbft_state

Thread 27:
process_hb_message

Thread 25:
process_hb_message

Thread 8 (Thread 0x7f5e7dffd700 (LWP 682798)):
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x000055f3b2e7b805 in parking_lot_core::thread_parker::imp::ThreadParker::futex_wait (self=0x7f5e7dffd4b8, ts=...) at /home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/parking_lot_core-0.8.3/src/thread_parker/linux.rs:112
#2  <parking_lot_core::thread_parker::imp::ThreadParker as parking_lot_core::thread_parker::ThreadParkerT>::park (self=0x7f5e7dffd4b8) at /home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/parking_lot_core-0.8.3/src/thread_parker/linux.rs:66
#3  parking_lot_core::parking_lot::park::{{closure}} (thread_data=0x7f5e7dffd460) at /home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/parking_lot_core-0.8.3/src/parking_lot.rs:611
#4  parking_lot_core::parking_lot::with_thread_data (f=...) at /home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/parking_lot_core-0.8.3/src/parking_lot.rs:183
#5  parking_lot_core::parking_lot::park (key=94505232111040, park_token=..., timeout=..., validate=..., before_sleep=..., timed_out=...) at /home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/parking_lot_core-0.8.3/src/parking_lot.rs:576
#6  parking_lot::raw_rwlock::RawRwLock::lock_common (self=0x55f3b6ca15c0, timeout=..., token=..., validate_flags=12, try_lock=...) at src/raw_rwlock.rs:1110
#7  parking_lot::raw_rwlock::RawRwLock::lock_exclusive_slow (self=0x55f3b6ca15c0, timeout=...) at src/raw_rwlock.rs:628
#8  0x000055f3b38d6047 in <parking_lot::raw_rwlock::RawRwLock as lock_api::rwlock::RawRwLock>::lock_exclusive (self=0x55f3b6ca15c0) at /home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/parking_lot-0.11.1/src/raw_rwlock.rs:74
#9  lock_api::rwlock::RwLock<R,T>::write (self=0x55f3b6ca15c0) at /home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/lock_api-0.4.3/src/rwlock.rs:461
#10 ethcore::engines::hbbft::hbbft_engine::HoneyBadgerBFT::replay_cached_messages (self=0x55f3b6ca1540) at crates/ethcore/src/engines/hbbft/hbbft_engine.rs:776
#11 <ethcore::engines::hbbft::hbbft_engine::TransitionHandler as ethcore_io::IoHandler<()>>::timeout (self=0x55f3b6c675d0, io=0x7f5e7dffaa00, timer=<optimized out>) at crates/ethcore/src/engines/hbbft/hbbft_engine.rs:194
#12 0x000055f3b388b431 in ethcore_io::worker::Worker::do_work (work=..., channel=...) at crates/runtime/io/src/worker.rs:135
#13 0x000055f3b3634e27 in ethcore_io::worker::Worker::new::{{closure}}::{{closure}} () at crates/runtime/io/src/worker.rs:98
#14 0x000055f3b367d3fd in <futures::future::loop_fn::LoopFn<A,F> as futures::future::Future>::poll (self=0x7f5e7dffae68) at /home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.29/src/future/loop_fn.rs:95
#15 futures::task_impl::Spawn<T>::poll_future_notify::{{closure}} (f=0x7f5e7dffae68) at /home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.29/src/task_impl/mod.rs:329
#16 futures::task_impl::Spawn<T>::enter::{{closure}} () at /home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.29/src/task_impl/mod.rs:399
#17 futures::task_impl::std::set (task=<optimized out>, f=...) at /home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.29/src/task_impl/std/mod.rs:83
#18 0x000055f3b35626eb in futures::task_impl::Spawn<T>::enter (self=<optimized out>, unpark=..., f=...) at /home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.29/src/task_impl/mod.rs:399
#19 futures::task_impl::Spawn<T>::poll_fn_notify (self=<optimized out>, notify=0x7f5e7dffad70, id=0, f=...) at /home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.29/src/task_impl/mod.rs:291
#20 futures::task_impl::Spawn<T>::poll_future_notify (self=<optimized out>, notify=0x7f5e7dffad70, id=0) at /home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.29/src/task_impl/mod.rs:329
#21 tokio_current_thread::Entered<P>::block_on::{{closure}} () at /home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-current-thread-0.1.6/src/lib.rs:494
#22 tokio_current_thread::Borrow<U>::enter::{{closure}}::{{closure}} () at /home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-current-thread-0.1.6/src/lib.rs:788
#23 tokio_current_thread::CurrentRunner::set_spawn (self=0x7f5e7dffc2e8, spawn=..., f=...) at /home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-current-thread-0.1.6/src/lib.rs:825
#24 0x000055f3b35628d5 in tokio_current_thread::Borrow<U>::enter::{{closure}} (current=<optimized out>) at /home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-current-thread-0.1.6/src/lib.rs:788
#25 std::thread::local::LocalKey<T>::try_with (f=..., self=<optimized out>) at /rustc/d5a82bbd26e1ad8b7401f6a718a9c57c96905483/library/std/src/thread/local.rs:446
#26 std::thread::local::LocalKey<T>::with (f=..., self=<optimized out>) at /rustc/d5a82bbd26e1ad8b7401f6a718a9c57c96905483/library/std/src/thread/local.rs:422
#27 tokio_current_thread::Borrow<U>::enter (self=0x7f5e7dffadc0, f=...) at /home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-current-thread-0.1.6/src/lib.rs:786
#28 tokio_current_thread::Entered<P>::block_on (self=0x7f5e7dffaf00, future=...) at /home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-current-thread-0.1.6/src/lib.rs:491
#29 0x000055f3b36366c0 in tokio::runtime::current_thread::runtime::Runtime::block_on::{{closure}} (executor=0x7f5e7dffaf00) at /home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.1.22/src/runtime/current_thread/runtime.rs:200
#30 tokio::runtime::current_thread::runtime::Runtime::enter::{{closure}}::{{closure}}::{{closure}}::{{closure}} (enter=<optimized out>) at /home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.1.22/src/runtime/current_thread/runtime.rs:241
#31 tokio_executor::global::with_default::{{closure}} (cell=<optimized out>) at /home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-executor-0.1.10/src/global.rs:221
#32 std::thread::local::LocalKey<T>::try_with (self=<optimized out>, f=...) at /rustc/d5a82bbd26e1ad8b7401f6a718a9c57c96905483/library/std/src/thread/local.rs:446
#33 std::thread::local::LocalKey<T>::with (self=<optimized out>, f=...) at /rustc/d5a82bbd26e1ad8b7401f6a718a9c57c96905483/library/std/src/thread/local.rs:422
#34 0x000055f3b35d704d in tokio_executor::global::with_default (executor=0x7f5e7dffb0c8, enter=0x7f5e7dffb320, f=...) at /home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-executor-0.1.10/src/global.rs:190
#35 tokio::runtime::current_thread::runtime::Runtime::enter::{{closure}}::{{closure}}::{{closure}} (enter=0x7f5e7dffb320) at /home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.1.22/src/runtime/current_thread/runtime.rs:239
#36 tokio_timer::timer::handle::with_default (handle=<optimized out>, enter=0x7f5e7dffb320, f=...) at /home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-timer-0.2.13/src/timer/handle.rs:74
#37 0x000055f3b3772463 in tokio::runtime::current_thread::runtime::Runtime::enter::{{closure}}::{{closure}} (enter=0x7f5e7dffb320) at /home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.1.22/src/runtime/current_thread/runtime.rs:232
#38 tokio_timer::clock::clock::with_default (clock=<optimized out>, enter=0x7f5e7dffb320, f=...) at /home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-timer-0.2.13/src/clock/clock.rs:125
#39 0x000055f3b36957a3 in tokio::runtime::current_thread::runtime::Runtime::enter::{{closure}} (enter=0x7f5e7dffb320) at /home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.1.22/src/runtime/current_thread/runtime.rs:231
#40 tokio_reactor::with_default (handle=<optimized out>, enter=0x7f5e7dffb320, f=...) at /home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-reactor-0.1.12/src/lib.rs:220
#41 0x000055f3b3621280 in tokio::runtime::current_thread::runtime::Runtime::enter (self=<optimized out>, f=...) at /home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.1.22/src/runtime/current_thread/runtime.rs:230
#42 0x000055f3b37e4753 in tokio::runtime::current_thread::runtime::Runtime::block_on (self=0x7f5e7dffb4e8, f=...) at /home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.1.22/src/runtime/current_thread/runtime.rs:198
#43 tokio::runtime::current_thread::block_on_all (future=...) at /home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.1.22/src/runtime/current_thread/mod.rs:92
#44 0x000055f3b38d2de4 in ethcore_io::worker::Worker::new::{{closure}} () at crates/runtime/io/src/worker.rs:108
#45 std::sys_common::backtrace::__rust_begin_short_backtrace (f=...) at /rustc/d5a82bbd26e1ad8b7401f6a718a9c57c96905483/library/std/src/sys_common/backtrace.rs:121
#46 0x000055f3b3897881 in std::thread::Builder::spawn_unchecked_::{{closure}}::{{closure}} () at /rustc/d5a82bbd26e1ad8b7401f6a718a9c57c96905483/library/std/src/thread/mod.rs:550
#47 <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once (self=..., _args=<optimized out>) at /rustc/d5a82bbd26e1ad8b7401f6a718a9c57c96905483/library/core/src/panic/unwind_safe.rs:271
#48 std::panicking::try::do_call (data=<optimized out>) at /rustc/d5a82bbd26e1ad8b7401f6a718a9c57c96905483/library/std/src/panicking.rs:483
#49 std::panicking::try (f=...) at /rustc/d5a82bbd26e1ad8b7401f6a718a9c57c96905483/library/std/src/panicking.rs:447
#50 std::panic::catch_unwind (f=...) at /rustc/d5a82bbd26e1ad8b7401f6a718a9c57c96905483/library/std/src/panic.rs:137
#51 std::thread::Builder::spawn_unchecked_::{{closure}} () at /rustc/d5a82bbd26e1ad8b7401f6a718a9c57c96905483/library/std/src/thread/mod.rs:549
#52 core::ops::function::FnOnce::call_once{{vtable-shim}} () at /rustc/d5a82bbd26e1ad8b7401f6a718a9c57c96905483/library/core/src/ops/function.rs:507
#53 0x000055f3b419ad13 in <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once () at library/alloc/src/boxed.rs:2000
#54 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once () at library/alloc/src/boxed.rs:2000
#55 std::sys::unix::thread::Thread::new::thread_start () at library/std/src/sys/unix/thread.rs:108
#56 0x00007f5e8d048609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#57 0x00007f5e8ce18133 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95


Thread 27 (Thread 0x7f5e27fff700 (LWP 682819)):
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x000055f3b2e7b805 in parking_lot_core::thread_parker::imp::ThreadParker::futex_wait (self=0x7f5e27fff4b8, ts=...) at /home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/parking_lot_core-0.8.3/src/thread_parker/linux.rs:112
#2  <parking_lot_core::thread_parker::imp::ThreadParker as parking_lot_core::thread_parker::ThreadParkerT>::park (self=0x7f5e27fff4b8) at /home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/parking_lot_core-0.8.3/src/thread_parker/linux.rs:66
#3  parking_lot_core::parking_lot::park::{{closure}} (thread_data=0x7f5e27fff460) at /home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/parking_lot_core-0.8.3/src/parking_lot.rs:611
#4  parking_lot_core::parking_lot::with_thread_data (f=...) at /home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/parking_lot_core-0.8.3/src/parking_lot.rs:183
#5  parking_lot_core::parking_lot::park (key=94505232111040, park_token=..., timeout=..., validate=..., before_sleep=..., timed_out=...) at /home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/parking_lot_core-0.8.3/src/parking_lot.rs:576
#6  parking_lot::raw_rwlock::RawRwLock::lock_common (self=0x55f3b6ca15c0, timeout=..., token=..., validate_flags=12, try_lock=...) at src/raw_rwlock.rs:1110
#7  parking_lot::raw_rwlock::RawRwLock::lock_exclusive_slow (self=0x55f3b6ca15c0, timeout=...) at src/raw_rwlock.rs:628
#8  0x000055f3b38dd816 in <parking_lot::raw_rwlock::RawRwLock as lock_api::rwlock::RawRwLock>::lock_exclusive (self=0x55f3b6ca15c0) at /home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/parking_lot-0.11.1/src/raw_rwlock.rs:74
#9  lock_api::rwlock::RwLock<R,T>::write (self=0x55f3b6ca15c0) at /home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/lock_api-0.4.3/src/rwlock.rs:461
#10 ethcore::engines::hbbft::hbbft_engine::HoneyBadgerBFT::process_hb_message (self=0x55f3b6ca1540, msg_idx=<optimized out>, message=..., sender_id=...) at crates/ethcore/src/engines/hbbft/hbbft_engine.rs:523
#11 <ethcore::engines::hbbft::hbbft_engine::HoneyBadgerBFT as ethcore::engines::Engine<ethcore::machine::impls::EthereumMachine>>::handle_message (self=0x55f3b6ca1540, message=..., node_id=...) at crates/ethcore/src/engines/hbbft/hbbft_engine.rs:1304
#12 0x000055f3b368fbd7 in <ethcore::client::client::Client as ethcore::client::traits::IoClient>::queue_consensus_message::{{closure}} (client=<optimized out>) at crates/ethcore/src/client/client.rs:2975
#13 ethcore::client::client::IoChannelQueue::queue::{{closure}} (client=<optimized out>) at crates/ethcore/src/client/client.rs:3519
#14 0x000055f3b32aa7d1 in <ethcore_service::service::ClientIoHandler as ethcore_io::IoHandler<ethcore::client::io_message::ClientIoMessage>>::message (self=0x55f3b6c6c170, _io=<optimized out>, net_message=0x7f5de000d4c0) at crates/ethcore/service/src/service.rs:229
#15 0x000055f3b32b3858 in ethcore_io::worker::Worker::do_work (work=..., channel=...) at crates/runtime/io/src/worker.rs:139
#16 0x000055f3b32b3e7f in ethcore_io::worker::Worker::new::{{closure}}::{{closure}} () at crates/runtime/io/src/worker.rs:98
#17 0x000055f3b32b611d in <futures::future::loop_fn::LoopFn<A,F> as futures::future::Future>::poll (self=0x7f5e27ffce48) at /home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.29/src/future/loop_fn.rs:95
#18 futures::task_impl::Spawn<T>::poll_future_notify::{{closure}} (f=0x7f5e27ffce48) at /home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.29/src/task_impl/mod.rs:329
#19 futures::task_impl::Spawn<T>::enter::{{closure}} () at /home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.29/src/task_impl/mod.rs:399
#20 futures::task_impl::std::set (task=<optimized out>, f=...) at /home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.29/src/task_impl/std/mod.rs:83
#21 0x000055f3b32b495b in futures::task_impl::Spawn<T>::enter (self=<optimized out>, unpark=..., f=...) at /home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.29/src/task_impl/mod.rs:399
#22 futures::task_impl::Spawn<T>::poll_fn_notify (self=<optimized out>, notify=0x7f5e27ffcd50, id=0, f=...) at /home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.29/src/task_impl/mod.rs:291
#23 futures::task_impl::Spawn<T>::poll_future_notify (self=<optimized out>, notify=0x7f5e27ffcd50, id=0) at /home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.29/src/task_impl/mod.rs:329
#24 tokio_current_thread::Entered<P>::block_on::{{closure}} () at /home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-current-thread-0.1.6/src/lib.rs:494
#25 tokio_current_thread::Borrow<U>::enter::{{closure}}::{{closure}} () at /home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-current-thread-0.1.6/src/lib.rs:788
#26 tokio_current_thread::CurrentRunner::set_spawn (self=0x7f5e27ffe2e8, spawn=..., f=...) at /home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-current-thread-0.1.6/src/lib.rs:825
#27 0x000055f3b32b4b45 in tokio_current_thread::Borrow<U>::enter::{{closure}} (current=<optimized out>) at /home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-current-thread-0.1.6/src/lib.rs:788
#28 std::thread::local::LocalKey<T>::try_with (f=..., self=<optimized out>) at /rustc/d5a82bbd26e1ad8b7401f6a718a9c57c96905483/library/std/src/thread/local.rs:446
#29 std::thread::local::LocalKey<T>::with (f=..., self=<optimized out>) at /rustc/d5a82bbd26e1ad8b7401f6a718a9c57c96905483/library/std/src/thread/local.rs:422
#30 tokio_current_thread::Borrow<U>::enter (self=0x7f5e27ffcda0, f=...) at /home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-current-thread-0.1.6/src/lib.rs:786
#31 tokio_current_thread::Entered<P>::block_on (self=0x7f5e27ffcee0, future=...) at /home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-current-thread-0.1.6/src/lib.rs:491
#32 0x000055f3b32b4370 in tokio::runtime::current_thread::runtime::Runtime::block_on::{{closure}} (executor=0x7f5e27ffcee0) at /home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.1.22/src/runtime/current_thread/runtime.rs:200
#33 tokio::runtime::current_thread::runtime::Runtime::enter::{{closure}}::{{closure}}::{{closure}}::{{closure}} (enter=<optimized out>) at /home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.1.22/src/runtime/current_thread/runtime.rs:241
#34 tokio_executor::global::with_default::{{closure}} (cell=<optimized out>) at /home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-executor-0.1.10/src/global.rs:221
#35 std::thread::local::LocalKey<T>::try_with (self=<optimized out>, f=...) at /rustc/d5a82bbd26e1ad8b7401f6a718a9c57c96905483/library/std/src/thread/local.rs:446
#36 std::thread::local::LocalKey<T>::with (self=<optimized out>, f=...) at /rustc/d5a82bbd26e1ad8b7401f6a718a9c57c96905483/library/std/src/thread/local.rs:422
#37 0x000055f3b32b630d in tokio_executor::global::with_default (executor=0x7f5e27ffd0a8, enter=0x7f5e27ffd300, f=...) at /home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-executor-0.1.10/src/global.rs:190
#38 tokio::runtime::current_thread::runtime::Runtime::enter::{{closure}}::{{closure}}::{{closure}} (enter=0x7f5e27ffd300) at /home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.1.22/src/runtime/current_thread/runtime.rs:239
#39 tokio_timer::timer::handle::with_default (handle=<optimized out>, enter=0x7f5e27ffd300, f=...) at /home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-timer-0.2.13/src/timer/handle.rs:74
#40 0x000055f3b32b5e83 in tokio::runtime::current_thread::runtime::Runtime::enter::{{closure}}::{{closure}} (enter=0x7f5e27ffd300) at /home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.1.22/src/runtime/current_thread/runtime.rs:232
#41 tokio_timer::clock::clock::with_default (clock=<optimized out>, enter=0x7f5e27ffd300, f=...) at /home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-timer-0.2.13/src/clock/clock.rs:125
#42 0x000055f3b32b4813 in tokio::runtime::current_thread::runtime::Runtime::enter::{{closure}} (enter=0x7f5e27ffd300) at /home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.1.22/src/runtime/current_thread/runtime.rs:231
#43 tokio_reactor::with_default (handle=<optimized out>, enter=0x7f5e27ffd300, f=...) at /home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-reactor-0.1.12/src/lib.rs:220
#44 0x000055f3b32b3a70 in tokio::runtime::current_thread::runtime::Runtime::enter (self=<optimized out>, f=...) at /home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.1.22/src/runtime/current_thread/runtime.rs:230
#45 0x000055f3b32b15d3 in tokio::runtime::current_thread::runtime::Runtime::block_on (self=0x7f5e27ffd4c8, f=...) at /home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.1.22/src/runtime/current_thread/runtime.rs:198
#46 tokio::runtime::current_thread::block_on_all (future=...) at /home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.1.22/src/runtime/current_thread/mod.rs:92
#47 0x000055f3b32b091c in ethcore_io::worker::Worker::new::{{closure}} () at crates/runtime/io/src/worker.rs:108
#48 std::sys_common::backtrace::__rust_begin_short_backtrace (f=...) at /rustc/d5a82bbd26e1ad8b7401f6a718a9c57c96905483/library/std/src/sys_common/backtrace.rs:121
#49 0x000055f3b32ab9d1 in std::thread::Builder::spawn_unchecked_::{{closure}}::{{closure}} () at /rustc/d5a82bbd26e1ad8b7401f6a718a9c57c96905483/library/std/src/thread/mod.rs:550
#50 <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once (self=<error reading variable: access outside bounds of object referenced via synthetic pointer>, _args=<optimized out>) at /rustc/d5a82bbd26e1ad8b7401f6a718a9c57c96905483/library/core/src/panic/unwind_safe.rs:271
#51 std::panicking::try::do_call (data=<optimized out>) at /rustc/d5a82bbd26e1ad8b7401f6a718a9c57c96905483/library/std/src/panicking.rs:483
#52 std::panicking::try (f=<error reading variable: access outside bounds of object referenced via synthetic pointer>) at /rustc/d5a82bbd26e1ad8b7401f6a718a9c57c96905483/library/std/src/panicking.rs:447
#53 std::panic::catch_unwind (f=<error reading variable: access outside bounds of object referenced via synthetic pointer>) at /rustc/d5a82bbd26e1ad8b7401f6a718a9c57c96905483/library/std/src/panic.rs:137
#54 std::thread::Builder::spawn_unchecked_::{{closure}} () at /rustc/d5a82bbd26e1ad8b7401f6a718a9c57c96905483/library/std/src/thread/mod.rs:549
#55 core::ops::function::FnOnce::call_once{{vtable-shim}} () at /rustc/d5a82bbd26e1ad8b7401f6a718a9c57c96905483/library/core/src/ops/function.rs:507
#56 0x000055f3b419ad13 in <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once () at library/alloc/src/boxed.rs:2000
#57 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once () at library/alloc/src/boxed.rs:2000
#58 std::sys::unix::thread::Thread::new::thread_start () at library/std/src/sys/unix/thread.rs:108
#59 0x00007f5e8d048609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#60 0x00007f5e8ce18133 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95



Thread 25 (Thread 0x7f5e2effe700 (LWP 682817)):
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x000055f3b2e7b805 in parking_lot_core::thread_parker::imp::ThreadParker::futex_wait (self=0x7f5e2effe4b8, ts=...) at /home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/parking_lot_core-0.8.3/src/thread_parker/linux.rs:112
#2  <parking_lot_core::thread_parker::imp::ThreadParker as parking_lot_core::thread_parker::ThreadParkerT>::park (self=0x7f5e2effe4b8) at /home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/parking_lot_core-0.8.3/src/thread_parker/linux.rs:66
#3  parking_lot_core::parking_lot::park::{{closure}} (thread_data=0x7f5e2effe460) at /home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/parking_lot_core-0.8.3/src/parking_lot.rs:611
#4  parking_lot_core::parking_lot::with_thread_data (f=...) at /home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/parking_lot_core-0.8.3/src/parking_lot.rs:183
#5  parking_lot_core::parking_lot::park (key=94505232111040, park_token=..., timeout=..., validate=..., before_sleep=..., timed_out=...) at /home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/parking_lot_core-0.8.3/src/parking_lot.rs:576
#6  parking_lot::raw_rwlock::RawRwLock::lock_common (self=0x55f3b6ca15c0, timeout=..., token=..., validate_flags=12, try_lock=...) at src/raw_rwlock.rs:1110
#7  parking_lot::raw_rwlock::RawRwLock::lock_exclusive_slow (self=0x55f3b6ca15c0, timeout=...) at src/raw_rwlock.rs:628
#8  0x000055f3b38dd816 in <parking_lot::raw_rwlock::RawRwLock as lock_api::rwlock::RawRwLock>::lock_exclusive (self=0x55f3b6ca15c0) at /home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/parking_lot-0.11.1/src/raw_rwlock.rs:74
#9  lock_api::rwlock::RwLock<R,T>::write (self=0x55f3b6ca15c0) at /home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/lock_api-0.4.3/src/rwlock.rs:461
#10 ethcore::engines::hbbft::hbbft_engine::HoneyBadgerBFT::process_hb_message (self=0x55f3b6ca1540, msg_idx=<optimized out>, message=..., sender_id=...) at crates/ethcore/src/engines/hbbft/hbbft_engine.rs:523
#11 <ethcore::engines::hbbft::hbbft_engine::HoneyBadgerBFT as ethcore::engines::Engine<ethcore::machine::impls::EthereumMachine>>::handle_message (self=0x55f3b6ca1540, message=..., node_id=...) at crates/ethcore/src/engines/hbbft/hbbft_engine.rs:1304
#12 0x000055f3b368fbd7 in <ethcore::client::client::Client as ethcore::client::traits::IoClient>::queue_consensus_message::{{closure}} (client=<optimized out>) at crates/ethcore/src/client/client.rs:2975
#13 ethcore::client::client::IoChannelQueue::queue::{{closure}} (client=<optimized out>) at crates/ethcore/src/client/client.rs:3519
#14 0x000055f3b32aa7d1 in <ethcore_service::service::ClientIoHandler as ethcore_io::IoHandler<ethcore::client::io_message::ClientIoMessage>>::message (self=0x55f3b6c6c170, _io=<optimized out>, net_message=0x7f5dd800d9f0) at crates/ethcore/service/src/service.rs:229
#15 0x000055f3b32b3858 in ethcore_io::worker::Worker::do_work (work=..., channel=...) at crates/runtime/io/src/worker.rs:139
#16 0x000055f3b32b3e7f in ethcore_io::worker::Worker::new::{{closure}}::{{closure}} () at crates/runtime/io/src/worker.rs:98
#17 0x000055f3b32b611d in <futures::future::loop_fn::LoopFn<A,F> as futures::future::Future>::poll (self=0x7f5e2effbe48) at /home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.29/src/future/loop_fn.rs:95
#18 futures::task_impl::Spawn<T>::poll_future_notify::{{closure}} (f=0x7f5e2effbe48) at /home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.29/src/task_impl/mod.rs:329
#19 futures::task_impl::Spawn<T>::enter::{{closure}} () at /home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.29/src/task_impl/mod.rs:399
#20 futures::task_impl::std::set (task=<optimized out>, f=...) at /home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.29/src/task_impl/std/mod.rs:83
#21 0x000055f3b32b495b in futures::task_impl::Spawn<T>::enter (self=<optimized out>, unpark=..., f=...) at /home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.29/src/task_impl/mod.rs:399
#22 futures::task_impl::Spawn<T>::poll_fn_notify (self=<optimized out>, notify=0x7f5e2effbd50, id=0, f=...) at /home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.29/src/task_impl/mod.rs:291
#23 futures::task_impl::Spawn<T>::poll_future_notify (self=<optimized out>, notify=0x7f5e2effbd50, id=0) at /home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.29/src/task_impl/mod.rs:329
#24 tokio_current_thread::Entered<P>::block_on::{{closure}} () at /home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-current-thread-0.1.6/src/lib.rs:494
#25 tokio_current_thread::Borrow<U>::enter::{{closure}}::{{closure}} () at /home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-current-thread-0.1.6/src/lib.rs:788
#26 tokio_current_thread::CurrentRunner::set_spawn (self=0x7f5e2effd2e8, spawn=..., f=...) at /home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-current-thread-0.1.6/src/lib.rs:825
#27 0x000055f3b32b4b45 in tokio_current_thread::Borrow<U>::enter::{{closure}} (current=<optimized out>) at /home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-current-thread-0.1.6/src/lib.rs:788
#28 std::thread::local::LocalKey<T>::try_with (f=..., self=<optimized out>) at /rustc/d5a82bbd26e1ad8b7401f6a718a9c57c96905483/library/std/src/thread/local.rs:446
#29 std::thread::local::LocalKey<T>::with (f=..., self=<optimized out>) at /rustc/d5a82bbd26e1ad8b7401f6a718a9c57c96905483/library/std/src/thread/local.rs:422
#30 tokio_current_thread::Borrow<U>::enter (self=0x7f5e2effbda0, f=...) at /home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-current-thread-0.1.6/src/lib.rs:786
#31 tokio_current_thread::Entered<P>::block_on (self=0x7f5e2effbee0, future=...) at /home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-current-thread-0.1.6/src/lib.rs:491
#32 0x000055f3b32b4370 in tokio::runtime::current_thread::runtime::Runtime::block_on::{{closure}} (executor=0x7f5e2effbee0) at /home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.1.22/src/runtime/current_thread/runtime.rs:200
#33 tokio::runtime::current_thread::runtime::Runtime::enter::{{closure}}::{{closure}}::{{closure}}::{{closure}} (enter=<optimized out>) at /home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.1.22/src/runtime/current_thread/runtime.rs:241
#34 tokio_executor::global::with_default::{{closure}} (cell=<optimized out>) at /home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-executor-0.1.10/src/global.rs:221
#35 std::thread::local::LocalKey<T>::try_with (self=<optimized out>, f=...) at /rustc/d5a82bbd26e1ad8b7401f6a718a9c57c96905483/library/std/src/thread/local.rs:446
#36 std::thread::local::LocalKey<T>::with (self=<optimized out>, f=...) at /rustc/d5a82bbd26e1ad8b7401f6a718a9c57c96905483/library/std/src/thread/local.rs:422
#37 0x000055f3b32b630d in tokio_executor::global::with_default (executor=0x7f5e2effc0a8, enter=0x7f5e2effc300, f=...) at /home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-executor-0.1.10/src/global.rs:190
#38 tokio::runtime::current_thread::runtime::Runtime::enter::{{closure}}::{{closure}}::{{closure}} (enter=0x7f5e2effc300) at /home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.1.22/src/runtime/current_thread/runtime.rs:239
#39 tokio_timer::timer::handle::with_default (handle=<optimized out>, enter=0x7f5e2effc300, f=...) at /home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-timer-0.2.13/src/timer/handle.rs:74
#40 0x000055f3b32b5e83 in tokio::runtime::current_thread::runtime::Runtime::enter::{{closure}}::{{closure}} (enter=0x7f5e2effc300) at /home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.1.22/src/runtime/current_thread/runtime.rs:232
#41 tokio_timer::clock::clock::with_default (clock=<optimized out>, enter=0x7f5e2effc300, f=...) at /home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-timer-0.2.13/src/clock/clock.rs:125
#42 0x000055f3b32b4813 in tokio::runtime::current_thread::runtime::Runtime::enter::{{closure}} (enter=0x7f5e2effc300) at /home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.1.22/src/runtime/current_thread/runtime.rs:231
#43 tokio_reactor::with_default (handle=<optimized out>, enter=0x7f5e2effc300, f=...) at /home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-reactor-0.1.12/src/lib.rs:220
#44 0x000055f3b32b3a70 in tokio::runtime::current_thread::runtime::Runtime::enter (self=<optimized out>, f=...) at /home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.1.22/src/runtime/current_thread/runtime.rs:230
#45 0x000055f3b32b15d3 in tokio::runtime::current_thread::runtime::Runtime::block_on (self=0x7f5e2effc4c8, f=...) at /home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.1.22/src/runtime/current_thread/runtime.rs:198
#46 tokio::runtime::current_thread::block_on_all (future=...) at /home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.1.22/src/runtime/current_thread/mod.rs:92
#47 0x000055f3b32b091c in ethcore_io::worker::Worker::new::{{closure}} () at crates/runtime/io/src/worker.rs:108
#48 std::sys_common::backtrace::__rust_begin_short_backtrace (f=...) at /rustc/d5a82bbd26e1ad8b7401f6a718a9c57c96905483/library/std/src/sys_common/backtrace.rs:121
#49 0x000055f3b32ab9d1 in std::thread::Builder::spawn_unchecked_::{{closure}}::{{closure}} () at /rustc/d5a82bbd26e1ad8b7401f6a718a9c57c96905483/library/std/src/thread/mod.rs:550
#50 <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once (self=<error reading variable: access outside bounds of object referenced via synthetic pointer>, _args=<optimized out>) at /rustc/d5a82bbd26e1ad8b7401f6a718a9c57c96905483/library/core/src/panic/unwind_safe.rs:271
#51 std::panicking::try::do_call (data=<optimized out>) at /rustc/d5a82bbd26e1ad8b7401f6a718a9c57c96905483/library/std/src/panicking.rs:483
#52 std::panicking::try (f=<error reading variable: access outside bounds of object referenced via synthetic pointer>) at /rustc/d5a82bbd26e1ad8b7401f6a718a9c57c96905483/library/std/src/panicking.rs:447
#53 std::panic::catch_unwind (f=<error reading variable: access outside bounds of object referenced via synthetic pointer>) at /rustc/d5a82bbd26e1ad8b7401f6a718a9c57c96905483/library/std/src/panic.rs:137
#54 std::thread::Builder::spawn_unchecked_::{{closure}} () at /rustc/d5a82bbd26e1ad8b7401f6a718a9c57c96905483/library/std/src/thread/mod.rs:549
#55 core::ops::function::FnOnce::call_once{{vtable-shim}} () at /rustc/d5a82bbd26e1ad8b7401f6a718a9c57c96905483/library/core/src/ops/function.rs:507
#56 0x000055f3b419ad13 in <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once () at library/alloc/src/boxed.rs:2000
#57 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once () at library/alloc/src/boxed.rs:2000
#58 std::sys::unix::thread::Thread::new::thread_start () at library/std/src/sys/unix/thread.rs:108
#59 0x00007f5e8d048609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#60 0x00007f5e8ce18133 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95



Thread 24 (Thread 0x7f5e2ffff700 (LWP 682816)):
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x000055f3b3e9d8c0 in parking_lot_core::thread_parker::imp::ThreadParker::futex_wait (self=0x7f5e2ffff4b8, ts=...) at src/thread_parker/linux.rs:112
#2  <parking_lot_core::thread_parker::imp::ThreadParker as parking_lot_core::thread_parker::ThreadParkerT>::park (self=0x7f5e2ffff4b8) at src/thread_parker/linux.rs:66
#3  0x000055f3b3e9c429 in parking_lot_core::parking_lot::deadlock_impl::on_unpark (td=<optimized out>) at src/parking_lot.rs:1187
#4  0x000055f3b2e7b8a5 in parking_lot_core::parking_lot::deadlock::on_unpark (_td=0x7f5e2ffff460) at /home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/parking_lot_core-0.8.3/src/parking_lot.rs:1112
#5  parking_lot_core::parking_lot::park::{{closure}} (thread_data=0x7f5e2ffff460) at /home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/parking_lot_core-0.8.3/src/parking_lot.rs:613
#6  parking_lot_core::parking_lot::with_thread_data (f=...) at /home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/parking_lot_core-0.8.3/src/parking_lot.rs:183
#7  parking_lot_core::parking_lot::park (key=94505232111040, park_token=..., timeout=..., validate=..., before_sleep=..., timed_out=...) at /home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/parking_lot_core-0.8.3/src/parking_lot.rs:576
#8  parking_lot::raw_rwlock::RawRwLock::lock_common (self=0x55f3b6ca15c0, timeout=..., token=..., validate_flags=12, try_lock=...) at src/raw_rwlock.rs:1110
#9  parking_lot::raw_rwlock::RawRwLock::lock_exclusive_slow (self=0x55f3b6ca15c0, timeout=...) at src/raw_rwlock.rs:628
#10 0x000055f3b38d9de5 in <parking_lot::raw_rwlock::RawRwLock as lock_api::rwlock::RawRwLock>::lock_exclusive (self=0x55f3b6ca15c0) at /home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/parking_lot-0.11.1/src/raw_rwlock.rs:74
#11 lock_api::rwlock::RwLock<R,T>::write (self=0x55f3b6ca15c0) at /home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/lock_api-0.4.3/src/rwlock.rs:461
#12 ethcore::engines::hbbft::hbbft_engine::HoneyBadgerBFT::join_hbbft_epoch (self=0x55f3b6ca1540) at crates/ethcore/src/engines/hbbft/hbbft_engine.rs:709
#13 0x000055f3b38de00a in ethcore::engines::hbbft::hbbft_engine::HoneyBadgerBFT::process_hb_message (self=0x55f3b6ca1540, msg_idx=<optimized out>, message=..., sender_id=...) at crates/ethcore/src/engines/hbbft/hbbft_engine.rs:545
#14 <ethcore::engines::hbbft::hbbft_engine::HoneyBadgerBFT as ethcore::engines::Engine<ethcore::machine::impls::EthereumMachine>>::handle_message (self=0x55f3b6ca1540, message=..., node_id=...) at crates/ethcore/src/engines/hbbft/hbbft_engine.rs:1304
#15 0x000055f3b368fbd7 in <ethcore::client::client::Client as ethcore::client::traits::IoClient>::queue_consensus_message::{{closure}} (client=<optimized out>) at crates/ethcore/src/client/client.rs:2975
#16 ethcore::client::client::IoChannelQueue::queue::{{closure}} (client=<optimized out>) at crates/ethcore/src/client/client.rs:3519
#17 0x000055f3b32aa7d1 in <ethcore_service::service::ClientIoHandler as ethcore_io::IoHandler<ethcore::client::io_message::ClientIoMessage>>::message (self=0x55f3b6c6c170, _io=<optimized out>, net_message=0x7f5de000c350) at crates/ethcore/service/src/service.rs:229
#18 0x000055f3b32b3858 in ethcore_io::worker::Worker::do_work (work=..., channel=...) at crates/runtime/io/src/worker.rs:139
#19 0x000055f3b32b3e7f in ethcore_io::worker::Worker::new::{{closure}}::{{closure}} () at crates/runtime/io/src/worker.rs:98
#20 0x000055f3b32b08a9 in futures::future::loop_fn::loop_fn (initial_state=..., func=...) at /home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.29/src/future/loop_fn.rs:79
#21 ethcore_io::worker::Worker::new::{{closure}} () at crates/runtime/io/src/worker.rs:86
#22 std::sys_common::backtrace::__rust_begin_short_backtrace (f=...) at /rustc/d5a82bbd26e1ad8b7401f6a718a9c57c96905483/library/std/src/sys_common/backtrace.rs:121
#23 0x000055f3b32ab9d1 in std::thread::Builder::spawn_unchecked_::{{closure}}::{{closure}} () at /rustc/d5a82bbd26e1ad8b7401f6a718a9c57c96905483/library/std/src/thread/mod.rs:550
#24 <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once (self=<error reading variable: access outside bounds of object referenced via synthetic pointer>, _args=<optimized out>) at /rustc/d5a82bbd26e1ad8b7401f6a718a9c57c96905483/library/core/src/panic/unwind_safe.rs:271
#25 std::panicking::try::do_call (data=<optimized out>) at /rustc/d5a82bbd26e1ad8b7401f6a718a9c57c96905483/library/std/src/panicking.rs:483
#26 std::panicking::try (f=<error reading variable: access outside bounds of object referenced via synthetic pointer>) at /rustc/d5a82bbd26e1ad8b7401f6a718a9c57c96905483/library/std/src/panicking.rs:447
#27 std::panic::catch_unwind (f=<error reading variable: access outside bounds of object referenced via synthetic pointer>) at /rustc/d5a82bbd26e1ad8b7401f6a718a9c57c96905483/library/std/src/panic.rs:137
#28 std::thread::Builder::spawn_unchecked_::{{closure}} () at /rustc/d5a82bbd26e1ad8b7401f6a718a9c57c96905483/library/std/src/thread/mod.rs:549
#29 core::ops::function::FnOnce::call_once{{vtable-shim}} () at /rustc/d5a82bbd26e1ad8b7401f6a718a9c57c96905483/library/core/src/ops/function.rs:507
#30 0x000055f3b419ad13 in <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once () at library/alloc/src/boxed.rs:2000
#31 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once () at library/alloc/src/boxed.rs:2000
#32 std::sys::unix::thread::Thread::new::thread_start () at library/std/src/sys/unix/thread.rs:108
#33 0x00007f5e8d048609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#34 0x00007f5e8ce18133 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

@SurfingNerd
Copy link
Collaborator Author

switching to std::sync resulted in the same deadlock situation.

parking_lot deadlock detection could not detect a deadlock, even it existed.

but after changing try_lock, the deadlock detection is able to find deadlocks again:

(checkin: d71be72)

023-03-17 20:56:07 UTC DeadlockDetection WARN diamond_node  1 deadlock(s) detected
2023-03-17 20:56:07 UTC DeadlockDetection WARN diamond_node  Deadlock #0
2023-03-17 20:56:07 UTC DeadlockDetection WARN diamond_node  Thread Id 139671665374976
2023-03-17 20:56:07 UTC DeadlockDetection WARN diamond_node     0:     0x5566001b13bc - backtrace::backtrace::libunwind::trace::h64cea6bb0e2da25e
                               at /home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.56/src/backtrace/libunwind.rs:90:5
                           backtrace::backtrace::trace_unsynchronized::hbadc31359dc71a0d
                               at /home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.56/src/backtrace/mod.rs:66:5
                           backtrace::backtrace::trace::hdd36da1833347942
                               at /home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.56/src/backtrace/mod.rs:53:14
                           backtrace::capture::Backtrace::create::hbabe7b7ff540ec1b
                               at /home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.56/src/capture.rs:176:9
                           backtrace::capture::Backtrace::new::hc18be4f6ec18f39d
                               at /home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.56/src/capture.rs:140:22
   1:     0x5566001a57cc - parking_lot_core::parking_lot::deadlock_impl::on_unpark::hc64b111c0a867e5c
                               at /home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/parking_lot_core-0.8.3/src/parking_lot.rs:1179:32
   2:     0x5565ff1848a5 - parking_lot_core::parking_lot::deadlock::on_unpark::h1490c8b38d698178
                               at /home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/parking_lot_core-0.8.3/src/parking_lot.rs:1112:9
                           parking_lot_core::parking_lot::park::{{closure}}::h6fba80aa10e5f37c
                               at /home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/parking_lot_core-0.8.3/src/parking_lot.rs:613:17
                           parking_lot_core::parking_lot::with_thread_data::ha42d0cde0f40b9e6
                               at /home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/parking_lot_core-0.8.3/src/parking_lot.rs:183:5
                           parking_lot_core::parking_lot::park::h246750c5955d20b5
                               at /home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/parking_lot_core-0.8.3/src/parking_lot.rs:576:5
                           parking_lot::raw_rwlock::RawRwLock::lock_common::h94c35aa760b1242e
                               at /home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/parking_lot-0.11.1/src/raw_rwlock.rs:1110:17
                           parking_lot::raw_rwlock::RawRwLock::lock_exclusive_slow::h80a06b2b28e36c28
                               at /home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/parking_lot-0.11.1/src/raw_rwlock.rs:628:26
   3:     0x5565ffbbefc5 - <parking_lot::raw_rwlock::RawRwLock as lock_api::rwlock::RawRwLock>::lock_exclusive::hc0723659f4f73d7f
                               at /home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/parking_lot-0.11.1/src/raw_rwlock.rs:74:26
                           lock_api::rwlock::RwLock<R,T>::write::hc0f05eb42ddb4977
                               at /home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/lock_api-0.4.6/src/rwlock.rs:480:9
                           ethcore::engines::hbbft::hbbft_engine::HoneyBadgerBFT::join_hbbft_epoch::h977abbda4456d6f8
                               at /home/sn/dmd/diamond-node/crates/ethcore/src/engines/hbbft/hbbft_engine.rs:717:20
   4:     0x5565ffbc326a - ethcore::engines::hbbft::hbbft_engine::HoneyBadgerBFT::process_hb_message::hfddfa1d7f6b1625b
                               at /home/sn/dmd/diamond-node/crates/ethcore/src/engines/hbbft/hbbft_engine.rs:545:17
                           <ethcore::engines::hbbft::hbbft_engine::HoneyBadgerBFT as ethcore::engines::Engine<ethcore::machine::impls::EthereumMachine>>::handle_message::hfe565c8eb08b3ca0
                               at /home/sn/dmd/diamond-node/crates/ethcore/src/engines/hbbft/hbbft_engine.rs:1339:17
   5:     0x5565ffb20fa7 - <ethcore::client::client::Client as ethcore::client::traits::IoClient>::queue_consensus_message::{{closure}}::h7ae9a9fa442d0eed
                               at /home/sn/dmd/diamond-node/crates/ethcore/src/client/client.rs:2975:33
                           ethcore::client::client::IoChannelQueue::queue::{{closure}}::h8259051c7852f67a
                               at /home/sn/dmd/diamond-node/crates/ethcore/src/client/client.rs:3519:13
   6:     0x5565ff5b7811 - <ethcore_service::service::ClientIoHandler as ethcore_io::IoHandler<ethcore::client::io_message::ClientIoMessage>>::message::hd5cefedceb582fb4
                               at /home/sn/dmd/diamond-node/crates/ethcore/service/src/service.rs:229:17
   7:     0x5565ff5bd308 - ethcore_io::worker::Worker::do_work::hf76f92698174c756
                               at /home/sn/dmd/diamond-node/crates/runtime/io/src/worker.rs:139:17
   8:     0x5565ff5beb8f - ethcore_io::worker::Worker::new::{{closure}}::{{closure}}::h457bd4d04f55de17
                               at /home/sn/dmd/diamond-node/crates/runtime/io/src/worker.rs:98:41
   9:     0x5565ff5bdff9 - futures::future::loop_fn::loop_fn::ha2db9ebcb0d2d5e6
                               at /home/sn/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.29/src/future/loop_fn.rs:79:17
                           ethcore_io::worker::Worker::new::{{closure}}::hab4f088dc75e0be2
                               at /home/sn/dmd/diamond-node/crates/runtime/io/src/worker.rs:86:25
                           std::sys_common::backtrace::__rust_begin_short_backtrace::hcbf69162312046f2
                               at /rustc/d5a82bbd26e1ad8b7401f6a718a9c57c96905483/library/std/src/sys_common/backtrace.rs:121:18
  10:     0x5565ff5bab61 - std::thread::Builder::spawn_unchecked_::{{closure}}::{{closure}}::h6cdbf9dda15c33d1
                               at /rustc/d5a82bbd26e1ad8b7401f6a718a9c57c96905483/library/std/src/thread/mod.rs:550:17
                           <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once::hb2bdf77ef5f1644a
                               at /rustc/d5a82bbd26e1ad8b7401f6a718a9c57c96905483/library/core/src/panic/unwind_safe.rs:271:9
                           std::panicking::try::do_call::hf53409f528a81a04
                               at /rustc/d5a82bbd26e1ad8b7401f6a718a9c57c96905483/library/std/src/panicking.rs:483:40
                           std::panicking::try::h143f7fcc467e44af
                               at /rustc/d5a82bbd26e1ad8b7401f6a718a9c57c96905483/library/std/src/panicking.rs:447:19
                           std::panic::catch_unwind::h812c86211caa5c20
                               at /rustc/d5a82bbd26e1ad8b7401f6a718a9c57c96905483/library/std/src/panic.rs:137:14
                           std::thread::Builder::spawn_unchecked_::{{closure}}::hdbec87ebf0680859
                               at /rustc/d5a82bbd26e1ad8b7401f6a718a9c57c96905483/library/std/src/thread/mod.rs:549:30
                           core::ops::function::FnOnce::call_once{{vtable.shim}}::hb1c3c75dacf961b7
                               at /rustc/d5a82bbd26e1ad8b7401f6a718a9c57c96905483/library/core/src/ops/function.rs:507:5
  11:     0x5566004a4273 - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once::hb77d8d72ebcf79c4
                               at /rustc/d5a82bbd26e1ad8b7401f6a718a9c57c96905483/library/alloc/src/boxed.rs:2000:9
                           <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once::hc08c3353e1568487
                               at /rustc/d5a82bbd26e1ad8b7401f6a718a9c57c96905483/library/alloc/src/boxed.rs:2000:9
                           std::sys::unix::thread::Thread::new::thread_start::h7168e596cd5e5ce6
                               at /rustc/d5a82bbd26e1ad8b7401f6a718a9c57c96905483/library/std/src/sys/unix/thread.rs:108:17
  12:     0x7f0835bbe609 - start_thread
  13:     0x7f083598e133 - clone
  14:                0x0 - <unknown>

@SurfingNerd
Copy link
Collaborator Author

version with a first cleanup that does not deadlock anymore:
2b65638

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@SurfingNerd