EthersDB deadlock

[publicqi]

repro: https://github.com/publicqi/revm_hang

If a EthersDB is constructed in an async test function (or single-threaded executor) with no block number specified, the program will hang.

#[tokio::test] uses a single-threaded runtime. revm submits the get_block_number using futures::executor::block_on, but this task will never execute since the runtime is always polling the previous task (test_hang, in this case).

Some refs when digging into the bug:
tokio-rs/tokio#2603
tokio-rs/tokio#2376

[publicqi]

I think I encountered another scenario triggering the deadlock outside single-threaded executor. I haven't found the root cause yet. Will update here.

Currently my guess is that if this bug exists in single-threaded executor, it should also happen in multi-threaded but just at a smaller chance.

[publicqi]

I added a min repro:

publicqi/revm_hang@2b736a7

[Otto-AA]

I just started to try out revm and found that the generate_block_traces.rs example gets stuck while executing the transactions.

From what I've found (as a Rust newbie):

• It's deterministic, ie in the normal setup it always gets stuck at tx 5, if I skip the first two transactions it gets stuck at tx 8. So it's not a problem of the transactions per se
• From the provider, the JsonRpcClient.request seems to wait infinitely for the response
• Further up the stacktrace, it's stuck in ethersdb futures::executor::block_on(f)

Am I correct, that this is another case of this issue?

[rakita]

can you test if #1089 fixes it?

[Otto-AA]

No, using this branch it panics with following error:

thread 'main' panicked at /home/oaie/.cargo/git/checkouts/revm-d03068627b7bf933/3c2af5f/crates/revm/src/db/ethersdb.rs:38:43:
Cannot start a runtime from within a runtime. This happens because a function (like block_on) attempted to block the current thread while the thread is being used to drive asynchronous tasks.

stack backtrace:

   0: rust_begin_unwind
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panicking.rs:645:5
   1: core::panicking::panic_fmt
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/core/src/panicking.rs:72:14
   2: tokio::runtime::context::runtime::enter_runtime
             at /home/oaie/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.36.0/src/runtime/context/runtime.rs:68:5
   3: tokio::runtime::handle::Handle::block_on
             at /home/oaie/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.36.0/src/runtime/handle.rs:309:9
   4: revm::db::ethersdb::EthersDB<M>::block_on
             at /home/oaie/.cargo/git/checkouts/revm-d03068627b7bf933/3c2af5f/crates/revm/src/db/ethersdb.rs:38:9
   5: <revm::db::ethersdb::EthersDB<M> as revm_primitives::db::DatabaseRef>::basic_ref
             at /home/oaie/.cargo/git/checkouts/revm-d03068627b7bf933/3c2af5f/crates/revm/src/db/ethersdb.rs:54:38
   6: <revm::db::in_memory_db::CacheDB<ExtDB> as revm_primitives::db::Database>::basic
             at /home/oaie/.cargo/git/checkouts/revm-d03068627b7bf933/3c2af5f/crates/revm/src/db/in_memory_db.rs:173:17
   7: revm::db::states::state::State<DB>::load_cache_account
             at /home/oaie/.cargo/git/checkouts/revm-d03068627b7bf933/3c2af5f/crates/revm/src/db/states/state.rs:186:28
   8: <revm::db::states::state::State<DB> as revm_primitives::db::Database>::basic
             at /home/oaie/.cargo/git/checkouts/revm-d03068627b7bf933/3c2af5f/crates/revm/src/db/states/state.rs:219:9
   9: revm_primitives::db::_::<impl revm_primitives::db::Database for &mut T>::basic
             at /home/oaie/.cargo/git/checkouts/revm-d03068627b7bf933/3c2af5f/crates/primitives/src/db.rs:10:1
  10: revm::journaled_state::JournaledState::load_account
             at /home/oaie/.cargo/git/checkouts/revm-d03068627b7bf933/3c2af5f/crates/revm/src/journaled_state.rs:538:54
  11: revm::handler::mainnet::validation::validate_tx_against_state
             at /home/oaie/.cargo/git/checkouts/revm-d03068627b7bf933/3c2af5f/crates/revm/src/handler/mainnet/validation.rs:22:31
  12: core::ops::function::Fn::call
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/core/src/ops/function.rs:79:5
  13: revm::handler::handle_types::validation::ValidationHandler<EXT,DB>::tx_against_state
             at /home/oaie/.cargo/git/checkouts/revm-d03068627b7bf933/3c2af5f/crates/revm/src/handler/handle_types/validation.rs:58:9
  14: revm::evm::Evm<EXT,DB>::transact
             at /home/oaie/.cargo/git/checkouts/revm-d03068627b7bf933/3c2af5f/crates/revm/src/evm.rs:166:9
  15: revm::evm::Evm<EXT,DB>::transact_commit
             at /home/oaie/.cargo/git/checkouts/revm-d03068627b7bf933/3c2af5f/crates/revm/src/evm.rs:49:48
  16: revm_replayer::main::{{closure}}
             at ./src/main.rs:169:29
  17: tokio::runtime::park::CachedParkThread::block_on::{{closure}}
             at /home/oaie/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.36.0/src/runtime/park.rs:281:63
  18: tokio::runtime::coop::with_budget
             at /home/oaie/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.36.0/src/runtime/coop.rs:107:5
  19: tokio::runtime::coop::budget
             at /home/oaie/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.36.0/src/runtime/coop.rs:73:5
  20: tokio::runtime::park::CachedParkThread::block_on
             at /home/oaie/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.36.0/src/runtime/park.rs:281:31
  21: tokio::runtime::context::blocking::BlockingRegionGuard::block_on
             at /home/oaie/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.36.0/src/runtime/context/blocking.rs:66:9
  22: tokio::runtime::scheduler::multi_thread::MultiThread::block_on::{{closure}}
             at /home/oaie/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.36.0/src/runtime/scheduler/multi_thread/mod.rs:87:13
  23: tokio::runtime::context::runtime::enter_runtime
             at /home/oaie/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.36.0/src/runtime/context/runtime.rs:65:16
  24: tokio::runtime::scheduler::multi_thread::MultiThread::block_on
             at /home/oaie/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.36.0/src/runtime/scheduler/multi_thread/mod.rs:86:9
  25: tokio::runtime::runtime::Runtime::block_on
             at /home/oaie/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.36.0/src/runtime/runtime.rs:350:45
  26: revm_replayer::main
             at ./src/main.rs:185:5
  27: core::ops::function::FnOnce::call_once
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/core/src/ops/function.rs:250:5
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

[rakita]

No, using this branch it panics with following error:

thread 'main' panicked at /home/oaie/.cargo/git/checkouts/revm-d03068627b7bf933/3c2af5f/crates/revm/src/db/ethersdb.rs:38:43:
Cannot start a runtime from within a runtime. This happens because a function (like block_on) attempted to block the current thread while the thread is being used to drive asynchronous tasks.

stack backtrace:

Should just make functions async, this would fix it.

[publicqi]

Some updates: If I change the main function in https://github.com/publicqi/revm_hang to synchronous main (not the tokio::main it's using right now), it works fine.

So the hang happens when

async main() -> sync ethers_db.storage() -> block_on async inner call

I'm not familiar with async internals. A temp fix in my code is to not to call ethersdb's synchronous functions inside an async function.

[wtdcode]

#1135 shall fix this by removing futures::executor::block_on as suggested in tokio-rs/tokio#2376 without breaking any code. @rakita

Quote:

Calling futures::executor::block_on(...) should never be done from within the context of a tokio task. Unfortunately, we do not control that function as it is from futures. If we did, we would make it panic.