panicked at 'Storage root must match that calculated.'

[crystalin]

We have 2 relay node.
We have 1 collator node (cumulu 96da14c)
We have 1 rpc node (same command as collator but without --validator)

It is very similar to the node template, but with the Frontier (Ethereum) pallet included

When we perform some tests on the rpc node (using Ethereum deploying contracts, transferring, calling contracts..._), it inconsistency fail with this error. (which then enters in a loop):

2020-09-14 19:18:54.273 tokio-runtime-worker DEBUG evm  Execution Succeed(Stopped) [source: 0x0848b04952763ff8ae726d4a910cb55243bf2427, value: 1000000000000000000, gas_li[4547/98211]used_gas: 71280, actual_fee: 0]                                                                                                                                                       2020-09-14 19:18:54.273 tokio-runtime-worker DEBUG evm  Updating storage for 0x38b32fc736babf8b3fe21af6cadb20992e0d4ef8 [index: 0x0000000000000000000000000000000000000000000000000000000000000000, value: 0x0000000000000000000000000000000000000000000000000000000000000004]
2020-09-14 19:18:54.273 tokio-runtime-worker DEBUG evm  Updating storage for 0x38b32fc736babf8b3fe21af6cadb20992e0d4ef8 [index: 0x1115d2871967fb8871f9118d2cbbd8a4929a50e53357b9b505987b18e433d712, value: 0x0000000000000000000000000000000000000000000000000000000000000001]
2020-09-14 19:18:54.273 tokio-runtime-worker DEBUG evm  Updating storage for 0x38b32fc736babf8b3fe21af6cadb20992e0d4ef8 [index: 0x290decd9548b62a8d60345a988386fc84ba6bc95484008f6362f93160ef3e566, value: 0x0000000000000000000000000848b04952763ff8ae726d4a910cb55243bf2427]
Hash: given=e93de8f5dd5f3ab6c21408bfd6caa57043f2a3a69b13fabe94f50140abb03e04, expected=d3c07cf589b2fd0906f4fcea298432c57738a85cf95a270a6cb518e24c34c558

====================

Version: 0.1.0-6d695cc-x86_64-linux-gnu

   0: sp_panic_handler::set::{{closure}}
   1: std::panicking::rust_panic_with_hook
             at /rustc/d3fb005a39e62501b8b0b356166e515ae24e2e54/src/libstd/panicking.rs:490
   2: std::panicking::begin_panic
   3: frame_executive::Executive<System,Block,Context,UnsignedValidator,AllModules,COnRuntimeUpgrade>::execute_block
   4: <moonbase_runtime::Runtime as sp_api::runtime_decl_for_Core::Core<sp_runtime::generic::block::Block<sp_runtime::generic::header::Header<u32,sp_runtime::traits::BlakeTwo256>,sp_runtime::generic::unchecked_extrinsic::UncheckedExtrinsic<<<sp_runtime::MultiSignature as sp_runtime::traits::Verify>::Signer as sp_runtime::traits::IdentifyAccount>::AccountId,moonbase_runtime::Call,sp_runtime::MultiSignature,(frame_system::extensions::check_spec_version::CheckSpecVersion<moonbase_runtime::Runtime>,frame_system::extensions::check_genesis::CheckGenesis<moonbase_runtime::Runtime>,frame_system::extensions::check_mortality::CheckMortality<moonbase_runtime::Runtime>,frame_system::extensions::check_nonce::CheckNonce<moonbase_runtime::Runtime>,frame_system::extensions::check_weight::CheckWeight<moonbase_runtime::Runtime>,pallet_transaction_payment::ChargeTransactionPayment<moonbase_runtime::Runtime>)>>>>::execute_block
   5: sp_api::runtime_decl_for_Core::execute_block_native_call_generator::{{closure}}
   6: std::thread::local::LocalKey<T>::with
   7: sc_executor::native_executor::WasmExecutor::with_instance::{{closure}}
   8: sc_executor::wasm_runtime::RuntimeCache::with_instance
   9: <sc_executor::native_executor::NativeExecutor<D> as sp_core::traits::CodeExecutor>::call
  10: sp_state_machine::StateMachine<B,H,N,Exec>::execute_aux
  11: sp_state_machine::StateMachine<B,H,N,Exec>::execute_using_consensus_failure_handler
  12: <sc_service::client::call_executor::LocalCallExecutor<B,E> as sc_client_api::call_executor::CallExecutor<Block>>::contextual_call
  13: <sc_service::client::client::Client<B,E,Block,RA> as sp_api::CallApiAt<Block>>::call_api_at
  14: sp_api::Core::execute_block_with_context
  15: <&sc_service::client::client::Client<B,E,Block,RA> as sp_consensus::block_import::BlockImport<Block>>::import_block
  16: <alloc::sync::Arc<T> as sp_consensus::block_import::BlockImport<B>>::import_block
  17: <futures_util::future::poll_fn::PollFn<F> as core::future::future::Future>::poll
  18: <futures_util::future::future::map::Map<Fut,F> as core::future::future::Future>::poll
  19: <futures_util::future::future::flatten::Flatten<Fut,<Fut as core::future::future::Future>::Output> as core::future::future::Future>::poll
  20: <futures_util::future::poll_fn::PollFn<F> as core::future::future::Future>::poll
  21: std::panicking::try
  22: <futures_util::future::select::Select<A,B> as core::future::future::Future>::poll
  23: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
  24: std::thread::local::LocalKey<T>::with
  25: futures_executor::local_pool::block_on
  26: tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut
  27: <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
  28: tokio::runtime::task::raw::poll
  29: tokio::runtime::blocking::pool::Inner::run
  30: tokio::runtime::context::enter
  31: std::sys_common::backtrace::__rust_begin_short_backtrace
  32: core::ops::function::FnOnce::call_once{{vtable.shim}}
  33: <alloc::boxed::Box<F> as core::ops::function::FnOnce<A>>::call_once
             at /rustc/d3fb005a39e62501b8b0b356166e515ae24e2e54/src/liballoc/boxed.rs:1076
      <alloc::boxed::Box<F> as core::ops::function::FnOnce<A>>::call_once
             at /rustc/d3fb005a39e62501b8b0b356166e515ae24e2e54/src/liballoc/boxed.rs:1076
      std::sys::unix::thread::Thread::new::thread_start
             at /rustc/d3fb005a39e62501b8b0b356166e515ae24e2e54/src/libstd/sys/unix/thread.rs:87
  34: start_thread
  35: __clone


Thread 'tokio-runtime-worker' panicked at 'Storage root must match that calculated.', /rustc/d3fb005a39e62501b8b0b356166e515ae24e2e54/src/libstd/macros.rs:13

This doesn't happen on the collator itself, only on the rpc node.

It doesn't seem to happen if we run the rpc node as a collator too (but because it is inconsistent, I'll have to confirm that after a day or 2)

[crystalin]

Ok, the --validator didn't help at the end. It is still failing with that error

[bkchr]

Can you share your parachain implementation?

And is the network public? So, that I can sync it on my own?

[crystalin]

The network is private for now as it is very early stage, and we haven't shared yet our implementation (we had to use a different repo as it required too many changes to support cumulus and ...).
Our implementation is pretty standard except for the frontier part that is added. And looking at the error I believe it is related as Frontier is storing data in the auxilary database, and the stacktrace shows an "execute_aux" call in it.

I can prepare a docker image with the full network running on it and a parachain binary to connect to if that helps.

Also another case of the same error (but didn't print the stacktrace this time so not sure if it is related to the same):

Sep 14, 2020 @ 17:02:48.209	2020-09-14 21:02:48.209 tokio-runtime-worker WARN sync  💔 Error importing block 0xc464db9a293a1f4794f702e697c7177d3022d97c59ce41249ef7bdcb4454f783: Err(UnknownParent)
Sep 14, 2020 @ 17:02:48.303	2020-09-14 21:02:48.303 tokio-runtime-worker INFO substrate  ✨ [Parachain] Imported #215 (0x42cf…aeb7)
Sep 14, 2020 @ 17:02:48.321	ogsBloom":"0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000","miner":"0x0000000000000000000000000000000000000000","number":"0xbb","parentHash":"0x0ce3d63e1adc0f75083a304c21c820f6bb6b345cf1c3950ce6048782e2b79b03","receiptsRoot":"0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347","sealFields":["0x0000000000000000000000000000000000000000000000000000000000000000","0x0000000000000000"],"sha3Uncles":"0x0000000000000000000000000000000000000000000000000000000000000000","size":"0x21c","stateRoot":"0x0000000000000000000000000000000000000000000000000000000000000000","timestamp":"0x5f5fd9c4","totalDifficulty":null,"transactions":[],"transactionsRoot":"0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347","uncles":[]},"id":"1274"},{"jsonrpc":"2.0","result":{"author":"0x0000000000000000000000000000000000000000","difficulty":"0x0","extraData":"0x0000000000000000000000000000000000000000000000000000000000000000","gasLimit":"0x0","gasUsed":"0x0","hash":"0xc9b4024d60841c3440bc792f5d25c4448602bfba9d5612d8e5cc85b312e0f9b9","logsBloom":"0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000","miner":"0x0000000000000000000000000000000000000000","number":"0xbc","parentHash":"0xcf7f6779b1d597117f2e61dc206e9b9c7e6b5d749fbd810ba6bf28bf6c4f859f","receiptsRoot":"0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347","sealFields":["0x0000000000000000000000000000000000000000000000000000000000000000","0x0000000000000000"],"sha3Uncles":"0x0000000000000000000000000000000000000000000000000000000000000000","size":"0x21c","stateRoot":"0x0000000000000000000000000000000000000000000000000000000000000000","timestamp":"0x5f5fd9ca","totalDifficulty":null,"transactions":[],"transactionsRoot":"0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347","uncles":[]},"id":"1275"},{"jsonrpc":"2.0","result":{"author":"0x0000000000000000000000000000000000000000","difficulty":"0x0","extraData":"0x0000000000000000000000000000000000000000000000000000000000000000","gasLimit":"0x0","gasUsed":"0x0","hash":"0xf6e7cdbf3d797c2790d5962e3cab43e0d8bd7c1caf91b0a4ce6848243bf6ddcb","logsBloom":"0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000","miner":"0x0000000000000000000000000000000000000000","number":"0xbd","parentHash":"0xeaec4a9df14fe9bc60475d28c2ef97ddb947275b741fbd5b8c28987bfbeb9e89","receiptsRoot":"0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347","sealFields":["0x0000000000000000000000000000000000000000000000000000000000000000","0x0000000000000000"],"sha3Uncles":"0x0000000000000000000000000000000000000000000000000000000000000000","size":"0x21c","stateRoot":"0x0000000000000000000000000000000000000000000000000000000000000000","timestamp":"0x5f5fd9d0","totalDifficulty":null,"transactions":[],"transactionsRoot":"0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347","uncles":[]},"id":"1276"},{"jsonrpc":"2.0","result":{"author":"0x0000000000000000000000000000000000000000","difficulty":"0x0","extraData":"0x0000000000000000000000000000000000000000000000000000000000000000","gasLimit":"0x0","gasUsed":"0x0","hash":"0xa3eb1d8e5cdf98e70d5c21e559a0d338cbf59b17c57cdfcd325fe838d337c5e2","logsBloom":"0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000","miner":"0x0000000000000000000000000000000000000000","number":"0xbe","parentHash":"0x98a3dd1c27e9d00a3da80105571bf1848ad60268332366e6856ad114a6dd08e6","receiptsRoot":"0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347","sealFields":["0x0000000000000000000000000000000000000000000000000000000000000000","0x0000000000000000"],"sha3Uncles":"0x0000000000000000000000000000000000000000000000000000000000000000","size":"0x21c","stateRoot":"0x0000000000000000000000000000000000000000000000000000000000000000","timestamp":"0x5f5fd9d6","totalDifficulty":null,"transactions":[],"transactionsRoot":"0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347","uncles":[]},"id":"1277"},{"jsonrpc":"2.0","result":{"author":"0x0000000000000000000000000000000000000000","difficulty":"0x0","extraData":"0x0000000000000000000000000000000000000000000000000000000000000000","gasLimit":"0x0","gasUsed":"0x0","hash":"0xe87380466b98bab46d6bdb1c1e6433e5ce855d56f7ef26d2113927f0ea3c5bcb","logsBloom":"0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000","miner":"0x0000000000000000000000000000000000000000","number":"0xbf","parentHash":"0x7e306d8c083c76b11641a3a375bcfc05571f32356efe611c0632918cd72f6ef2","receiptsRoot":"0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347","sealFields":["0x0000000000000000000000000000000000000000000000000000000000000000","0x0000000000000000"],"sha3Uncles":"0x0000000000000000000000000000000000000000000000000000000000000000","size":"0x21c","stateRoot":"0x0000000000000000000000000000000000000000000000000000000000000000","timestamp":"0x5f5fd9dc","totalDifficulty":null,"transactions":[],"transactionsRoot":"0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347","uncles":[]},"id":"1278"},{"jsonrpc":"2.0","result":{"author":"0x0000000000000000000000000000000000000000","difficulty":"0x0","extraData":"0x0000000000000000000000000000000000000000000000000000000000000000","gasLimit":"0x0","gasUsed":"0x0","hash":"0x012c6ca19d63802612de79ef4b531769301f2aa09e91f90028a844e79d3069b2","logsBloom":"0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000","miner":"0x0000000000000000000000000000000000000000","number":"0xc0","parentHash":"0x1bb9ead95a1c74aab2093896d2d20245170cc4246ccb9da24decf5e25e426bbf","receiptsRoot":"0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347","sealFields":["0x0000000000000000000000000000000000000000000000000000000000000000","0x0000000000000000"],"sha3Uncles":"0x0000000000000000000000000000000000000000000000000000000000000000","size":"0x21c","stateRoot":"0x0000000000000000000000000000000000000000000000000000000000000000","timestamp":"0x5f5fd9e2","totalDifficulty":null,"transactions":[],"transactionsRoot":"0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347","uncles":[]},"id":"1279"},{"jsonrpc":"2.0","result":{"author":"0x0000000000000000000000000000000000000000","difficulty":"0x0","extraData":"0x0000000000000000000000000000000000000000000000000000000000000000","gasLimit":"0x0","gasUsed":"0x0","hash":"0xf69ae585ba5cdbe8fe0bdf1cfacd85444fb2ec3264c5bafde57ce6cae4860156","logsBloom":"0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000","miner":"0x0000000000000000000000000000000000000000","number":"0xc1","parentHash":"0x6529dd53a1b47087194f2677c9693f64d920aa1d8a93c69512f9400114b94276","receiptsRoot":"0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347","sealFields":["0x0000000000000000000000000000000000000000000000000000000000000000","0x0000000000000000"],"sha3Uncles":"0x0000000000000000000000000000000000000000000000000000000000000000","size":"0x21c","stateRoot":"0x0000000000000000000000000000000000000000000000000000000000000000","timestamp":"0x5f5fd9e8","totalDifficulty":null,"transactions":[],"transactionsRoot":"0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347","uncles":[]},"id":"1280"},{"jsonrpc":"2.0","result":{"author":"0x0000000000000000000000000000000000000000","difficulty":"0x0","extraData":"0x0000000000000000000000000000000000000000000000000000000000000000","gasLimit":"0x0","gasUsed":"0x0","hash":"0x90aa8ea7907e0ba67d08fd7b6acc84ba217a570bfe3e6e34b1bedc4b03e0180a","logsBloom":"0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000","miner":"0x0000000000000000000000000000000000000000","number":"0xc2","parentHash":"0xd4256a8c8305a70dfefcdaafdd2047033a97d097762d86ae5dcb56c67c0f5a51","receiptsRoot":"0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347","sealFields":["0x0000000000000000000000000000000000000000000000000000000000000000","0x0000000000000000"],"sha3Uncles":"0x0000000000000000000000000000000000000000000000000000000000000000","size":"0x21c","stateRoot":"0x0000000000000000000000000000000000000000000000000000000000000000","timestamp":"0x5f5fd9ee","totalDifficulty":null,"transactions":[],"transactionsRoot":"0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347","uncles":[]},"id":"1281"},{"jsonrpc":"2.0","result":{"author":"0x0000000000000000000000000000000000000000","difficulty":"0x0","extraData":"0x0000000000000000000000000000000000000000000000000000000000000000","gasLimit":"0x0","gasUsed":"0x0","hash":"0x0f5f8ca44c5b5124c3c38d80033269aca3177e4b302c8659783afeeb35bd9af3","logsBloom":"0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000","miner":"0x0000000000000000000000000000000000000000","number":"0xc3","parentHash":"0x2e9421c554b43ecc97b9cff90181b967370cb8ad02b1b6abef4e2bb32ca104b2","receiptsRoot":"0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347","sealFields":["0x0000000000000000000000000000000000000000000000000000000000000000","0x0000000000000000"],"sha3Uncles":"0x0000000000000000000000000000000000000000000000000000000000000000","size":"0x21c","stateRoot":"0x0000000000000000000000000000000000000000000000000000000000000000","timestamp":"0x5f5fd9f4","totalDifficulty":null,"transactions":[],"transactionsRoot":"0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347","uncles":[]},"id":"1282"},{"jsonrpc":"2.0","result":{"author":"0x0000000000000000000000000000000000000000","difficulty":"0x0","extraData":"0x0000000000000000000000000000000000000000000000000000000000000000","gasLimit":"0x0","gasUsed":"0x0","hash":"0xa76a1eb06f0fb563c01a3f31381a67df06a4b5afd4a33001632b3686115074c4","logsBloom":"0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000","miner":"0x0000000000000000000000000000000000000000","number":"0xc4","parentHash":"0x62685d2d4a471a62717cef00a35f5729d430ccbbc172f4fff07c2fee2296fade","receiptsRoot":"0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347","sealFields":["0x0000000000000000000000000000000000000000000000000000000000000000","0x0000000000000000"],"sha3Uncles":"0x0000000000000000000000000000000000000000000000000000000000000000","size":"0x21c","stateRoot":"0x0000000000000000000000000000000000000000000000000000000000000000","timestamp":"0x5f5fd9fa","totalDifficulty":null,"transactions":[],"transactionsRoot":"0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347","uncles":[]},"id":"1283"},{"jsonrpc":"2.0","result":{"author":"0x0000000000000000000000000000000000000000","difficulty":"0x0","extraData":"0x0000000000000000000000000000000000000000000000000000000000000000","gasLimit":"0x0","gasUsed":"0x0","hash":"0x0d39b4c5062e48ab3eb4ece59a752aea785fcb6122c28072fb773f011cb178dd","logsBloom":"0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000","miner":"0x0000000000000000000000000000000000000000","number":"0xc5","parentHash":"0xc17dffc97d15596e9b34b307449d9daac185628de7c7a9683b930c0a2e3de566","receiptsRoot":"0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347","sealFields":["0x0000000000000000000000000000000000000000000000000000000000000000","0x0000000000000000"],"sha3Uncles":"0x0000000000000000000000000000000000000000000000000000000000000000","size":"0x21c","stateRoot":"0x0000000000000000000000000000000000000000000000000000000000000000","timestamp":"0x5f5fda00","totalDifficulty":null,"transactions":[],"transactionsRoot":"0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347","uncles":[]},"id":"1284"},{"jsonrpc":"2.0","result":{"author":"0x0000000000000000000000000000000000000000","difficulty":"0x0","extraData":"0x0000000000000000000000000000000000000000000000000000000000000000","gasLimit":"0x0","gasUsed":"0x0","hash":"0xf4a304e9f50840a2d657bb0bbdfab85eec1234d3bfe6fe76b5d93ecaf6a13cab","logsBloom":"0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000","miner":"0x0000000000000000000000000000000000000000","number":"0xc6","parentHash":"0x6c8e96bcbf21e1d579f701038f363614ab439b15cf8a40d358ac6870154b2937","receiptsRoot":"0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347","sealFields":["0x0000000000000000000000000000000000000000000000000000000000000000","0x0000000000000000"],"sha3Uncles":"0x0000000000000000000000000000000000000000000000000000000000000000","size":"0x21c","stateRoot":"0x0000000000000000000000000000000000000000000000000000000000000000","timestamp":"0x5f5fda06","totalDifficulty":null,"transactions":[],"transactionsRoot":"0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347","uncles":[]},"id":"1285"},{"jsonrpc":"2.0","result":{"author":"0x0000000000000000000000000000000000000000","difficulty":"0x0","extraData":"0x0000000000000000000000000000000000000000000000000000000000000000","gasLimit":"0x0","gasUsed":"0x0","hash":"0x41b1f10b2bbf9973b58d31f68060cfbf702618a7ac8559152c21726b180266da","logsBloom":"0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000","miner":"0x0000000000000000000000000000000000000000","number":"0xc7","parentHash":"0xf277090b6946d1ac3e49b490bbde8c8ce4d3420104f15ccb0bc7b1128365dd5c","receiptsRoot":"0x78bc580ad1b04e7c8b46a72af423057742242269f03c8ee926314a4c01cdce9b","sealFields":["0x0000000000000000000000000000000000000000000000000000000000000000","0x0000000000000000"],"sha3Uncles":"0x0000000000000000000000000000000000000000000000000000000000000000","size":"0x314","stateRoot":"0x0000000000000000000000000000000000000000000000000000000000000000","timestamp":"0x5f5fda0c","totalDifficulty":null,"transactions":[{"blockHash":"0x41b1f10b2bbf9973b58d31f68060cfbf702618a7ac8559152c21726b180266da","blockNumber":"0xc7","chainId":"0x2b","condition":null,"creates":"0x339973540320db6f819e288c7b29241286899ecb","from":"0x0848b04952763ff8ae726d4a910cb55243bf2427","gas":"0x18f63","gasPrice":"0x0","hash":"0x53809bc201bcbf23a3902f27137c88340af943b09cc463302feff8f0aeace9b2","input":"0x60806040526002600055348015601457600080fd5b506085806100236000396000f3fe6080604052348015600f57600080fd5b506004361060285760003560e01c80638381f58a14602d575b600080fd5b60336049565b6040518082815260200191505060405180910390f35b6000548156fea2646970667358221220b4717705c0aa682d8b256f6c3a5a0fc8bb72284b2c80899d990dcdc61c953e1164736f6c634300060c0033","nonce":"0x3","publicKey":null,"r":"0xcdcbb18a917bc9a724049b91edb1803ae9d1c9145338e0293d6edc2b1ace40dd","raw":"0xf8f5038083018f638080b8a860806040526002600055348015601457600080fd5b506085806100236000396000f3fe6080604052348015600f57600080fd5b506004361060285760003560e01c80638381f58a14602d575b600080fd5b60336049565b6040518082815260200191505060405180910390f35b6000548156fea2646970667358221220b4717705c0aa682d8b256f6c3a5a0fc8bb72284b2c80899d990dcdc61c953e1164736f6c634300060c003379a0cdcbb18a917bc9a724049b91edb1803ae9d1c9145338e0293d6edc2b1ace40dda0610d462b94afa40245f8acb92f09a7598e597faba98de6ec93cce99d63f543ac","s":"0x610d462b94afa40245f8acb92f09a7598e597faba98de6ec93cce99d63f543ac","standardV":"0x0","to":null,"transactionIndex":"0x0","v":"0x79","value":"0x0"}],"transactionsRoot":"0x306d1e9ecd63f0b29b409b11218559e9bd06a153fe045b993962a14e87d69582","uncles":[]},"id":"1286"},{"jsonrpc":"2.0","result":{"author":"0x0000000000000000000000000000000000000000","difficulty":"0x0","extraData":"0x0000000000000000000000000000000000000000000000000000000000000000","gasLimit":"0x0","gasUsed":"0x0","hash":"0x20f1620a57a3c5fc9cafc79fc3e58fe1a37d62a7ae811480717713c0123e50ee","logsBloom":"0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000","miner":"0x0000000000000000000000000000000000000000","number":"0xc8","parentHash":"0x1d2e5b646746f50bf10df0c3f497ce5dc58634620e79c006e9065fb3eafba9d9","receiptsRoot":"0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347","sealFields":["0x0000000000000000000000000000000000000000000000000000000000000000","0x0000000000000000"],"sha3Uncles":"0x0000000000000000000000000000000000000000000000000000000000000000","size":"0x21c","stateRoot":"0x0000000000000000000000000000000000000000000000000000000000000000","timestamp":"0x5f5fda18","totalDifficulty":null,"transactions":[],"transactionsRoot":"0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347","uncles":[]},"id":"1287"},{"jsonrpc":"2.0","result":{"author":"0x0000000000000000000000000000000000000000","difficulty":"0x0","extraData":"0x0000000000000000000000000000000000000000000000000000000000000000","gasLimit":"0x0","gasUsed":"0x0","hash":"0x79be1a3286ddfbdb4f28383a7896886a7b33c299b5056fd052c35c91398e4b66","logsBloom":"0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000","miner":"0x0000000000000000000000000000000000000000","number":"0xc9","parentHash":"0xda20cffe126b640a49e09b9d9b358075ef2ffc9e6db9162baaeb894b4423846d","receiptsRoot":"0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347","sealFields":["0x0000000000000000000000000000000000000000000000000000000000000000","0x0000000000000000"],"sha3Uncles":"0x0000000000000000000000000000000000000000000000000000000000000000","size":"0x21c","stateRoot":"0x0000000000000000000000000000000000000000000000000000000000000000","timestamp":"0x5f5fda1e","totalDifficulty":null,"transactions":[],"transactionsRoot":"0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347","uncles":[]},"id":"1288"},{"jsonrpc":"2.0","result":{"author":"0x0000000000000000000000000000000000000000","difficulty":"0x0","extraData":"0x0000000000000000000000000000000000000000000000000000000000000000","gasLimit":"0x0","gasUsed":"0x0","hash":"0x2b2e8034f694e2624d45c4882e27fe2efded6c529d83561084537e7352c4df94","logsBloom":"0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000","miner":"0x0000000000000000000000000000000000000000","number":"0xca","parentHash":"0x09c020652ccd5dfe11162315d97b39ce66b3edbc82a91ccf9bcb43a5c5baf53f","receiptsRoot":"0x78bc580ad1b04e7c8b46a72af423057742242269f03c8ee926314a4c01cdce9b","sealFields":["0x0000000000000000000000000000000000000000000000000000000000000000","0x0000000000000000"],"sha3Uncles":"0x0000000000000000000000000000000000000000000000000000000000000000","size":"0x542","stateRoot":"0x0000000000000000000000000000000000000000000000000000000000000000","timestamp":"0x5f5fda24","totalDifficulty":null,"transactions":[{"blockHash":"0x2b2e8034f694e2624d45c4882e27fe2efded6c529d83561084537e7352c4df94","blockNumber":"0xca","chainId":"0x2b","condition":null,"creates":"0x470dcb8e085736fa573aa2786b5ab2dadf86e3a2","from":"0x0848b04952763ff8ae726d4a910cb55243bf2427","gas":"0x33936","gasPrice":"0x0","hash":"0xf82a5a37cf7d8f7a03ea0fafe4c87f00a29ae6462bcb537cd713b4c225f46904","input":"0x608060405234801561001057600080fd5b506040516102b33803806102b38339818101604052602081101561003357600080fd5b81019080805190602001909291905050508060008190555033600160006101000a81548173ffffffffffffffffffffffffffffffffffffffff021916908373ffffffffffffffffffffffffffffffffffffffff160217905550506102178061009c6000396000f3fe608060405234801561001057600080fd5b506004361061004c5760003560e01c80637cf5dab0146100515780638381f58a1461007f5780638da5cb5b1461009d578063d826f88f146100d1575b600080fd5b61007d6004803603602081101561006757600080fd5b81019080803590602001909291905050506100db565b005b6100876100e9565b6040518082815260200191505060405180910390f35b6100a56100ef565b604051808273ffffffffffffffffffffffffffffffffffffffff16815260200191505060405180910390f35b6100d9610115565b005b806000540160008190555050565b60005481565b600160009054906101000a900473ffffffffffffffffffffffffffffffffffffffff1681565b600160009054906101000a900473ffffffffffffffffffffffffffffffffffffffff1673ffffffffffffffffffffffffffffffffffffffff163373ffffffffffffffffffffffffffffffffffffffff16146101d8576040517f08c379a00000000000000000000000000000000000000000000000000000000081526004018080602001828103825260098152602001807f4e6f74206f776e6572000000000000000000000000000000000000000000000081525060200191505060405180910390fd5b6000808190555056fea2646970667358221220d2798ebd8732b993fe81a2ab9660634455febeb37f5b43ca5206c5dadb14078264736f6c634300060c00330000000000000000000000000000000000000000000000000000000000000013","nonce":"0x4","publicKey":null,"r":"0x467156f0c71d7e4a4155b4a3a0aaaee09997fc409e9cec85f2e94d803f07e1db","raw":"0xf903210480830339368080b902d3608060405234801561001057600080fd5b506040516102b33803806102b38339818101604052602081101561003357600080fd5b81019080805190602001909291905050508060008190555033600160006101000a81548173ffffffffffffffffffffffffffffffffffffffff021916908373ffffffffffffffffffffffffffffffffffffffff160217905550506102178061009c6000396000f3fe608060405234801561001057600080fd5b506004361061004c5760003560e01c80637cf5dab0146100515780638381f58a1461007f5780638da5cb5b1461009d578063d826f88f146100d1575b600080fd5b61007d6004803603602081101561006757600080fd5b81019080803590602001909291905050506100db565b005b6100876100e9565b6040518082815260200191505060405180910390f35b6100a56100ef565b604051808273ffffffffffffffffffffffffffffffffffffffff16815260200191505060405180910390f35b6100d9610115565b005b806000540160008190555050565b60005481565b600160009054906101000a900473ffffffffffffffffffffffffffffffffffffffff1681565b600160009054906101000a900473ffffffffffffffffffffffffffffffffffffffff1673ffffffffffffffffffffffffffffffffffffffff163373ffffffffffffffffffffffffffffffffffffffff16146101d8576040517f08c379a00000000000000000000000000000000000000000000000000000000081526004018080602001828103825260098152602001807f4e6f74206f776e6572000000000000000000000000000000000000000000000081525060200191505060405180910390fd5b6000808190555056fea2646970667358221220d2798ebd8732b993fe81a2ab9660634455febeb37f5b43ca5206c5dadb14078264736f6c634300060c0033000000000000000000000000000000000000000000000000000000000000001379a0467156f0c71d7e4a4155b4a3a0aaaee09997fc409e9cec85f2e94d803f07e1dba00233db3b3f8d828f7c6fe266800283165a6c1eb3578fe16e868f79b0783e217f","s":"0x233db3b3f8d828f7c6fe266800283165a6c1eb3578fe16e868f79b0783e217f","standardV":"0x0","to":null,"transactionIndex":"0x0","v":"0x79","value":"0x0"}],"transactionsRoot":"0x948990068a977d4abfbc52277703bdc09b82eadd8fe892e105effda962be583f","uncles":[]},"id":"1289"},{"jsonrpc":"2.0","result":{"author":"0x0000000000000000000000000000000000000000","difficulty":"0x0","extraData":"0x0000000000000000000000000000000000000000000000000000000000000000","gasLimit":"0x0","gasUsed":"0x0","hash":"0x02ccf5ccfcf18bb08070eb673f4a56540d7e606a26c25a67e563e8db3855dae1","logsBloom":"0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000","miner":"0x0000000000000000000000000000000000000000","number":"0xcb","parentHash":"0xd7153299d3fb06fcb5dfbe2aa9468e38b02cbc2cd75b3b8f23849b80f5c4c628","receiptsRoot":"0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347","sealFields":["0x0000000000000000000000000000000000000000000000000000000000000000","0x0000000000000000"],"sha3Uncles":"0x0000000000000000000000000000000000000000000000000000000000000000","size":"0x21c","stateRoot":"0x0000000000000000000000000000000000000000000000000000000000000000","timestamp":"0x5f5fda2a","totalDifficulty":null,"transactions":[],"transactionsRoot":"0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347","uncles":[]},"id":"1290"},{"jsonrpc":"2.0","result":{"author":"0x0000000000000000000000000000000000000000","difficulty":"0x0","extraData":"0x0000000000000000000000000000000000000000000000000000000000000000","gasLimit":"0x0","gasUsed":"0x0","hash":"0x754bfb7290d20ce7edb964bbf4eb492c506c8a945d7243429f5c1694f3b3e119","logsBloom":"0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000","miner":"0x0000000000000000000000000000000000000000","number":"0xcc","parentHash":"0xfba7a1c0f14950b285cdb6a93046f5389bc8319ebafb028e42ffb2ddb58eed01","receiptsRoot":"0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347","sealFields":["0x0000000000000000000000000000000000000000000000000000000000000000","0x0000000000000000"],"sha3Uncles":"0x0000000000000000000000000000000000000000000000000000000000000000","size":"0x21c","stateRoot":"0x0000000000000000000000000000000000000000000000000000000000000000","timestamp":"0x5f5fda30","totalDifficulty":null,"transactions":[],"transactionsRoot":"0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347","uncles":[]},"id":"1291"},{"jsonrpc":"2.0","result":{"author":"0x0000000000000000000000000000000000000000","difficulty":"0x0","extraData":"0x0000000000000000000000000000000000000000000000000000000000000000","gasLimit":"0x0","gasUsed":"0x0","hash":"0x1e84959bc11598c8b1da892a02006c7a8dbf85a4232bb32d635849172b47e9ee","logsBloom":"0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000","miner":"0x0000000000000000000000000000000000000000","number":"0xcd","parentHash":"0xf77eb5abecbb3926821d06f5e6b18a386584e99d461ed42e69fcb14dee11c444","receiptsRoot":"0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347","sealFields":["0x0000000000000000000000000000000000000000000000000000000000000000","0x0000000000000000"],"sha3Uncles":"0x0000000000000000000000000000000000000000000000000000000000000000","size":"0x21c","stateRoot":"0x0000000000000000000000000000000000000000000000000000000000000000","timestamp":"0x5f5fda36","totalDifficulty":null,"transactions":[],"transactionsRoot":"0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347","uncles":[]},"id":"1292"},{"jsonrpc":"2.0","result":{"author":"0x0000000000000000000000000000000000000000","difficulty":"0x0","extraData":"0x0000000000000000000000000000000000000000000000000000000000000000","gasLimit":"0x0","gasUsed":"0x0","hash":"0xdc970c29105989e2e3b1982c46d76aa4b09785d7788c2d5086d31e1f2ef4a476","logsBloom":"0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000","miner":"0x0000000000000000000000000000000000000000","number":"0xce","parentHash":"0xb28fb8b617e434f70c732fecc9f30e78698da8b0b05bf00a6a06f3b75f2105b3","receiptsRoot":"0x78bc580ad1b04e7c8b46a72af423057742242269f03c8ee926314a4c01cdce9b","sealFields":["0x0000000000000000000000000000000000000000000000000000000000000000","0x0000000000000000"],"sha3Uncles":"0x0000000000000000000000000000000000000000000000000000000000000000","size":"0x2a2","stateRoot":"0x0000000000000000000000000000000000000000000000000000000000000000","timestamp":"0x5f5fda3c","totalDifficulty":null,"transactions":[{"blockHash":"0xdc970c29105989e2e3b1982c46d76aa4b09785d7788c2d5086d31e1f2ef4a476","blockNumber":"0xce","chainId":"0x2b","condition":null,"creates":null,"from":"0x0848b04952763ff8ae726d4a910cb55243bf2427","gas":"0x6a5e","gasPrice":"0x0","hash":"0xc6acc59804b4d413ab9eba73cab544f653570dd599a8dd6eb06d2b8202c2957b","input":"0x7cf5dab00000000000000000000000000000000000000000000000000000000000000005","nonce":"0x5","publicKey":null,"r":"0x8fe885136c61b0a7fa21b1c1d2f7f45518f99249f4a13e51a2f6861f3988dee0","raw":"0xf8830580826a5e94470dcb8e085736fa573aa2786b5ab2dadf86e3a280a47cf5dab000000000000000000000000000000000000000000000000000000000000000057aa08fe885136c61b0a7fa21b1c1d2f7f45518f99249f4a13e51a2f6861f3988dee0a06b1d4086dc85816eea52d44748d91abf04b73d72f7b37c2a21376402f0dada72","s":"0x6b1d4086dc85816eea52d44748d91abf04b73d72f7b37c2a21376402f0dada72","standardV":"0x1","to":"0x470dcb8e085736fa573aa2786b5ab2dadf86e3a2","transactionIndex":"0x0","v":"0x7a","value":"0x0"}],"transactionsRoot":"0xa71601b766401b06456fefda9f04c0e9c13df2ed847232f9da14800ec8307cf6","uncles":[]},"id":"1293"},{"jsonrpc":"2.0","result":{"author":"0x0000000000000000000000000000000000000000","difficulty":"0x0","extraData":"0x0000000000000000000000000000000000000000000000000000000000000000","gasLimit":"0x0","gasUsed":"0x0","hash":"0x262bf1d92da09509001073d4e1df28117dc5bf378bf50f7f92b268ee7ee1705e","logsBloom":"0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000","miner":"0x0000000000000000000000000000000000000000","number":"0xcf","parentHash":"0x6b5ef00904bc6a758b004fa3d20ad58ccee41a785823ce709cc42140441021e2","receiptsRoot":"0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347","sealFields":["0x0000000000000000000000000000000000000000000000000000000000000000","0x0000000000000000"],"sha3Uncles":"0x0000000000000000000000000000000000000000000000000000000000000000","size":"0x21c","stateRoot":"0x0000000000000000000000000000000000000000000000000000000000000000","timestamp":"0x5f5fda42","totalDifficulty":null,"transactions":[],"transactionsRoot":"0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347","uncles":[]},"id":"1294"},{"jsonrpc":"2.0","result":{"author":"0x0000000000000000000000000000000000000000","difficulty":"0x0","extraData":"0x0000000000000000000000000000000000000000000000000000000000000000","gasLimit":"0x0","gasUsed":"0x0","hash":"0xb93663f2d357f56867f81fec1b9e71ed73a158f7642f12d4aaa48826acd32cb6","logsBloom":"0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000","miner":"0x0000000000000000000000000000000000000000","number":"0xd0","parentHash":"0xc47c226ded8f715edc62d36e48df73ce34584c6e684422ca2a9896b0e738ff7e","receiptsRoot":"0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347","sealFields":["0x0000000000000000000000000000000000000000000000000000000000000000","0x0000000000000000"],"sha3Uncles":"0x0000000000000000000000000000000000000000000000000000000000000000","size":"0x21c","stateRoot":"0x0000000000000000000000000000000000000000000000000000000000000000","timestamp":"0x5f5fda48","totalDifficulty":null,"transactions":[],"transactionsRoot":"0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347","uncles":[]},"id":"1295"},{"jsonrpc":"2.0","result":{"author":"0x0000000000000000000000000000000000000000","difficulty":"0x0","extraData":"0x0000000000000000000000000000000000000000000000000000000000000000","gasLimit":"0x0","gasUsed":"0x0","hash":"0x67f77bc0caed18b13641bfc078d3cfaf8ce362828b0f91b621a69c09196b092e","logsBloom":"0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000","miner":"0x0000000000000000000000000000000000000000","number":"0xd1","parentHash":"0x7dc23e20684ba6e1a14d46fad68e7d0614b0abcadddc35e5c6c44b153f24ab7e","receiptsRoot":"0x78bc580ad1b04e7c8b46a72af423057742242269f03c8ee926314a4c01cdce9b","sealFields":["0x0000000000000000000000000000000000000000000000000000000000000000","0x0000000000000000"],"sha3Uncles":"0x0000000000000000000000000000000000000000000000000000000000000000","size":"0x282","stateRoot":"0x0000000000000000000000000000000000000000000000000000000000000000","timestamp":"0x5f5fda54","totalDifficulty":null,"transactions":[{"blockHash":"0x67f77bc0caed18b13641bfc078d3cfaf8ce362828b0f91b621a69c09196b092e","blockNumber":"0xd1","chainId":"0x2b","condition":null,"creates":null,"from":"0x0848b04952763ff8ae726d4a910cb55243bf2427","gas":"0x56db","gasPrice":"0x0","hash":"0xd53de4127a7dfac902bb5ad02529c5835394e96697cebfe20bb187defa7b9973","input":"0xd826f88f","nonce":"0x6","publicKey":null,"r":"0xc557aa593f71f987aabb515443947d3d82b12325f36f60f759dcd48f5afe87bc","raw":"0xf86306808256db94470dcb8e085736fa573aa2786b5ab2dadf86e3a28084d826f88f7aa0c557aa593f71f987aabb515443947d3d82b12325f36f60f759dcd48f5afe87bca07cbb1d36d64e4b2a8c158318a1e621d88178190e288aa0a6eba4e1a0949eb949","s":"0x7cbb1d36d64e4b2a8c158318a1e621d88178190e288aa0a6eba4e1a0949eb949","standardV":"0x1","to":"0x470dcb8e085736fa573aa2786b5ab2dadf86e3a2","transactionIndex":"0x0","v":"0x7a","value":"0x0"}],"transactionsRoot":"0x59bdd2da28d3d0901c4b3447d397d35df8b800f5ae4ed51a1e823a8fb2a273ad","uncles":[]},"id":"1296"},{"jsonrpc":"2.0","result":{"author":"0x0000000000000000000000000000000000000000","difficulty":"0x0","extraData":"0x0000000000000000000000000000000000000000000000000000000000000000","gasLimit":"0x0","gasUsed":"0x0","hash":"0x0fc9581e51dad2f68accde9bd66e512c88404f6fc4bab1ca49ff1b659e03e85c","logsBloom":"0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000","miner":"0x0000000000000000000000000000000000000000","number":"0xd2","parentHash":"0xfed3f148617bbe93f5ab03be002297d5f51d245358bb2b421fd086eb5b7882fd","receiptsRoot":"0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347","sealFields":["0x0000000000000000000000000000000000000000000000000000000000000000","0x0000000000000000"],"sha3Uncles":"0x0000000000000000000000000000000000000000000000000000000000000000","size":"0x21c","stateRoot":"0x0000000000000000000000000000000000000000000000000000000000000000","timestamp":"0x5f5fda5a","totalDifficulty":null,"transactions":[],"transactionsRoot":"0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347","uncles":[]},"id":"1297"},{"jsonrpc":"2.0","result":{"author":"0x0000000000000000000000000000000000000000","difficulty":"0x0","extraData":"0x0000000000000000000000000000000000000000000000000000000000000000","gasLimit":"0x0","gasUsed":"0x0","hash":"0x9426ac4e8a00120a848558bc8b02327ee8cc17ffe349ca7861fc8a42d974c7a6","logsBloom":"0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000","miner":"0x0000000000000000000000000000000000000000","number":"0xd3","parentHash":"0xda153d4a0c771bfb855ab403cf4d11174712031e319d23f8800c1dc31ff80e58","receiptsRoot":"0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347","sealFields":["0x0000000000000000000000000000000000000000000000000000000000000000","0x0000000000000000"],"sha3Uncles":"0x0000000000000000000000000000000000000000000000000000000000000000","size":"0x21c","stateRoot":"0x0000000000000000000000000000000000000000000000000000000000000000","timestamp":"0x5f5fda60","totalDifficulty":null,"transactions":[],"transactionsRoot":"0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347","uncles":[]},"id":"1298"},{"jsonrpc":"2.0","result":{"author":"0x0000000000000000000000000000000000000000","difficulty":"0x0","extraData":"0x0000000000000000000000000000000000000000000000000000000000000000","gasLimit":"0x0","gasUsed":"0x0","hash":"0x74119506ef0ca6094fc43e7978501c460c2a6d4ef68e10654255c861e3437223","logsBloom":"0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000","miner":"0x0000000000000000000000000000000000000000","number":"0xd4","parentHash":"0x2140666b6e2c55cf03c367e59ddda966f731bdbe57a4b73dfec9d67f11512d21","receiptsRoot":"0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347","sealFields":["0x0000000000000000000000000000000000000000000000000000000000000000","0x0000000000000000"],"sha3Uncles":"0x0000000000000000000000000000000000000000000000000000000000000000","size":"0x21c","stateRoot":"0x0000000000000000000000000000000000000000000000000000000000000000","timestamp":"0x5f5fda66","totalDifficulty":null,"transactions":[],"transactionsRoot":"0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347","uncles":[]},"id":"1299"},{"jsonrpc":"2.0","result":{"author":"0x0000000000000000000000000000000000000000","difficulty":"0x0","extraData":"0x0000000000000000000000000000000000000000000000000000000000000000","gasLimit":"0x0","gasUsed":"0x0","hash":"0xdb0adc1803c891af78da06558a7430d288dc7406744b6b3667cd55e2c1d01944","logsBloom":"0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000","miner":"0x0000000000000000000000000000000000000000","number":"0xd5","parentHash":"0x71ee2718979b962b85414bb3992018b2fde0378adb823c51162140a122ba0fed","receiptsRoot":"0x78bc580ad1b04e7c8b46a72af423057742242269f03c8ee926314a4c01cdce9b","sealFields":["0x0000000000000000000000000000000000000000000000000000000000000000","0x0000000000000000"],"sha3Uncles":"0x0000000000000000000000000000000000000000000000000000000000000000","size":"0x283","stateRoot":"0x0000000000000000000000000000000000000000000000000000000000000000","timestamp":"0x5f5fda6c","totalDifficulty":null,"transactions":[{"blockHash":"0xdb0adc1803c891af78da06558a7430d288dc7406744b6b3667cd55e2c1d01944","blockNumber":"0xd5","chainId":"0x2b","condition":null,"creates":null,"from":"0x0848b04952763ff8ae726d4a910cb55243bf2427","gas":"0x3648e","gasPrice":"0x0","hash":"0x7eb77234dea62137709ad42e54cc339b8208c1ee868e1bc1489f36b2eff14e87","input":"0xd826f88f","nonce":"0x7","publicKey":null,"r":"0xc2a040e8283e503c214b9b6cd79462778483f528c0699f77cadd6138dd25729","raw":"0xf86407808303648e94470dcb8e085736fa573aa2786b5ab2dadf86e3a28084d826f88f79a00c2a040e8283e503c214b9b6cd79462778483f528c0699f77cadd6138dd25729a0524db43fdd900bdd8b6682af2759badf339bd96113373fa92ca85d6ded4045f3","s":"0x524db43fdd900bdd8b6682af2759badf339bd96113373fa92ca85d6ded4045f3","standardV":"0x0","to":"0x470dcb8e085736fa573aa2786b5ab2dadf86e3a2","transactionIndex":"0x0","v":"0x79","value":"0x0"}],"transactionsRoot":"0x2b1616c8a43fc3b5d0295f392bccf6104745040a9537d7a8d73c3b4542e5875c","uncles":[]},"id":"1300"}].
Sep 14, 2020 @ 17:02:48.347	2020-09-14 21:02:48.347 tokio-runtime-worker WARN sc_service::client::client  Block prepare storage changes error:
Sep 14, 2020 @ 17:02:48.347	2020-09-14 21:02:48.347 tokio-runtime-worker ERROR runtime  panicked at 'Storage root must match that calculated.', /home/alan/.cargo/git/checkouts/substrate-7e08433d4c370a21/f397f24/frame/executive/src/lib.rs:398:9

It has this UnknownParent that seems to be the trigger in that case.

[bkchr]

Hmm, an UnknownParent will not lead to a storage root mismatch.

Would be really nice to have something to reproduce this :)

[crystalin]

Updating the ticket based on our discuss.
Flow description from Basti:

The flow is the following:

1. A new round is started in the moment a collator imports a new best relay chain block.
2. Each collator builds a new block.
3. The block is "converted" to a PoVBlock that is send to the relay chain.
4. Each PovBlock that is validated by a parachain validator results in a "candidate message". This message gets also back to the collator.
5. The collator takes this message and appends it to a block announcement message and starts gossiping this block announcement message to all its peers.
6. Other collators and full nodes will start to import these blocks (while importing we execute the transactions)
7. Each imported block is put into the db, but we don't tag any of them directly as "best" block.
8. A new relay chain block arrives (that also means a new round starts) and we extract the parachain block header that was included in this relay block.
9. We tag the block that belongs to the parachain block header as new best block.
   Step 9. does not involves any further transaction execution or similar, it basically sets some value in the db.

As requested I attached the logs including the TRACE state.
rpc_node_1_fail.log
rpc_node_2.log

The first node has a different value set for state:
26aa394eea5630e07c48ae0c9558cef7b99d880ec681799c0cf30e8886371da9fe10649cb7f172bb3a22f04390ca3399000ed8be9804204d3953ed753e729c641935aad68e6e1bc6ee6a93af2ab50348

The left side (node 1 failing) has failed import because of re-org. But the block crashing is the parachain block #84

[crystalin]

I initially wanted to explain how to reproduce here but it is too long and complex, so I made a branch for it:
https://github.com/PureStake/moonbeam-parachain/tree/parachain-state-bug

The readme contains all the steps

[crystalin]

@bkchr If that helps, I can try to keep one network online for you to connect and see it. It will make the reproduction steps a bit simpler.

[crystalin]

With the suggestion of @JoshOrndorff , I tried to reproduce it only using substrate transactions (instead of the EVM) and so far I didn't get the issue, but I observed something that could be similar to what is happening.

When running 1 parachain collator and 3 (non validating) parachain rpc node, those rpc nodes sometime stop importing blocks (after a re-org also) for a long time and finally re-sync.

2020-10-06 02:33:53.386 tokio-runtime-worker INFO substrate  💤 [Relaychain] Idle (3 peers), best: #87 (0xdd6f…adaf), finalized #0 (0xe2b2…5f2e), ⬇ 0 ⬆ 0
2020-10-06 02:33:54.126 tokio-runtime-worker INFO substrate  ✨ [Relaychain] Imported #88 (0x412e…1dd6)
2020-10-06 02:33:57.499 tokio-runtime-worker INFO substrate  💤 [Parachain] Idle (3 peers), best: #40 (0x2a4c…644f), finalized #0 (0xd209…c8a1), ⬇ 0.1kiB/s ⬆ 0.1kiB/s
2020-10-06 02:33:58.386 tokio-runtime-worker INFO substrate  💤 [Relaychain] Idle (3 peers), best: #88 (0x412e…1dd6), finalized #0 (0xe2b2…5f2e), ⬇ 0.7kiB/s ⬆ 0.6kiB/s
2020-10-06 02:34:00.050 tokio-runtime-worker INFO substrate  ✨ [Relaychain] Imported #89 (0x2f29…480f)
2020-10-06 02:34:02.499 tokio-runtime-worker INFO substrate  💤 [Parachain] Idle (3 peers), best: #40 (0x2a4c…644f), finalized #0 (0xd209…c8a1), ⬇ 70 B/s ⬆ 72 B/s
2020-10-06 02:34:03.387 tokio-runtime-worker INFO substrate  💤 [Relaychain] Idle (3 peers), best: #89 (0x2f29…480f), finalized #0 (0xe2b2…5f2e), ⬇ 0.4kiB/s ⬆ 0.3kiB/s
2020-10-06 02:34:06.089 tokio-runtime-worker INFO substrate  ✨ [Relaychain] Imported #90 (0xfd39…c0af)
2020-10-06 02:34:06.124 tokio-runtime-worker INFO sc_informant  ♻️  [Relaychain] Reorg on #90,0xfd39…c0af to #90,0xb514…9ff1, common ancestor #89,0x2f29…480f
2020-10-06 02:34:06.124 tokio-runtime-worker INFO substrate  ✨ [Relaychain] Imported #90 (0xb514…9ff1)
2020-10-06 02:34:06.179 tokio-runtime-worker INFO substrate  ✨ [Parachain] Imported #41 (0x4bcc…4dc7)
2020-10-06 02:34:06.188 tokio-runtime-worker INFO substrate  ✨ [Parachain] Imported #42 (0x39c2…d297)
2020-10-06 02:34:06.197 tokio-runtime-worker INFO substrate  ✨ [Parachain] Imported #43 (0x3a31…9503)
2020-10-06 02:34:06.206 tokio-runtime-worker INFO substrate  ✨ [Parachain] Imported #44 (0xe7cf…d96e)
2020-10-06 02:34:06.215 tokio-runtime-worker INFO substrate  ✨ [Parachain] Imported #45 (0x74c8…154d)
2020-10-06 02:34:06.224 tokio-runtime-worker INFO substrate  ✨ [Parachain] Imported #46 (0x5be7…b2f3)
2020-10-06 02:34:06.233 tokio-runtime-worker INFO substrate  ✨ [Parachain] Imported #47 (0x8699…08fc)
2020-10-06 02:34:06.242 tokio-runtime-worker INFO substrate  ✨ [Parachain] Imported #48 (0x7f51…7942)
2020-10-06 02:34:06.251 tokio-runtime-worker INFO substrate  ✨ [Parachain] Imported #49 (0x72c1…9a44)
2020-10-06 02:34:06.260 tokio-runtime-worker INFO substrate  ✨ [Parachain] Imported #50 (0x965f…00bf)
2020-10-06 02:34:06.269 tokio-runtime-worker INFO substrate  ✨ [Parachain] Imported #51 (0x45fc…e17c)
2020-10-06 02:34:06.278 tokio-runtime-worker INFO substrate  ✨ [Parachain] Imported #52 (0x911c…517a)
2020-10-06 02:34:06.287 tokio-runtime-worker INFO substrate  ✨ [Parachain] Imported #53 (0xa6ac…26f0)
2020-10-06 02:34:06.295 tokio-runtime-worker INFO substrate  ✨ [Parachain] Imported #54 (0xd695…9e96)
2020-10-06 02:34:06.304 tokio-runtime-worker INFO substrate  ✨ [Parachain] Imported #55 (0x566a…0f2b)
2020-10-06 02:34:06.313 tokio-runtime-worker INFO substrate  ✨ [Parachain] Imported #56 (0x04fd…ff5b)
2020-10-06 02:34:06.321 tokio-runtime-worker INFO substrate  ✨ [Parachain] Imported #57 (0x8eb8…2ba7)
2020-10-06 02:34:06.330 tokio-runtime-worker INFO substrate  ✨ [Parachain] Imported #58 (0x987f…c1e2)
2020-10-06 02:34:06.338 tokio-runtime-worker INFO substrate  ✨ [Parachain] Imported #59 (0x830a…a015)
2020-10-06 02:34:06.347 tokio-runtime-worker INFO substrate  ✨ [Parachain] Imported #60 (0x0731…3a91)
2020-10-06 02:34:06.356 tokio-runtime-worker INFO substrate  ✨ [Parachain] Imported #61 (0xbabd…faa6)
2020-10-06 02:34:06.364 tokio-runtime-worker INFO substrate  ✨ [Parachain] Imported #62 (0x0021…b335)
2020-10-06 02:34:06.373 tokio-runtime-worker INFO substrate  ✨ [Parachain] Imported #63 (0x89f0…1def)
2020-10-06 02:34:06.381 tokio-runtime-worker INFO substrate  ✨ [Parachain] Imported #64 (0xf114…1d2a)
2020-10-06 02:34:06.390 tokio-runtime-worker INFO substrate  ✨ [Parachain] Imported #65 (0x6f4b…d0ab)
2020-10-06 02:34:06.398 tokio-runtime-worker INFO substrate  ✨ [Parachain] Imported #66 (0xaeae…1b8f)
2020-10-06 02:34:06.407 tokio-runtime-worker INFO substrate  ✨ [Parachain] Imported #67 (0x7894…190f)
2020-10-06 02:34:06.415 tokio-runtime-worker INFO substrate  ✨ [Parachain] Imported #68 (0x98b9…7a61)
2020-10-06 02:34:07.499 tokio-runtime-worker INFO substrate  💤 [Parachain] Idle (3 peers), best: #40 (0x2a4c…644f), finalized #0 (0xd209…c8a1), ⬇ 2.1kiB/s ⬆ 39 B/s
2020-10-06 02:34:08.387 tokio-runtime-worker INFO substrate  💤 [Relaychain] Idle (3 peers), best: #90 (0xb514…9ff1), finalized #0 (0xe2b2…5f2e), ⬇ 0.8kiB/s ⬆ 0.6kiB/s
2020-10-06 02:34:12.038 tokio-runtime-worker INFO substrate  ✨ [Relaychain] Imported #91 (0x6505…fc02)
2020-10-06 02:34:12.500 tokio-runtime-worker INFO substrate  💤 [Parachain] Idle (3 peers), best: #68 (0x98b9…7a61), finalized #0 (0xd209…c8a1), ⬇ 0.2kiB/s ⬆ 0.2kiB/s
2020-10-06 02:34:13.387 tokio-runtime-worker INFO substrate  💤 [Relaychain] Idle (3 peers), best: #91 (0x6505…fc02), finalized #0 (0xe2b2…5f2e), ⬇ 0.6kiB/s ⬆ 0.7kiB/s

@bkchr Could it be that it is the same issue, but the difference is that substrate when failing to import the block doesn't complain but in the case of the EVM it does because it panics if the state is invalid ?

[JoshOrndorff]

The output posted in the comment immediately before this one does not look to me like the parachain stopped importing blocks. It looks like a nice demonstration of the steps that @bkchr outlined earlier.

The first thing we see is a normally operating collator. Then there is a relay chain re-org. A re-org is one way that a collator could "import a new best relay chain block", which is exactly what starts the process listed. After the reorg, the collator suddenly imports a bunch of blocks all at once (but not not mark them as best). To me this indicates that the old relay fork did not reference these blocks, but the new relay fork does, so the collator imports them.

Then after another 1 second of waiting we get another new relay chain block (#40) which references all those just-imported parablocks, and thus the best parachain block is updated.

So my point is that the ouput in the immediately preceeding comment looks like correct expected behavior to me assuming that only the relay block #39 that we re-orged to knew about all those parablocks.

[JoshOrndorff]

I guess the one mystery that remains is why did the collator not import all of those parablocks before the reorg?

[JoshOrndorff]

This issue does seem to be related to re-orgs on the relay chain. Has paritytech/substrate#7118 made its way back into polkadot yet?

[bkchr]

The linked pr was only required for the transaction pool and nothing else. So, I don't assume that it would change something here.

[JoshOrndorff]

Just to keep everyone up to date I want to share an observation and a hypothesis.

hypothesis The crash reported here is a symptom of two interacting issues. The first is that weird stuff (TM) happens on the parachain after a re-org on the relay chain. The second is probably something in moonbeam or frontier. In order to eliminate this from the equation I'll try to reproduce the relevant part with the parachain template.

observation A lot more weird / bad / unstable stuff like nodes getting stuck, and sudden mass imports is much more visible when running without grandpa.

I'm able to reproduce paranodes getting stuck quite reliably by running the relay chain nodes with --no-grandpa.

Here is one full set of logs I generated and found pretty helpful. I'm open to hearing what additional logs I should be recording, if any. https://gist.github.com/JoshOrndorff/fb8230fd059449395730814098d60051

[JoshOrndorff]

Here's another complete set of logs from Alice validator, Bob validator, and three moonbeam collators. https://gist.github.com/JoshOrndorff/57842a1da7ea464a235afaa5499a721e

All the collators demonstrate getting stuck after a relay chain organization at various points.

[JoshOrndorff]

As I mentioned in #225 this seems to be fixed on newer cumulus, so this issue can be closed.

[bkchr]

Ty for reporting