assertion failed: key > self.key

[Manalope]

Describe the bug

Using reth 0.1.0-alpha.4 (0ef68b7) on Ubuntu 20.04 LTS, after syncing for ~2 days I now get an error: "assertion failed: key > self.key" immediately upon starting reth from the command line.

Steps to reproduce

Steps to reproduce:

1. cargo install --locked --path bin/reth --bin reth
2. RUST_LOG=info RUST_BACKTRACE=1 reth node --http.api all --ws.api all

Node logs

root@reth:~/reth# RUST_LOG=info RUST_BACKTRACE=1 reth node --http.api all --ws.api all
2023-07-22T15:33:22.139508Z  INFO reth::cli: reth 0.1.0-alpha.4 (0ef68b7d) starting
2023-07-22T15:33:22.139792Z  INFO reth::cli: Configuration loaded path="/root/.local/share/reth/mainnet/reth.toml"
2023-07-22T15:33:22.139809Z  INFO reth::cli: Opening database path="/root/.local/share/reth/mainnet/db"
2023-07-22T15:33:22.147788Z  INFO reth::cli: Database opened
2023-07-22T15:33:22.147842Z  INFO reth::cli: Pre-merge hard forks (block based):
- Frontier                         @0
- Homestead                        @1150000
- Dao                              @1920000
- Tangerine                        @2463000
- SpuriousDragon                   @2675000
- Byzantium                        @4370000
- Constantinople                   @7280000
- Petersburg                       @7280000
- Istanbul                         @9069000
- MuirGlacier                      @9200000
- Berlin                           @12244000
- London                           @12965000
- ArrowGlacier                     @13773000
- GrayGlacier                      @15050000
Merge hard forks:
- Paris                            @58750000000000000000000 (network is not known to be merged)

Post-merge hard forks (timestamp based):
- Shanghai                         @1681338455

2023-07-22T15:33:22.148109Z  INFO reth::cli: Transaction pool initialized
2023-07-22T15:33:22.148127Z  INFO reth::cli: Connecting to P2P network
2023-07-22T15:33:22.148220Z  INFO net::peers: Loading saved peers file=/root/.local/share/reth/mainnet/known-peers.json
2023-07-22T15:33:22.260003Z  INFO reth::cli: Connected to P2P network peer_id=0xca51…a516 local_addr=0.0.0.0:30303
2023-07-22T15:33:22.260149Z  WARN consensus::engine: Pipeline sync progress is inconsistent first_stage_checkpoint=17721430 inconsistent_stage_id=MerkleExecute inconsistent_stage_checkpoint=0
2023-07-22T15:33:22.260162Z  INFO reth::cli: Consensus engine initialized
2023-07-22T15:33:22.260169Z  INFO reth::cli: Engine API handler initialized
2023-07-22T15:33:22.260325Z  INFO reth::cli: RPC auth server started url=127.0.0.1:8551
2023-07-22T15:33:22.260340Z  INFO reth::cli: RPC IPC server started url=/tmp/reth.ipc
2023-07-22T15:33:22.260346Z  INFO reth::cli: Starting consensus engine
2023-07-22T15:33:22.260652Z  INFO reth::node::events: Executing stage pipeline_stages=1/13 stage=Headers from=17721430 checkpoint=100.0% eta=unknown
2023-07-22T15:33:22.260669Z  INFO reth::cli: Status connected_peers=0 stage=Headers checkpoint=100.0% eta=unknown
2023-07-22T15:33:22.260669Z  INFO execute{stage=Headers}: sync::stages::headers: Target block already reached checkpoint=100.0% target=Hash(0xda1578b300bd2b572e6d281fde6e81a41a774b7eb325a8ba6640166d8cd9600a)
2023-07-22T15:33:22.260702Z  INFO reth::node::events: Stage finished executing pipeline_stages=1/13 stage=Headers block=17721430 checkpoint=100.0% eta=unknown
2023-07-22T15:33:22.260739Z  INFO reth::node::events: Executing stage pipeline_stages=2/13 stage=TotalDifficulty from=17721430 checkpoint=100.0% eta=unknown
2023-07-22T15:33:22.260743Z  INFO reth::node::events: Stage finished executing pipeline_stages=2/13 stage=TotalDifficulty block=17721430 checkpoint=100.0% eta=unknown
2023-07-22T15:33:22.260767Z  INFO reth::node::events: Executing stage pipeline_stages=3/13 stage=Bodies from=17721430 checkpoint=100.0% eta=unknown
2023-07-22T15:33:22.260774Z  INFO reth::node::events: Stage finished executing pipeline_stages=3/13 stage=Bodies block=17721430 checkpoint=100.0% eta=unknown
2023-07-22T15:33:22.260799Z  INFO reth::node::events: Executing stage pipeline_stages=4/13 stage=SenderRecovery from=17721430 checkpoint=100.0% eta=unknown
2023-07-22T15:33:22.260803Z  INFO reth::node::events: Stage finished executing pipeline_stages=4/13 stage=SenderRecovery block=17721430 checkpoint=100.0% eta=unknown
2023-07-22T15:33:22.260821Z  INFO reth::node::events: Executing stage pipeline_stages=5/13 stage=Execution from=17721430 checkpoint=100.0% eta=unknown
2023-07-22T15:33:22.260893Z  INFO reth::node::events: Stage finished executing pipeline_stages=5/13 stage=Execution block=17721430 checkpoint=100.0% eta=unknown
2023-07-22T15:33:22.260908Z  INFO execute{stage=MerkleUnwind}: sync::stages::merkle::unwind: Stage is always skipped
2023-07-22T15:33:22.260917Z  INFO reth::node::events: Executing stage pipeline_stages=6/13 stage=MerkleUnwind from=17721430 checkpoint=17721430 eta=unknown
2023-07-22T15:33:22.260927Z  INFO reth::node::events: Stage finished executing pipeline_stages=6/13 stage=MerkleUnwind block=17721430 checkpoint=17721430 eta=unknown
2023-07-22T15:33:22.260955Z  INFO reth::node::events: Executing stage pipeline_stages=7/13 stage=AccountHashing from=17721430 checkpoint=100.0% eta=unknown
2023-07-22T15:33:22.260962Z  INFO reth::node::events: Stage finished executing pipeline_stages=7/13 stage=AccountHashing block=17721430 checkpoint=100.0% eta=unknown
2023-07-22T15:33:22.260978Z  INFO reth::node::events: Executing stage pipeline_stages=8/13 stage=StorageHashing from=17721430 checkpoint=100.0% eta=unknown
2023-07-22T15:33:22.260983Z  INFO reth::node::events: Stage finished executing pipeline_stages=8/13 stage=StorageHashing block=17721430 checkpoint=100.0% eta=unknown
2023-07-22T15:33:22.261004Z  INFO reth::node::events: Executing stage pipeline_stages=9/13 stage=MerkleExecute from=0 checkpoint=35.5% eta=unknown
thread 'tokio-runtime-worker' panicked at 'assertion failed: key > self.key', crates/primitives/src/trie/hash_builder/mod.rs:125:9
stack backtrace:
   0: rust_begin_unwind
             at /rustc/8ede3aae28fe6e4d52b38157d7bfe0d3bceef225/library/std/src/panicking.rs:593:5
   1: core::panicking::panic_fmt
             at /rustc/8ede3aae28fe6e4d52b38157d7bfe0d3bceef225/library/core/src/panicking.rs:67:14
   2: core::panicking::panic
             at /rustc/8ede3aae28fe6e4d52b38157d7bfe0d3bceef225/library/core/src/panicking.rs:117:5
   3: reth_primitives::trie::hash_builder::HashBuilder::add_leaf
   4: reth_trie::trie::StateRoot<TX,H>::calculate
   5: <reth_stages::stages::merkle::MerkleStage as reth_stages::stage::Stage<DB>>::execute::{{closure}}
   6: reth_stages::pipeline::Pipeline<DB>::execute_stage_to_completion::{{closure}}
   7: <tracing::instrument::Instrumented<T> as core::future::future::Future>::poll
   8: reth_stages::pipeline::Pipeline<DB>::run_loop::{{closure}}
   9: reth_stages::pipeline::Pipeline<DB>::run_as_fut::{{closure}}
  10: reth_beacon_consensus::engine::sync::EngineSyncController<DB,Client>::try_spawn_pipeline::{{closure}}
  11: <futures_util::future::future::map::Map<Fut,F> as core::future::future::Future>::poll
  12: <tracing_futures::Instrumented<T> as core::future::future::Future>::poll
  13: <futures_util::future::select::Select<A,B> as core::future::future::Future>::poll
  14: tokio::runtime::park::CachedParkThread::block_on
  15: tokio::runtime::handle::Handle::block_on
  16: <tokio::runtime::blocking::task::BlockingTask<T> as core::future::future::Future>::poll
  17: tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut
  18: tokio::runtime::task::core::Core<T,S>::poll
  19: tokio::runtime::task::harness::Harness<T,S>::poll
  20: tokio::runtime::blocking::pool::Inner::run
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
2023-07-22T15:33:22.337775Z ERROR reth_tasks: Critical task `pipeline task` panicked: `assertion failed: key > self.key`
Error: Critical task `pipeline task` panicked: `assertion failed: key > self.key`

Location:
    /rustc/8ede3aae28fe6e4d52b38157d7bfe0d3bceef225/library/core/src/convert/mod.rs:717:9
root@reth:~/reth#

Platform(s)

Linux (x86)

What version/commit are you on?

0.1.0-alpha.4 (0ef68b7)

What database version are you on?

Current database version: 1
Local database version: 1

If you've built Reth from source, provide the full command you used

No response

Code of Conduct

• I agree to follow the Code of Conduct

[github-actions]

This issue is stale because it has been open for 14 days with no activity.

[github-actions]

This issue was closed because it has been inactive for 7 days since being marked as stale.

[mjm114514]

Facing the same issue.