UnexpectedStaticFileBlockNumber Error Using Engine2

[BowTiedDevil]

Describe the bug

Running reth v1.0.5 with the --engine.experimental flag results in a Fatal error shutdown after some time.

Steps to reproduce

Start reth v1.0.5 against a previously-synced v1.0.4 database with the --engine.experimental flag.

Node logs

2024-08-12T16:17:04.481793Z  INFO Initialized tracing, debug log directory: /root/.cache/reth/logs/mainnet
2024-08-12T16:17:04.483818Z  INFO Starting reth version="1.0.5 (603e39ab)"
2024-08-12T16:17:04.484556Z  INFO Opening database path="/data/db"
2024-08-12T16:17:04.490489Z  INFO Configuration loaded path="/data/reth.toml"
2024-08-12T16:17:04.493603Z  INFO Verifying storage consistency.
2024-08-12T16:17:04.508503Z  INFO Database opened
2024-08-12T16:17:04.508512Z  INFO Starting metrics endpoint at 127.0.0.1:9001
2024-08-12T16:17:04.508520Z  INFO Starting metrics endpoint addr=127.0.0.1:9001
2024-08-12T16:17:04.508688Z  INFO 
Pre-merge hard forks (block based):
- Frontier                         @0
- Homestead                        @1150000
- Dao                              @1920000
- Tangerine                        @2463000
- SpuriousDragon                   @2675000
- Byzantium                        @4370000
- Constantinople                   @7280000
- Petersburg                       @7280000
- Istanbul                         @9069000
- MuirGlacier                      @9200000
- Berlin                           @12244000
- London                           @12965000
- ArrowGlacier                     @13773000
- GrayGlacier                      @15050000
Merge hard forks:
- Paris                            @58750000000000000000000 (network is known to be merged)
Post-merge hard forks (timestamp based):
- Shanghai                         @1681338455
- Cancun                           @1710338135
2024-08-12T16:17:04.509263Z  INFO Transaction pool initialized
2024-08-12T16:17:04.509509Z  INFO Loading saved peers file=/data/known-peers.json
2024-08-12T16:17:04.511156Z  INFO StaticFileProducer initialized
2024-08-12T16:17:04.511452Z  INFO Pruner initialized prune_config=PruneConfig { block_interval: 5, segments: PruneModes { sender_recovery: None, transaction_lookup: None, receipts: None, account_history: None, storage_history: None, receipts_log_filter: ReceiptsLogPruneConfig({}) } }
2024-08-12T16:17:04.511665Z  INFO Consensus engine initialized
2024-08-12T16:17:04.511711Z  INFO Engine API handler initialized
2024-08-12T16:17:04.513565Z  INFO RPC auth server started url=127.0.0.1:8551
2024-08-12T16:17:04.513869Z  INFO RPC IPC server started path=/ipc/reth.ipc
2024-08-12T16:17:04.513880Z  INFO RPC HTTP server started url=127.0.0.1:8543
2024-08-12T16:17:04.513883Z  INFO RPC WS server started url=127.0.0.1:8544
2024-08-12T16:17:04.513932Z  INFO Starting consensus engine
2024-08-12T16:17:07.512069Z  INFO Status connected_peers=0 freelist=67966 latest_block=20513665
2024-08-12T16:17:12.950280Z  INFO Received forkchoice updated message when syncing head_block_hash=0x94af7e1463a4d4710500dd62e81f2902397a4ab01e3e8138ad06e7ffc4a19439 safe_block_hash=0x2035318984193d60e8f2420dab5384a48be243554b88e3aa18194300820f9b13 finalized_block_hash=0x31a0cde0752215a2b27168b1095080b7f605c8031578488b6945073a0e50f062
2024-08-12T16:17:26.472128Z  INFO Block added to fork chain number=20513666 hash=0x1911a5fb3724dbc36f4a2c620d5751e410c1377b98f3f7758ed70d37dc70c85d
2024-08-12T16:17:26.668188Z  INFO Block added to fork chain number=20513667 hash=0xc30851f1b9ec8369f306a4382cfdc2ca2b1b2f6024945360d8adb09f11bcc6fa
2024-08-12T16:17:26.899688Z  INFO Block added to fork chain number=20513668 hash=0xd4279be21afeb738c7cab00d314a36a518909d2b4d5d38536ea4f8191d4adf35
2024-08-12T16:17:27.141305Z  INFO Block added to fork chain number=20513669 hash=0xed8ae497c2aff655e292790c8f6cf8b9b83eb97928c6957d56b3ccfbe368c006
2024-08-12T16:17:30.854553Z  INFO Block added to fork chain number=20513670 hash=0x94af7e1463a4d4710500dd62e81f2902397a4ab01e3e8138ad06e7ffc4a19439
2024-08-12T16:17:32.512351Z  INFO Status connected_peers=5 freelist=67966 latest_block=20513665
2024-08-12T16:17:36.830869Z  INFO Block added to fork chain number=20513671 hash=0x69fde0a70479f5504676997324391093c9622c8f0df3bd0213b0034eaca437a1
2024-08-12T16:17:36.830885Z  INFO Canonical chain committed number=20513671 hash=0x69fde0a70479f5504676997324391093c9622c8f0df3bd0213b0034eaca437a1 elapsed=198.995µs
2024-08-12T16:17:36.844840Z ERROR Persistence service failed err=ProviderError(UnexpectedStaticFileBlockNumber(Headers, 20513666, 20513571))
2024-08-12T16:17:39.175958Z ERROR Advancing persistence failed err=channel closed
2024-08-12T16:17:39.176503Z ERROR Fatal error
2024-08-12T16:17:39.176516Z ERROR Fatal error in consensus engine
2024-08-12T16:17:39.176559Z ERROR shutting down due to error
2024-08-12T16:17:39.176567Z  INFO Block added to canonical chain number=20513672 hash=0x9649d771b8a46003fe8441435d58ab7da8102d0ede5d99e06b3e51f5f52920ed peers=5 txs=158 gas=11.97 Mgas gas_throughput=5.68 Mgas/second full=39.9% base_fee=8.86gwei blobs=6 excess_blobs=1 elapsed=2.106420673s
Error: Fatal error in consensus engine

Location:
    /project/crates/node/builder/src/launch/engine.rs:297:43

Platform(s)

Linux (x86)

What version/commit are you on?

reth Version: 1.0.5
Commit SHA: 603e39a
Build Timestamp: 2024-08-12T14:15:11.150189859Z
Build Features: asm_keccak,jemalloc
Build Profile: maxperf

What database version are you on?

Current database version: 2
Local database version: 2

Which chain / network are you on?

mainnet

What type of node are you running?

Archive (default)

What prune config do you use, if any?

No response

If you've built Reth from source, provide the full command you used

No response

Code of Conduct

• I agree to follow the Code of Conduct