e3 mainnet tip deadloop block delivery (alpha5)

[RedstonedLife]

System information

Erigon version: 3.00.0-alpha5-78f3647d

OS & Version: Ubuntu 22.04.4 LTS x86_64

Commit hash: 78f3647d

Erigon Command (with flags/config):

Consensus Layer: Internal (Caplin)

Consensus Layer Command (with flags/config): /home/erigon/build/bin/erigon --internalcl --datadir /cryptonode/ethereum --chain mainnet --torrent.download.rate=5000mb --torrent.maxpeers=200 --batchSize=3G --etl.buffersize=3G

Chain/Network: Ethereum Mainnet

Expected behaviour

Caplin processing block batch as intended

Actual behaviour

Caplin fails to process a block batch, saying bad blocks segment received: replay block, status block has invalid parent.

Steps to reproduce the behaviour

Install Erigon, sync with internalcl

Backtrace

Nov 17 23:02:59 enode-01 erigon[185849]: [INFO] [11-17|23:02:59.583] [Caplin] Forward Sync                    app=caplin stage=ForwardSync progress=10421617 distance-from-chain-tip=18m36s estimated-time-remaining=56s
Nov 17 23:03:11 enode-01 erigon[185849]: [WARN] [11-17|23:03:11.892] Invalid beacon block                     slot=10421667 blockRoot="[63 103 251 107 29 230 116 6 183 240 176 117 170 42 30 135 155 177 245 255 68 62 176 127 112 11 225 168 220 217 47 66]" reason="processBlock: failed to process block body operations: ProcessAttestation: invalid validator index"
Nov 17 23:03:11 enode-01 erigon[185849]: [WARN] [11-17|23:03:11.893] [Caplin] Failed to process block batch   app=caplin stage=ForwardSync err="bad blocks segment received: processBlock: failed to process block body operations: ProcessAttestation: invalid validator index"
Nov 17 23:03:12 enode-01 erigon[185849]: [WARN] [11-17|23:03:12.995] [Caplin] Failed to process block batch   app=caplin stage=ForwardSync err="bad blocks segment received: replay block, status block has invalid parent"
Nov 17 23:03:13 enode-01 erigon[185849]: [WARN] [11-17|23:03:13.772] [Caplin] Failed to process block batch   app=caplin stage=ForwardSync err="bad blocks segment received: replay block, status block has invalid parent"
Nov 17 23:03:15 enode-01 erigon[185849]: [WARN] [11-17|23:03:15.010] [Caplin] Failed to process block batch   app=caplin stage=ForwardSync err="bad blocks segment received: replay block, status block has invalid parent"
Nov 17 23:03:16 enode-01 erigon[185849]: [WARN] [11-17|23:03:16.874] [Caplin] Failed to process block batch   app=caplin stage=ForwardSync err="bad blocks segment received: replay block, status block has invalid parent"
Nov 17 23:03:18 enode-01 erigon[185849]: [WARN] [11-17|23:03:18.295] [Caplin] Failed to process block batch   app=caplin stage=ForwardSync err="bad blocks segment received: replay block, status block has invalid parent"
Nov 17 23:03:18 enode-01 erigon[185849]: [WARN] [11-17|23:03:18.992] [Caplin] Failed to process block batch   app=caplin stage=ForwardSync err="bad blocks segment received: replay block, status block has invalid parent"
Nov 17 23:03:20 enode-01 erigon[185849]: [WARN] [11-17|23:03:20.935] [Caplin] Failed to process block batch   app=caplin stage=ForwardSync err="bad blocks segment received: replay block, status block has invalid parent"
Nov 17 23:03:24 enode-01 erigon[185849]: [WARN] [11-17|23:03:24.289] [Caplin] Failed to process block batch   app=caplin stage=ForwardSync err="bad blocks segment received: replay block, status block has invalid parent"
Nov 17 23:03:25 enode-01 erigon[185849]: [WARN] [11-17|23:03:25.957] [Caplin] Failed to process block batch   app=caplin stage=ForwardSync err="bad blocks segment received: replay block, status block has invalid parent"
Nov 17 23:03:28 enode-01 erigon[185849]: [WARN] [11-17|23:03:28.573] [Caplin] Failed to process block batch   app=caplin stage=ForwardSync err="bad blocks segment received: replay block, status block has invalid parent"
Nov 17 23:03:48 enode-01 erigon[185849]: [INFO] [11-17|23:03:48.227] P2P                                      app=caplin peers=0
Nov 17 23:04:22 enode-01 erigon[185849]: [WARN] [11-17|23:04:22.085] [Caplin] Failed to process block batch   app=caplin stage=ForwardSync err="bad blocks segment received: replay block, status block has invalid parent"
Nov 17 23:04:22 enode-01 erigon[185849]: [INFO] [11-17|23:04:22.085] [Caplin] Forward Sync                    app=caplin stage=ForwardSync progress=10421655 distance-from-chain-tip=11m0s estimated-time-remaining=43s
Nov 17 23:04:23 enode-01 erigon[185849]: [WARN] [11-17|23:04:23.254] [Caplin] Failed to process block batch   app=caplin stage=ForwardSync err="bad blocks segment received: replay block, status block has invalid parent"
rs=017 23:04:48 enode-01 erigon[185849]: [INFO] [11-17|23:04:48.227] P2P                                      app=caplin peeNov 17 23:04:54 enode-01 erigon[185849]: [WARN] [11-17|23:04:54.450] [Caplin] Failed to process block batch   app=caplin stage=ForwardSync err="bad blocks segment received: replay block, status block has invalid parent"
Nov 17 23:04:54 enode-01 erigon[185849]: [INFO] [11-17|23:04:54.450] [Caplin] Forward Sync                    app=caplin stage=ForwardSync progress=10421655 distance-from-chain-tip=11m0s estimated-time-remaining=999h0m0s
Nov 17 23:05:00 enode-01 erigon[185849]: [WARN] [11-17|23:05:00.396] [Caplin] Failed to process block batch   app=caplin stage=ForwardSync err="bad blocks segment received: replay block, status block has invalid parent"
Nov 17 23:05:00 enode-01 erigon[185849]: [INFO] [11-17|23:05:00.396] [Caplin] Forward Sync                    app=caplin stage=ForwardSync progress=10421655 distance-from-chain-tip=11m0s estimated-time-remaining=999h0m0s
Nov 17 23:05:12 enode-01 erigon[185849]: [WARN] [11-17|23:05:12.584] [Caplin] Failed to process block batch   app=caplin stage=ForwardSync err="bad blocks segment received: replay block, status block has invalid parent"
Nov 17 23:05:16 enode-01 erigon[185849]: [WARN] [11-17|23:05:16.350] [Caplin] Failed to process block batch   app=caplin stage=ForwardSync err="bad blocks segment received: replay block, status block has invalid parent"
Nov 17 23:05:17 enode-01 erigon[185849]: [WARN] [11-17|23:05:17.335] [Caplin] Failed to process block batch   app=caplin stage=ForwardSync err="bad blocks segment received: replay block, status block has invalid parent"
Nov 17 23:05:18 enode-01 erigon[185849]: [WARN] [11-17|23:05:18.397] [Caplin] Failed to process block batch   app=caplin stage=ForwardSync err="bad blocks segment received: replay block, status block has invalid parent"

Related to issue #10484 from what it seems.

[RedstonedLife]

A restart seems to fix it temporarily, I'm watching the node closely for any hiccups.

[AskAlexSharov]

i think we fixed it in main

[AskAlexSharov]

closing for now. feel free to reopen