geth didn't sync as expected! retrieved hash chain is invalid: invalid gas used

[niekaixuan]

System information

Geth version: 1.11.4-stable
CL client & version: prysm@v3.2.2
OS & Version: Linux

problem

geth was stuck in this error: Beacon backfilling failed err="retrieved hash chain is invalid: invalid gas used (remote: 8051585 local: 8068685)"

as the same time, error occurs :
########## BAD BLOCK #########
Block: 16841878 (0xcfdac6d94781ca87cdc530936032bc37cefa399a529a6672f46d530cd3fc87f9)
Error: invalid gas used (remote: 8051585 local: 8068685)
Platform: geth (devel) go1.20.2 amd64 linux
VCS: 7e3b149b-20230310
Chain config: &params.ChainConfig{ChainID:1, HomesteadBlock:1150000, DAOForkBlock:1920000, DAOForkSupport:true, EIP150Block:2463000, EIP150Hash:0x2086799aeebeae135c246c65021c82b4e15a2c451340993aacfd2751886514f0, EIP155Block:2675000, EIP158Block:2675000, ByzantiumBlock:4370000, ConstantinopleBlock:7280000, PetersburgBlock:7280000, IstanbulBlock:9069000, MuirGlacierBlock:9200000, BerlinBlock:12244000, LondonBlock:12965000, ArrowGlacierBlock:13773000, GrayGlacierBlock:15050000, MergeNetsplitBlock:<nil>, ShanghaiTime:(*uint64)(nil), CancunTime:(*uint64)(nil), PragueTime:(*uint64)(nil), TerminalTotalDifficulty:58750000000000000000000, TerminalTotalDifficultyPassed:true, Ethash:(*params.EthashConfig)(0x2b76920), Clique:(*params.CliqueConfig)(nil)}

issue #14435 has similar problem，but current geth doesn‘t provide delete or reset apis.

before error occurs

i shutdown geth use kill -9 command，as i want to know what will happen

can i just remove the bad block instead of re-sync from 0?

[jsvisa]

Maybe you can try resetting block header with debug.setHead() to a block number before 16841878(eg 16841878-1000) instead for resync from 0

[lightclient]

Can you share some logs from before this incident?

[niekaixuan]

Can you share some logs from before this incident?

I got logs below，there is a prysm connection to geth, but still log out "no beacon client seen". prysm log out error beacause of the Bad block as well.

==== log from geth

WARN [03-23|00:18:57.671] Post-merge network, but no beacon client seen. Please launch one to follow the chain! 
WARN [03-23|00:22:37.692] Post-merge network, but no beacon client seen. Please launch one to follow the chain! 
WARN [03-23|00:22:47.424] Checkpoint challenge timed out, dropping id=bdfa6602f579b499 conn=dyndial             addr=34.202.193.18:30303   type=Geth/v1.11.1-stable/...
WARN [03-23|00:24:12.700] Post-merge network, but no beacon client seen. Please launch one to follow the chain! 
WARN [03-23|00:25:27.709] Post-merge network, but no beacon client seen. Please launch one to follow the chain! 
WARN [03-23|00:28:32.728] Post-merge network, but no beacon client seen. Please launch one to follow the chain! 
WARN [03-23|00:29:32.734] Post-merge network, but no beacon client seen. Please launch one to follow the chain! 
WARN [03-23|00:31:22.745] Post-merge network, but no beacon client seen. Please launch one to follow the chain! 
WARN [03-23|00:32:27.751] Post-merge network, but no beacon client seen. Please launch one to follow the chain! 
WARN [03-23|00:33:07.755] Post-merge network, but no beacon client seen. Please launch one to follow the chain! 
WARN [03-23|00:38:27.790] Post-merge network, but no beacon client seen. Please launch one to follow the chain! 
WARN [03-23|00:39:27.795] Post-merge network, but no beacon client seen. Please launch one to follow the chain! 
WARN [03-23|00:40:12.800] Post-merge network, but no beacon client seen. Please launch one to follow the chain! 
WARN [03-23|00:41:07.806] Post-merge network, but no beacon client seen. Please launch one to follow the chain! 
WARN [03-23|00:45:12.831] Post-merge network, but no beacon client seen. Please launch one to follow the chain! 
WARN [03-23|00:48:12.852] Post-merge network, but no beacon client seen. Please launch one to follow the chain! 
WARN [03-23|00:48:57.857] Post-merge network, but no beacon client seen. Please launch one to follow the chain! 
WARN [03-23|00:52:27.876] Post-merge network, but no beacon client seen. Please launch one to follow the chain! 
INFO [03-23|00:53:36.065] Writing clean trie cache to disk         path=/data/eth/geth/geth/triecache threads=1
INFO [03-23|00:53:36.065] Regenerated local transaction journal    transactions=0 accounts=0
INFO [03-23|00:53:37.075] Persisted the clean trie cache           path=/data/eth/geth/geth/triecache elapsed=1.010s
WARN [03-23|00:54:20.323] Snapshot extension registration failed   peer=8ea667b8 err="peer connected on snap without compatible eth support"
WARN [03-23|00:55:02.891] Post-merge network, but no beacon client seen. Please launch one to follow the chain! 
WARN [03-23|00:56:32.902] Post-merge network, but no beacon client seen. Please launch one to follow the chain! 
WARN [03-23|00:57:47.909] Post-merge network, but no beacon client seen. Please launch one to follow the chain! 
WARN [03-23|01:00:37.925] Post-merge network, but no beacon client seen. Please launch one to follow the chain! 
WARN [03-23|01:01:47.933] Post-merge network, but no beacon client seen. Please launch one to follow the chain! 
WARN [03-23|01:05:07.954] Post-merge network, but no beacon client seen. Please launch one to follow the chain! 
WARN [03-23|01:06:52.961] Post-merge network, but no beacon client seen. Please launch one to follow the chain! 
WARN [03-23|01:08:02.968] Post-merge network, but no beacon client seen. Please launch one to follow the chain! 
WARN [03-23|01:10:17.982] Post-merge network, but no beacon client seen. Please launch one to follow the chain! 
WARN [03-23|01:13:43.006] Post-merge network, but no beacon client seen. Please launch one to follow the chain! 
WARN [03-23|01:14:33.011] Post-merge network, but no beacon client seen. Please launch one to follow the chain! 
WARN [03-23|01:17:23.028] Post-merge network, but no beacon client seen. Please launch one to follow the chain! 
WARN [03-23|01:18:18.034] Post-merge network, but no beacon client seen. Please launch one to follow the chain! 
WARN [03-23|01:18:45.256] Too many bad block import attempt, trying number=16,841,878 hash=cfdac6..fc87f9
INFO [03-23|01:18:45.300] Forkchoice requested sync to new head    number=16,867,920 hash=886916..75bfa6 finalized=16,867,858
INFO [03-23|01:18:45.687] Forkchoice requested sync to new head    number=16,867,921 hash=d950f5..0156c9 finalized=16,867,858
INFO [03-23|01:18:45.861] Forkchoice requested sync to new head    number=16,867,922 hash=d7d175..35d12f finalized=16,867,858
INFO [03-23|01:18:46.016] Forkchoice requested sync to new head    number=16,867,923 hash=d6eddc..defa02 finalized=16,867,858
INFO [03-23|01:18:46.171] Forkchoice requested sync to new head    number=16,867,924 hash=bdd159..009cd3 finalized=16,867,858
INFO [03-23|01:18:46.315] Forkchoice requested sync to new head    number=16,867,925 hash=65e8ff..a1145b finalized=16,867,858
INFO [03-23|01:18:46.519] Forkchoice requested sync to new head    number=16,867,926 hash=f440e9..48be37 finalized=16,867,858
INFO [03-23|01:18:46.657] Forkchoice requested sync to new head    number=16,867,927 hash=34417d..0777c2 finalized=16,867,858
INFO [03-23|01:18:46.798] Forkchoice requested sync to new head    number=16,867,928 hash=39c95b..39c1b8 finalized=16,867,858
INFO [03-23|01:18:46.961] Forkchoice requested sync to new head    number=16,867,929 hash=749b27..d818ae finalized=16,867,858
INFO [03-23|01:18:47.094] Forkchoice requested sync to new head    number=16,867,930 hash=493ea4..b364d1 finalized=16,867,858
INFO [03-23|01:18:47.255] Forkchoice requested sync to new head    number=16,867,931 hash=de2ba5..65acd1 finalized=16,867,858
INFO [03-23|01:18:47.393] Forkchoice requested sync to new head    number=16,867,932 hash=23c5e1..d699a6 finalized=16,867,858
INFO [03-23|01:18:47.473] Skip duplicated bad block                number=16,841,878 hash=cfdac6..fc87f9
########## BAD BLOCK #########
Block: 16841878 (0xcfdac6d94781ca87cdc530936032bc37cefa399a529a6672f46d530cd3fc87f9)
Error: invalid gas used (remote: 8051585 local: 8068685)
Platform: geth (devel) go1.20.2 amd64 linux
VCS: 7e3b149b-20230310

=== log from prysm ===

time="2023-03-24 06:28:32" level=warning msg="Block is not processed" error="could not process block: could not validate new payload: received an INVALID payload from execution engine" prefix=initial-sync

[niekaixuan]

Maybe you can try resetting block header with debug.setHead() to a block number before 16841878(eg 16841878-1000) instead for resync from 0

I have tried to find any api such like setHead() ,even in current master geth source code, but I didn't find one. could you please tell me how?

=== updated ===
I find this api in geth.ipc, and rewind the head, it makes sense! thinks a lot

[holiman]

i shutdown geth use kill -9 command，as i want to know what will happen

Well, now you know. Crashing geth might unfortunately lead to data corruption. I'm afraid the best course of action from here is to resync from scratch.

[niekaixuan]

well, debug.setHead() doesn't work eventually, geth stuck in same situation after rewinding head,as a result I use geth removedb to resync statedb, time-consuming but practicable