Ful Sync: BAD BLOCK

[elsampsa]

Synopsis

Geth stops working after finding a bad block:

Error: could not apply tx 122 [0x2e91ec8bde129768ad253384dc6002e66cd878de2bc393373998a8c4ac3a456e]: gas limit reached

(please see the whole backlog in the end).

Similar issues:

• BAD BLOCK errors #20478
• Full Sync: BAD BLOCK: gas limit reached  #20485

What I understand from these issues is that, as geth sifts through enormous amounts of data, with more time and data, the probability for a random erroneous bit-flip goes up. And then it happens and then the (whole?) blockchain goes sour.

Do the unclean shutdowns (see the backlog below) increase this probability as well?

So instead of a bug, this is blockchain corruption?

Is there any way to remove the corrupted block .. or is the origin of the problem hidden somewhere deep down in the blockchain and the only way to solve this, is to download the whole thing again? (a think I'd rather avoid since I never got the fast sync to actually sync, see this).

System information

Geth version: 1.10
OS & Version: LInux (Ubuntu 18.04 LTS)

Expected behaviour

Actual behaviour

Geth stops as it finds a "bad block"

Steps to reproduce the behaviour

N/A

Backtrace

INFO [04-16|23:48:11.401] Starting Geth on Ethereum mainnet... 
INFO [04-16|23:48:11.405] Maximum peer count                       ETH=50 LES=0 total=50
INFO [04-16|23:48:11.406] Smartcard socket not found, disabling    err="stat /run/pcscd/pcscd.comm: no such file or directory"
INFO [04-16|23:48:11.408] Set global gas cap                       cap=25000000
INFO [04-16|23:48:11.408] Allocated trie memory caches             clean=511.00MiB dirty=512.00MiB
INFO [04-16|23:48:11.408] Allocated cache and file handles         database=/home/sampsa/.ethereum/geth/chaindata cache=1024.00MiB handles=524288
INFO [04-16|23:48:17.383] Opened ancient database                  database=/home/sampsa/.ethereum/geth/chaindata/ancient
INFO [04-16|23:48:17.417] Initialised chain configuration          config="{ChainID: 1 Homestead: 1150000 DAO: 1920000 DAOSupport: true EIP150: 2463000 EIP155: 2675000 EIP158: 2675000 Byzantium: 4370000 Constantinople: 7280000 Petersburg: 7280000 Istanbul: 9069000, Muir Glacier: 9200000, Berlin: <nil>, YOLO v3: <nil>, Engine: ethash}"
INFO [04-16|23:48:20.487] Disk storage enabled for ethash caches   dir=/home/sampsa/.ethereum/geth/ethash count=3
INFO [04-16|23:48:20.487] Disk storage enabled for ethash DAGs     dir=/home/sampsa/.ethash               count=2
INFO [04-16|23:48:20.498] Initialising Ethereum protocol           network=1 dbversion=8
INFO [04-16|23:48:21.073] Loaded most recent local header          number=12244002 hash="96f625…3cc4fb" td=23316431817258711456735 age=1d3h39m
INFO [04-16|23:48:21.073] Loaded most recent local full block      number=12244002 hash="96f625…3cc4fb" td=23316431817258711456735 age=1d3h39m
INFO [04-16|23:48:21.073] Loaded most recent local fast block      number=12244002 hash="96f625…3cc4fb" td=23316431817258711456735 age=1d3h39m
INFO [04-16|23:48:21.090] Loaded local transaction journal         transactions=0 dropped=0
INFO [04-16|23:48:21.090] Regenerated local transaction journal    transactions=0 accounts=0
WARN [04-16|23:48:21.102] Unclean shutdown detected                booted=2021-03-06T11:38:58+0200 age=1mo1w4d
WARN [04-16|23:48:21.102] Unclean shutdown detected                booted=2021-03-10T22:27:41+0200 age=1mo1w20m
WARN [04-16|23:48:21.102] Unclean shutdown detected                booted=2021-03-18T19:50:26+0200 age=4w1d2h
WARN [04-16|23:48:21.103] Unclean shutdown detected                booted=2021-03-19T21:34:14+0200 age=4w1h14m
WARN [04-16|23:48:21.103] Unclean shutdown detected                booted=2021-03-24T19:33:27+0200 age=3w2d3h
WARN [04-16|23:48:21.103] Unclean shutdown detected                booted=2021-04-02T13:21:11+0300 age=2w10h27m
WARN [04-16|23:48:21.103] Unclean shutdown detected                booted=2021-04-03T11:16:43+0300 age=1w6d12h
INFO [04-16|23:48:21.103] Starting peer-to-peer node               instance=Geth/v1.10.0-stable-56dec25a/linux-amd64/go1.15.6
INFO [04-16|23:48:21.203] New local node record                    seq=231 id=42e508921cacda09 ip=127.0.0.1 udp=30303 tcp=30303
INFO [04-16|23:48:21.203] Started P2P networking                   self=enode://46eaa513d960e6edaa88575ad966c46ad75fea4e39c1eddd6cfe522a6341f05c9358d0421ceb5a55b4670ecbc3b2fb986a9f301c889339f5d1d70b04b28875d7@127.0.0.1:30303
INFO [04-16|23:48:21.205] IPC endpoint opened                      url=/home/sampsa/.ethereum/geth.ipc
INFO [04-16|23:48:23.662] New local node record                    seq=232 id=42e508921cacda09 ip=84.248.88.68 udp=30303 tcp=30303
INFO [04-16|23:48:31.865] Looking for peers                        peercount=4 tried=44 static=0
INFO [04-16|23:48:41.899] Looking for peers                        peercount=2 tried=48 static=0
INFO [04-16|23:48:52.107] Looking for peers                        peercount=5 tried=45 static=0
INFO [04-16|23:49:02.157] Looking for peers                        peercount=20 tried=50 static=0
INFO [04-16|23:49:12.166] Looking for peers                        peercount=20 tried=50 static=0
INFO [04-16|23:49:22.491] Looking for peers                        peercount=25 tried=50 static=0
INFO [04-16|23:49:32.491] Looking for peers                        peercount=24 tried=42 static=0
WARN [04-16|23:50:28.181] Checkpoint challenge timed out, dropping id=c1c7674136926899 conn=inbound addr=195.211.153.101:34594 type=OpenEthereum/v3.1.1-...
ERROR[04-16|23:50:28.181] Ethereum peer removal failed             peer=c1c76741 err="peer not registered"
INFO [04-16|23:59:36.508] Block synchronisation started 
INFO [04-16|23:59:58.658] Downloader queue stats                   receiptTasks=0 blockTasks=0 itemSize=5.86KiB throttle=8192
INFO [04-16|23:59:59.024] Sidechain written to disk                start=12244000 end=12244001 sidetd=23316426079725045704185 localtd=23316431817258711456735
INFO [04-16|23:59:59.538] Importing sidechain segment              start=12243876 end=12244005
INFO [04-17|00:00:08.970] Imported new chain segment               blocks=3 txs=431 mgas=37.542 elapsed=9.432s mgasps=3.980 number=12243878 hash="42f4ce…59c635" age=1d11h18m dirty=4.88MiB
INFO [04-17|00:00:18.397] Imported new chain segment               blocks=3 txs=425 mgas=37.267 elapsed=9.427s mgasps=3.953 number=12243881 hash="ff9acf…826bfa" age=1d11h18m dirty=10.36MiB
INFO [04-17|00:00:28.956] Imported new chain segment               blocks=3 txs=486 mgas=37.551 elapsed=10.559s mgasps=3.556 number=12243884 hash="63491c…74a594" age=1d11h18m dirty=15.66MiB
ERROR[04-17|00:00:51.919] Ethereum peer removal failed             peer=425f6a40 err="peer not registered"
INFO [04-17|00:00:54.381] Imported new chain segment               blocks=3 txs=663 mgas=37.532 elapsed=25.424s mgasps=1.476 number=12243887 hash="6dd100…203d05" age=1d11h18m dirty=21.36MiB
INFO [04-17|00:01:03.272] Imported new chain segment               blocks=5 txs=943 mgas=62.511 elapsed=8.890s  mgasps=7.031 number=12243892 hash="abf749…04fa47" age=1d11h16m dirty=30.25MiB
INFO [04-17|00:01:13.146] Imported new chain segment               blocks=6 txs=788 mgas=62.010 elapsed=9.873s  mgasps=6.280 number=12243898 hash="4b9544…d7e661" age=1d11h16m dirty=39.06MiB
INFO [04-17|00:01:21.828] Imported new chain segment               blocks=5 txs=1006 mgas=62.417 elapsed=8.681s  mgasps=7.189 number=12243903 hash="ee0e53…c9f232" age=1d11h15m dirty=48.26MiB
INFO [04-17|00:01:30.515] Imported new chain segment               blocks=6 txs=896  mgas=74.888 elapsed=8.687s  mgasps=8.620 number=12243909 hash="20c84b…b61f3a" age=1d11h14m dirty=57.94MiB
INFO [04-17|00:01:39.002] Imported new chain segment               blocks=7 txs=1147 mgas=86.826 elapsed=8.487s  mgasps=10.230 number=12243916 hash="924c26…f41bab" age=1d11h13m dirty=69.67MiB
INFO [04-17|00:01:50.733] Imported new chain segment               blocks=6 txs=1015 mgas=74.916 elapsed=11.730s mgasps=6.386  number=12243922 hash="ec004d…24144d" age=1d11h11m dirty=79.03MiB
INFO [04-17|00:01:59.928] Imported new chain segment               blocks=7 txs=1031 mgas=87.493 elapsed=9.194s  mgasps=9.515  number=12243929 hash="8cdb7f…b85302" age=1d11h10m dirty=90.27MiB
INFO [04-17|00:02:08.658] Imported new chain segment               blocks=6 txs=1054 mgas=74.879 elapsed=8.730s  mgasps=8.577  number=12243935 hash="583fad…11b657" age=1d11h9m  dirty=101.01MiB
INFO [04-17|00:02:17.643] Imported new chain segment               blocks=4 txs=573  mgas=49.884 elapsed=8.984s  mgasps=5.552  number=12243939 hash="d03e0f…e881a3" age=1d11h8m  dirty=107.69MiB
INFO [04-17|00:02:26.580] Imported new chain segment               blocks=8 txs=1089 mgas=99.309 elapsed=8.936s  mgasps=11.112 number=12243947 hash="144eed…fdc690" age=1d11h7m  dirty=120.88MiB
INFO [04-17|00:02:35.469] Imported new chain segment               blocks=8 txs=928  mgas=99.564 elapsed=8.889s  mgasps=11.200 number=12243955 hash="57d564…81c335" age=1d11h5m  dirty=133.39MiB
INFO [04-17|00:02:44.549] Imported new chain segment               blocks=7 txs=1380 mgas=87.358 elapsed=9.080s  mgasps=9.621  number=12243962 hash="60542f…57e059" age=1d11h3m  dirty=145.74MiB
INFO [04-17|00:02:53.111] Imported new chain segment               blocks=10 txs=1558 mgas=124.122 elapsed=8.561s  mgasps=14.497 number=12243972 hash="389fea…96c61f" age=1d11h2m  dirty=162.32MiB
INFO [04-17|00:03:01.999] Imported new chain segment               blocks=6  txs=1134 mgas=74.896  elapsed=8.888s  mgasps=8.427  number=12243978 hash="79bff2…1401e4" age=1d11h23s dirty=173.03MiB
INFO [04-17|00:03:10.331] Imported new chain segment               blocks=10 txs=1158 mgas=124.494 elapsed=8.331s  mgasps=14.942 number=12243988 hash="2f6744…2cbf08" age=1d10h59m dirty=188.37MiB
INFO [04-17|00:03:18.467] Imported new chain segment               blocks=7  txs=1173 mgas=87.290  elapsed=8.136s  mgasps=10.729 number=12243995 hash="679f3d…df8daa" age=1d10h57m dirty=199.74MiB
INFO [04-17|00:03:23.080] Skip duplicated bad block                number=12244000 hash="163838…c42b21"
ERROR[04-17|00:03:23.080] 
########## BAD BLOCK #########
Chain config: {ChainID: 1 Homestead: 1150000 DAO: 1920000 DAOSupport: true EIP150: 2463000 EIP155: 2675000 EIP158: 2675000 Byzantium: 4370000 Constantinople: 7280000 Petersburg: 7280000 Istanbul: 9069000, Muir Glacier: 9200000, Berlin: <nil>, YOLO v3: <nil>, Engine: ethash}

Number: 12244000
Hash: 0x1638380ab737e0e916bd1c7f23bd2bab2a532e44b90047f045f262ee21c42b21


Error: could not apply tx 122 [0x2e91ec8bde129768ad253384dc6002e66cd878de2bc393373998a8c4ac3a456e]: gas limit reached
##############################
 
WARN [04-17|00:03:23.080] Synchronisation failed, dropping peer    peer=99f531968015743ac8aff0f2873156be9d73251f4d88b50e652552e59b82db2f err="retrieved hash chain is invalid: no peers available or all tried for download"
ERROR[04-17|00:03:23.080] Ethereum peer removal failed             peer=99f53196 err="peer not registered"
INFO [04-17|00:03:35.875] Downloader queue stats                   receiptTasks=0 blockTasks=3304 itemSize=14.00KiB throttle=4683
INFO [04-17|00:03:35.939] Importing sidechain segment              start=12244000 end=12244009
INFO [04-17|00:03:36.004] Skip duplicated bad block                number=12244000 hash="163838…c42b21"
ERROR[04-17|00:03:36.004] 
########## BAD BLOCK #########
Chain config: {ChainID: 1 Homestead: 1150000 DAO: 1920000 DAOSupport: true EIP150: 2463000 EIP155: 2675000 EIP158: 2675000 Byzantium: 4370000 Constantinople: 7280000 Petersburg: 7280000 Istanbul: 9069000, Muir Glacier: 9200000, Berlin: <nil>, YOLO v3: <nil>, Engine: ethash}

Number: 12244000
Hash: 0x1638380ab737e0e916bd1c7f23bd2bab2a532e44b90047f045f262ee21c42b21


Error: could not apply tx 122 [0x2e91ec8bde129768ad253384dc6002e66cd878de2bc393373998a8c4ac3a456e]: gas limit reached
##############################
 
WARN [04-17|00:03:36.004] Synchronisation failed, dropping peer    peer=0ec3556e4394dd8e6a722bb93ab11751842a9c34c16b4043c44007b37831ff20 err="retrieved hash chain is invalid: could not apply tx 122 [0x2e91ec8bde129768ad253384dc6002e66cd878de2bc393373998a8c4ac3a456e]: gas limit reached"
ERROR[04-17|00:03:36.004] Ethereum peer removal failed             peer=0ec3556e err="peer not registered"
WARN [04-17|00:05:48.574] Synchronisation failed, dropping peer    peer=844e984310fb16ecbcfe7f2ced0284dbd25014d304cf828ad2b8a0956cd4ddb9 err=timeout
ERROR[04-17|00:05:48.574] Ethereum peer removal failed             peer=844e9843 err="peer not registered"

[Texnomic]

I'm facing the same issue on the same exact block

ERROR[04-17|09:36:21.648]  
########## BAD BLOCK #########

Chain config: {ChainID: 1 Homestead: 1150000 DAO: 1920000 DAOSupport: true EIP150: 2463000 EIP155: 2675000 EIP158: 2675000 Byzantium: 4370000 Constantinople: 7280000 Petersburg: 7280000 Istanbul: 9069000, Muir Glacier: 9200000, Berlin: <nil>, YOLO v3: <nil>, Engine: ethash}

Number: 12244000
Hash: 0x1638380ab737e0e916bd1c7f23bd2bab2a532e44b90047f045f262ee21c42b21
Error: could not apply tx 122 [0x2e91ec8bde129768ad253384dc6002e66cd878de2bc393373998a8c4ac3a456e]: gas limit reached 

##############################                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                

[ligi]

This block is the Berlin HardFork - please read: https://blog.ethereum.org/2021/03/08/ethereum-berlin-upgrade-announcement/ and install the latest geth

[Texnomic]

This block is the Berlin HardFork - please read: https://blog.ethereum.org/2021/03/08/ethereum-berlin-upgrade-announcement/ and install the latest geth

I confirm this fixed the issue for me.

[elsampsa]

Fixed for me as well