Sync error - ########## BAD BLOCK #########

[fernandezpaco]

Hi there,

System information

Geth version: 1.9.21
OS & Version: OSX
RAM: 24GB
DISK: 512GB EXTERNAL SSD

Expected behaviour

Sync smoothly

I have tried three times but i always get this error.

I have tried deleting the cache folder but still get this error

Any idea why?

Actual behaviour

Bad block error received

INFO [09-11|11:30:25.707] Imported new chain segment               blocks=19   txs=671  mgas=28.304  elapsed=8.097s    mgasps=3.495   number=2701098 hash="4fa380…e0d76c" age=3y10mo4d dirty=508.73MiB
INFO [09-11|11:30:33.978] Imported new chain segment               blocks=14   txs=442  mgas=21.670  elapsed=8.271s    mgasps=2.620   number=2701112 hash="67dd8e…9664e4" age=3y10mo4d dirty=511.49MiB
INFO [09-11|11:30:42.779] Imported new chain segment               blocks=19   txs=96   mgas=16.310  elapsed=8.801s    mgasps=1.853   number=2701131 hash="5f0e2e…738cbf" age=3y10mo4d dirty=511.74MiB
INFO [09-11|11:30:52.118] Imported new chain segment               blocks=25   txs=142  mgas=19.795  elapsed=9.338s    mgasps=2.120   number=2701156 hash="783b2c…b34ab1" age=3y10mo4d dirty=511.13MiB
INFO [09-11|11:31:00.131] Imported new chain segment               blocks=21   txs=94   mgas=14.228  elapsed=8.013s    mgasps=1.776   number=2701177 hash="96db97…65add3" age=3y10mo4d dirty=504.46MiB
INFO [09-11|11:31:08.717] Imported new chain segment               blocks=23   txs=134  mgas=20.142  elapsed=8.586s    mgasps=2.346   number=2701200 hash="da7f00…6fb8ab" age=3y10mo4d dirty=507.67MiB
INFO [09-11|11:31:11.778] Deep froze chain segment                 blocks=140  elapsed=75.230ms  number=2611202 hash="026544…e2b872"
INFO [09-11|11:31:17.917] Imported new chain segment               blocks=10   txs=52   mgas=11.921  elapsed=9.200s    mgasps=1.296   number=2701210 hash="c45184…7433f3" age=3y10mo4d dirty=509.77MiB
WARN [09-11|11:31:17.923] Synchronisation failed, dropping peer    peer=113b461c9c0caae4 err="retrieved hash chain is invalid: invalid mix digest"
INFO [09-11|11:31:21.758] Looking for peers                        peercount=36 tried=2 static=0
INFO [09-11|11:31:22.958] Downloader queue stats                   receiptTasks=0 blockTasks=0     itemSize=1.91KiB throttle=8192
ERROR[09-11|11:31:22.965] 
########## BAD BLOCK #########
Chain config: {ChainID: 1 Homestead: 1150000 DAO: 1920000 DAOSupport: true EIP150: 2463000 EIP155: 2675000 EIP158: 2675000 Byzantium: 4370000 Constantinople: 7280000 Petersburg: 7280000 Istanbul: 9069000, Muir Glacier: 9200000, YOLO v1: <nil>, Engine: ethash}
Number: 2701212
Hash: 0x69b4d581b7639665f6850655652c3ff2b2f6d92ee8170edd9f1004c4057bc580


Error: invalid mix digest
##############################
 
WARN [09-11|11:31:22.965] Synchronisation failed, dropping peer    peer=5df77b3492a3dca7 err="retrieved hash chain is invalid: invalid mix digest"
ERROR[09-11|11:31:25.912] 

Steps to reproduce the behaviour

geth --cache=2048 --rpcapi "eth,net,web3,txpool" --rpc --txpool.globalslots=250000 --txpool.globalqueue=50000 --syncmode=full --datadir "/Volumes/EthNode/gethdata/" --ipcdisable

Backtrace

[backtrace]

[holiman]

That looks very odd indeed! I started a full-sync with master only yesterday, on the following commits:

Sept 10 19:47:03 bench03.ethdevops.io geth INFO [09-10|17:47:02.985] Starting peer-to-peer node instance=Geth/v1.9.21-unstable-1008730f-20200910/linux-amd64/go1.15.2
Sept 10 19:47:10 bench04.ethdevops.io geth INFO [09-10|17:47:09.945] Starting peer-to-peer node instance=Geth/v1.9.22-unstable-a3cd8a04-20200910/linux-amd64/go1.15.2 

They went through the 2M segment without any problems, so I don't think we have any regression there.

The error you have there is 'invalid mix digest' -- it's happened once before, @MysticRyuujin hit that, and I spent a lot of time investigating. Eventually we found that the ethash disk lookup had been corrupted, a bit had flipped.

I would suspect that this resolves itself if you wipe out the ethash directory. I would also be interested to know more about your hardware, like what disks you are using, what filesystem, and whether the machine has ECC or Non-ECC ram memory..?

[fernandezpaco]

I already tried deleting the ethash directory and got the same error.

I have a imac (osx 10.15.6).
I have a 1TB internal fusion drive but i am running geth on an external WD my passport SSD 512GB ExFAT file system
Memory: 24 GB 1600 MHz DDR3

Any help would be much appreciated

[fernandezpaco]

Just to clarify, after first error i updated to latest version (I was just one version behind), then i deleted the cache restarted geth, run for a while and then got the same error.

I have it running again

[fernandezpaco]

It has been running for an hour or so and then same error happened:

INFO [09-11|15:53:30.387] Imported new chain segment               blocks=2033  txs=12951 mgas=477.355 elapsed=8.000s    mgasps=59.666  number=2957315 hash="5c4ac5…f3c5dd" age=3y8mo3w   dirty=276.93MiB
INFO [09-11|15:53:30.429] Imported new chain segment               blocks=15    txs=103   mgas=2.700   elapsed=41.806ms  mgasps=64.587  number=2957330 hash="7270f6…b3cdaf" age=3y8mo3w   dirty=276.93MiB
INFO [09-11|15:53:38.442] Imported new chain segment               blocks=1985  txs=14975 mgas=504.220 elapsed=8.000s    mgasps=63.022  number=2959315 hash="319f53…d1f763" age=3y8mo3w   dirty=275.59MiB
INFO [09-11|15:53:38.595] Imported new chain segment               blocks=63    txs=377   mgas=12.547  elapsed=153.623ms mgasps=81.677  number=2959378 hash="9058c6…1031be" age=3y8mo3w   dirty=275.94MiB
INFO [09-11|15:53:46.651] Imported new chain segment               blocks=1648  txs=10139 mgas=345.949 elapsed=8.043s    mgasps=43.010  number=2961026 hash="b0be81…676b26" age=3y8mo3w   dirty=275.73MiB
INFO [09-11|15:53:47.685] Imported new chain segment               blocks=400   txs=1732  mgas=57.889  elapsed=1.033s    mgasps=56.004  number=2961426 hash="f31a3c…7a917a" age=3y8mo3w   dirty=276.09MiB
INFO [09-11|15:53:51.231] Deep froze chain segment                 blocks=15517 elapsed=6.359s    number=2870553 hash="89d3a9…a12394"
INFO [09-11|15:53:55.699] Imported new chain segment               blocks=1749  txs=13052 mgas=401.234 elapsed=8.001s    mgasps=50.143  number=2963175 hash="dc6b06…3e4702" age=3y8mo3w   dirty=277.01MiB
INFO [09-11|15:53:56.600] Imported new chain segment               blocks=299   txs=2658  mgas=97.368  elapsed=901.205ms mgasps=108.042 number=2963474 hash="c6874a…07d2af" age=3y8mo3w   dirty=278.57MiB
INFO [09-11|15:54:04.613] Imported new chain segment               blocks=1504  txs=12519 mgas=384.873 elapsed=8.000s    mgasps=48.105  number=2964978 hash="3c53d0…3aa3ce" age=3y8mo2w   dirty=281.59MiB
INFO [09-11|15:54:06.760] Imported new chain segment               blocks=544   txs=4241  mgas=138.152 elapsed=2.146s    mgasps=64.349  number=2965522 hash="5f76c1…6dabb3" age=3y8mo2w   dirty=279.17MiB
INFO [09-11|15:54:06.761] Downloader queue stats                   receiptTasks=0 blockTasks=33035 itemSize=1.07KiB throttle=8192
INFO [09-11|15:54:14.772] Imported new chain segment               blocks=1802  txs=13459 mgas=554.213 elapsed=8.000s    mgasps=69.275  number=2967324 hash="849361…fc3e68" age=3y8mo2w   dirty=279.01MiB
INFO [09-11|15:54:15.386] Imported new chain segment               blocks=246   txs=1618  mgas=50.693  elapsed=614.090ms mgasps=82.550  number=2967570 hash="66be70…c87f45" age=3y8mo2w   dirty=279.29MiB
INFO [09-11|15:54:23.400] Imported new chain segment               blocks=1506  txs=14644 mgas=462.190 elapsed=8.001s    mgasps=57.765  number=2969076 hash="9066d4…19fe4e" age=3y8mo2w   dirty=280.13MiB
INFO [09-11|15:54:25.888] Imported new chain segment               blocks=542   txs=4758  mgas=154.290 elapsed=2.487s    mgasps=62.014  number=2969618 hash="00faf6…f8c995" age=3y8mo2w   dirty=278.85MiB
WARN [09-11|15:54:29.451] Synchronisation failed, dropping peer    peer=94b82260db33e63d err="retrieved hash chain is invalid: invalid mix digest"
ERROR[09-11|15:54:32.765] 
########## BAD BLOCK #########
Chain config: {ChainID: 1 Homestead: 1150000 DAO: 1920000 DAOSupport: true EIP150: 2463000 EIP155: 2675000 EIP158: 2675000 Byzantium: 4370000 Constantinople: 7280000 Petersburg: 7280000 Istanbul: 9069000, Muir Glacier: 9200000, YOLO v1: <nil>, Engine: ethash}

Number: 2970000
Hash: 0xec7fd8f3dc31efea22b238b19d872832e76d8f11ea9335e9ff33fa1fb6275773


Error: invalid mix digest
##############################
 
WARN [09-11|15:54:32.768] Synchronisation failed, dropping peer    peer=ff028cf0ab48a8d4 err="retrieved hash chain is invalid: invalid mix digest"
WARN [09-11|15:54:33.542] Multiple headers for single request      peer=29adb981d4269867 headers=192
WARN [09-11|15:54:33.542] Synchronisation failed, dropping peer    peer=29adb981d4269867 err="action from bad peer ignored: multiple headers (192) for single request"
ERROR[09-11|15:54:41.687] 

[holiman]

So, exactly which cache did you delete? Just want to be very explicit here, because the datadir and the ethash dir are not necessarily the same.

[holiman]

Hm, those are two different epochs, so I guess you did delete the proper thing. My next best guess would be that your drive is wonky, suffering from random corruptions. For reference, the issue I was referring to earlier: #20478

[holiman]

And here, one can see the exact steps I followed to find the corrupted bit: #20478 (comment)

[fernandezpaco]

I have changed the external disk's usb cable and now it is syncing...

[fernandezpaco]

thx a lot :)