BAD BLOCK with version v1.1.12

[stonetown1975]

I have installed new version and it was running fine for a day. On block 20,023,433 it stopped syncing with BAD BLOCK message and can not move forward.

it was running with -diffsync -rpc.allow-unprotected-txs --txlookuplimit 0

message from bsc.log

t=2022-08-01T05:00:02+0000 lvl=info msg="Skip duplicated bad block" number=20,023,433 hash=0xf6b9492dce5716f85dc92e3ee9b5da80a15f79b13a8b6d401b135f56a4e75959
t=2022-08-01T05:00:02+0000 lvl=eror msg="\n########## BAD BLOCK #########\nChain config: {ChainID: 56 Homestead: 0 DAO: DAOSupport: false EIP150: .

.......
t=2022-08-01T05:00:15+0000 lvl=warn msg="Synchronisation failed, dropping peer" peer=2b7cbface401635a0164e6629ad02b6d5999d91297e429006c19788087098513 err="retrieved hash chain is invalid: expected tx hash 0x204ebcd85a33b1adde4677b28e41dbc950717
6b9fc16691dc5600cacb33dadf0, get 0x8d4920c073a5b1b26fa3fd59c85264cd246311a8db4596532f56aa2a13f8cc50, nonce 715087, to 0x0000000000000000000000000000000000001000, value 99221563210686458, gas 9223372036854775807, gasPrice 0, data f340fa0100000000
0000000000000000e9ae3261a475a27bb1028f140bc2a7c843318afd"

I have tried solution recommended in #628. I was trying to restart it with --snapshot=false and then with --snapshot=true. It did not help. Any suggestion how to fix it?

Any technical discord forum related to BSC? The invite https://discord.com/invite/binancesmartchain does not work anymore.

[forcodedancing]

@stonetown1975 Can you share the full command you are using, as well as the related logs?

For joining discord forum, please use this link: https://discord.com/invite/z2VpC455eU

[stonetown1975]

full command:

/d1/bsc/geth --config /d1/bsc/config.toml --datadir /d1/bsc/node --diffsync --cache=16000 --rpc.allow-unprotected-txs --txlookuplimit=0 --http.addr 192.168.1.8 --ws.addr 192.168.1.8

fragment of bsc.log

=2022-08-01T05:00:02+0000 lvl=info msg="Skip duplicated bad block" number=20,023,433 hash=0xf6b9492dce5716f85dc92e3ee9b5da80a15f79b13a8b6d401b135f56a4e75959
t=2022-08-01T05:00:02+0000 lvl=eror msg="\n########## BAD BLOCK #########\nChain config: {ChainID: 56 Homestead: 0 DAO: DAOSupport: false EIP150: .

.......
t=2022-08-01T05:00:15+0000 lvl=warn msg="Synchronisation failed, dropping peer" peer=2b7cbface401635a0164e6629ad02b6d5999d91297e429006c19788087098513 err="retrieved hash chain is invalid: expected tx hash 0x204ebcd85a33b1adde4677b28e41dbc950717
6b9fc16691dc5600cacb33dadf0, get 0x8d4920c073a5b1b26fa3fd59c85264cd246311a8db4596532f56aa2a13f8cc50, nonce 715087, to 0x0000000000000000000000000000000000001000, value 99221563210686458, gas 9223372036854775807, gasPrice 0, data f340fa0100000000
0000000000000000e9ae3261a475a27bb1028f140bc2a7c843318afd"

[stonetown1975]

FYI, I used snapshot geth-snapshot (db size 554.35 gb, 566.15 gb after decompression) from https://github.com/BNB48Club/bsc-snapshots. Can it be the problem? It was able to get in sync and worked fine for a day.

[forcodedancing]

@stonetown1975 We tired your command in our node and cannot observe the issue as you encountered now. To bring your node back, can you try the following options:

1. remove --diffsync in your command, OR,
2. revert blocks using ./geth --datadir /d1/bsc/node --exec 'debug.setHead("0x"+(eth.blockNumber - 15).toString(16))' attach - 15 is the number of blocks you want to revert

Can you have a try to see it works or not?

[Blackcol]

@stonetown1975 It happened to me too, I downloaded the same snapshot and when bad block messages appeared I was using snapshot=true, then gracefully shutdown, start again using snapshot=false and now its working ok..

[stonetown1975]

@Blackcol snapshot=true and then snapshot=false helped !!! thanks a lot for help. now it is syncing

[stonetown1975]

the issue is resolved

[stonetown1975]

it happened again. new bad block at 20,047,357

t=2022-08-02T00:24:09+0000 lvl=info msg="Skip duplicated bad block" number=20,047,357 hash=0x3ce54642aff4b0796be8e0d1a8de560aa13556afb3be150df0b9a6c535f50121
t=2022-08-02T00:24:09+0000 lvl=eror msg="\n########## BAD BLOCK #########\nChain config: {ChainID: 56 Homestead: 0 DAO: DAOSupport: false EIP150: 0 EIP155: 0 EIP158: 0 Byzantium: 0 Constantinople: 0 Petersburg: 0 Istanbul: 0, Muir Glacier: 0, Ramanujan: 0, Niels: 0, MirrorSync: 5184000, Bruno: 13082000, Berlin: , YOLO v3: , CatalystBlock: , London: , ArrowGlacier: , MergeFork:, Euler: 18907621, Engine:

.....
t=2022-08-02T00:24:09+0000 lvl=warn msg="Synchronisation failed, dropping peer" peer=d3d8b72b55385fb27d2ac7f1ae506b4e177f25a32fa052a2555f8a5c74fd8197 err="retrieved hash chain is invalid: invalid merkle root (remote: 9c15ec35443848a0911f1ad8fb2287a3a0b1cafc561a5e864ecdedbd5fa5dbcc local: a7a8f4438f78aab0bd107ff334e1d5bb878e3e1f9bce0e5a9e1f23e8c704464c)"
t=2022-08-02T00:24:15+0000 lvl=warn msg="Synchronisation failed, dropping peer" peer=83e978047228375ea6bf4304655a7c04c252978761c0ff3593b4eb3f9d96f425 err=timeout

....

t=2022-08-02T00:51:31+0000 lvl=eror msg="validate state failed" error="invalid merkle root (remote: 9c15ec35443848a0911f1ad8fb2287a3a0b1cafc561a5e864ecdedbd5fa5dbcc local: a7a8f4438f78aab0bd107ff334e1d5bb878e3e1f9bce0e5a9e1f23e8c704464c)"
t=2022-08-02T00:51:31+0000 lvl=info msg="Skip duplicated bad block" number=20,047,357 hash=0x3ce54642aff4b0796be8e0d1a8de560aa13556afb3be150df0b9a6c535f50121
t=2022-08-02T00:51:31+0000 lvl=eror msg="\n########## BAD BLOCK #########\nChain config: {ChainID: 56 Homestead: 0 DAO: DAOSupport: false EIP150: 0 EIP155: 0 EIP158: 0 Byzantium: 0 Constantinople: 0 Petersburg: 0 Istanbul: 0, Muir Glacier: 0, Ramanujan: 0

full command

/d1/bsc/geth --config /d1/bsc/config.toml --datadir /d1/bsc/node --nat extip:xxx.xxx.xxx.xxx --cache=32000 --rpc.allow-unprotected-txs --txlookuplimit=0 --http.addr 192.168.1.8 --ws.addr 192.168.1.8 --pruneancient=true --syncmode=full

Any idea why it started happening with new version? and how to fix it?

[stonetown1975]

I tried to rewind back by ./geth --datadir /d1/bsc/node --exec 'debug.setHead("0x"+(eth.blockNumber - 100).toString(16))' and it successfully processed 100 blocks until it hit block 20,047,357. Then it stuck again. Trick with snapshot=true and then snapshot=false does not help anymore

state: \n\n\nError: invalid merkle root (remote: 9c15ec35443848a0911f1ad8fb2287a3a0b1cafc561a5e864ecdedbd5fa5dbcc local: a7a8f4438f78aab0bd107ff334e1d5bb878e3e1f9bce0e5a9e1f23e8c704464c)\n##############################\n"

[stonetown1975]

Should I downgrade to v1.1.11?

[Blackcol]

Maybe db is already corrupted, you should start syncing from snapshot again with snapshot=false from the start, debug.setHead didn't work for me too..

[forcodedancing]

@stonetown1975 Can you try the recent snapshot to see whether it works or not ? https://github.com/bnb-chain/bsc-snapshots

[stonetown1975]

Got synced from latest official snapshot https://download.bsc-snapshot.workers.dev/geth-20220802.tar.lz4. Let's see how it will go ..

[stonetown1975]

after sync from official snapshot everything works fine for 2 days already. Only errors in log file are like
t=2022-08-04T17:14:15+0000 lvl=eror msg="do light process err at block" num=20,152,043 err="account storage root mismatch" but it is expected according to #672