Error during handshake: error on replay: wrong Block.Header.AppHash. Expected DD1BA... , got BE03...

[kenorb]

While started a testnet node with sync from scratch:

$ fxcored start --log_level trace --trace
...
3:02PM INF starting ABCI with Tendermint
3:02PM INF Starting multiAppConn service impl=multiAppConn module=proxy
3:02PM INF Starting localClient service connection=query impl=localClient module=abci-client
3:02PM INF Starting localClient service connection=snapshot impl=localClient module=abci-client
3:02PM INF Starting localClient service connection=mempool impl=localClient module=abci-client
3:02PM INF Starting localClient service connection=consensus impl=localClient module=abci-client
3:02PM INF Starting EventBus service impl=EventBus module=events
3:02PM INF Starting PubSub service impl=PubSub module=pubsub
3:02PM INF Starting IndexerService service impl=IndexerService module=txindex
3:02PM INF ABCI Handshake App Info hash="�\x1b��\x01��ӷ�#������Cٴ흑%��\x05T��Q^" height=373101 module=consensus protocol-version=0 software-version=
3:02PM INF ABCI Replay Blocks appHeight=373101 module=consensus stateHeight=373101 storeHeight=373102
3:02PM INF Replay last block using real app module=consensus
Error: error during handshake: error on replay: wrong Block.Header.AppHash.  Expected DD1BAFC20187A9D3B7FB239693AE8CE6D043D9B4ED9D912588F905548D99515E, got BE037FED818D28340E99DC38B8B0EF11B120C7A7ECC2A7E587D5AA158A849681

Can you try to upgrade tendermint to the latest? Maybe it's going to help.

As for the workaround, I can only try to re-do the sync again.

Related: cosmos/cosmos-sdk#10281

--

I've also tried from the snaphot (2021-11-08), similar error:

6:54PM INF executed block height=1830584 module=state num_invalid_txs=0 num_valid_txs=0
6:54PM INF commit synced commit=436F6D6D697449447B5B31333020343520323035203130392037312031392031323020353820363620323132203133362031323420342031303220313636203134302031323820323133203837203134342031383420313838203138312031333220313037203239203235312031383920323233203131352034302039385D3A3142454542387D
6:54PM INF committed state app_hash=822DCD6D4713783A42D4887C0466A68C80D55790B8BCB5846B1DFBBDDF732862 height=1830584 module=state num_txs=0
6:54PM INF indexed block height=1830584 module=txindex
panic: Failed to process committed block (1830585:332EB9767DB8F86636B6881A73D8701B313B6D9F64046BB64DEB20D3232D0FE8): wrong Block.Header.AppHash.  Expected 822DCD6D4713783A42D4887C0466A68C80D55790B8BCB5846B1DFBBDDF732862, got DAA4A98A1248CF5F3A5D37FEC87B1E4E56FC744FB6D3EED71BB857FDC2A1B90C

goroutine 56 [running]:
github.com/tendermint/tendermint/blockchain/v0.(*BlockchainReactor).poolRoutine(0xc000cf8fc0, 0x0)
	github.com/tendermint/tendermint@v0.34.9/blockchain/v0/reactor.go:401 +0x1265
created by github.com/tendermint/tendermint/blockchain/v0.(*BlockchainReactor).OnStart
	github.com/tendermint/tendermint@v0.34.9/blockchain/v0/reactor.go:110 +0x85

---

Another one after re-run from scratch:

10:39PM INF Starting IndexerService service impl=IndexerService module=txindex
10:39PM INF ABCI Handshake App Info hash="��ơL<���\x1b\x02�J!��7�P\"�$C�O���'\x17b�" height=1397101 module=consensus protocol-version=0 software-version=
10:39PM INF ABCI Replay Blocks appHeight=1397101 module=consensus stateHeight=1397101 storeHeight=1397102
10:39PM INF Replay last block using real app module=consensus
Error: error during handshake: error on replay: wrong Block.Header.AppHash.  Expected F0D2C6A14C3CBCBFE01B02984A21C8F637AC5022802443974FAAA8F127176297, got 48A1C3DE946F7EF5FC490104887E12FC9D737356581764AFB552417A2CEF1BD8

[zakir-code]

I am very sorry for the inconvenience. Due to the difference in the upgrade block height between the fxCore mainnet and the testnet, they do not use the same fxcored program. You can check the fxcored corresponding network currently in use through the fxcored network command. Here is more detail , We are actively improving and looking forward to your feedback.

If you expect to run the testnet, please use make install-testnet to install the fxCore testnet program, and then resynchronize the data or use the testnet snapshot data.

[kenorb]

If you expect to run the testnet, please use make install-testnet to install the fxCore testnet program, and then resynchronize the data or use the testnet snapshot data.

All of above was done for testnet using container. Using snapshot data and from scratch. Each time failing at random times (at different block heights). I can try using non-container version if that makes any difference.

When using snapshot data, it fails with different error:

failed to load latest version: failed to load store: initial version set to 2062000, but found earlier version 1788900

so I think the version is mismatched. Maybe the snapshot was done using earlier version? Btw where this version is defined, so I can check which I have?

[kenorb]

If you expect to run the testnet, please use make install-testnet to install the fxCore testnet program, and then resynchronize the data or use the testnet snapshot data.

Which branch has the latest testnet changes? master is a bit behind from testnet branch and they've different seeds.

[zakir-code]

All of above was done for testnet using container. Using snapshot data and from scratch. Each time failing at random times (at different block heights). I can try using non-container version if that makes any difference.

We are very sorry, we submitted the wrong testnet docker image, we have fixed it, please re-acquire functionx/fx-core:testnet-1.0 sha256:bfa8d96a563d2ef797dd1afed84c517768aef5a671597d9bc4c4f5c8b77f92af

[zakir-code]

so I think the version is mismatched. Maybe the snapshot was done using earlier version? Btw where this version is defined, so I can check which I have?

Because we have upgraded the testnet many times, resynchronization will check the fork as soon as possible. You can use the fxcored network or docker run --rm functionx/fx-core:testnet-1.0 network command to check the network and the upgrade height, for example:

ubuntu@ip-192-168-69-144:~/fx-core$ fxcored network
CrossChainSupportBscBlock: "1079000"
CrossChainSupportPolygonBlock: "1788888"
CrossChainSupportTronBlock: "1788888"
GravityPruneValsetsAndAttestationBlock: "373100"
GravityValsetSlashBlock: "1380300"
Network: testnet

[zakir-code]

Which branch has the latest testnet changes? master is a bit behind from testnet branch and they've different seeds.

The master branch is the latest, and the testnet branch is to be compatible with the old developer documentation. The seed nodes provided by both versions are available. After a period of time, we will abandon the testnet branch. I’m sorry to bring it to you trouble.

[kenorb]

No worries, good the problems has been discovered, I'll re-test now using the fixed container.

[kenorb]

I'm going to close this one, as the above errors could be related to the issues with testnet container (as per GH-11).

I've now problem that other peers doesn't want to talk to my peer which is causing i/o timeouts and the data doesn't sync anymore. Reported at GH-10.