Software upgrade fails from v0.45.4 to v0.46.0-rc1

[kaustubhkapatral]

Summary of Bug

While testing the upgrade path from v0.45.x to v0.46-0-rc1, node crashes with a tendermint error before applying the upgrade handler.

7:01AM INF ABCI Handshake App Info hash="\b\x11\x17�%L\x13gY�\x19v\x0e�.ء|w�X���\uea72,\x10I��" height=269 module=consensus protocol-version=0
 software-version=0.46.0-rc1
7:01AM INF ABCI Replay Blocks appHeight=269 module=consensus stateHeight=0 storeHeight=0
panic: state.AppHash does not match AppHash after replay. Got
081117C0254C136759D719760E842ED8A17C779C58B6C7C0EEA9B22C1049F7A1, expected .

State: {{{11 0} 0.35.0-unreleased} test 1 0 :0:000000000000 2022-05-24 06:09:21.547165957 +0000 UTC ValidatorSet{
  Proposer: nil-Validator
  Validators:

} ValidatorSet{
  Proposer: nil-Validator
  Validators:

} ValidatorSet{
  Proposer: nil-Validator
  Validators:

} 1 {{22020096 -1} {100000 48h0m0s 1048576} {[ed25519]} {0}} 1 [] []}

Did you reset Tendermint without resetting your application's data?

goroutine 1 [running]:
github.com/tendermint/tendermint/internal/consensus.assertAppHashEqualsOneFromState({_, _, _}, {{{0xb, 0x0}, {0x201a9d5, 0x11}}, {0xc0008acb0c, 0x4}, 0x1, ...})

Upgrade height during testing was kept at 270. Once the binary of v0.46.0-rc1 was built and node was restarted, it crashed during the replay of the previous block. It did not corrupt the db of the node as the upgrade handler was not applied and could be skipped by rolling back to v0.45.4 and using --unsafe-skip-upgrades <height>.

Version

v0.46.0-rc1

Steps to Reproduce

• Start a local node with v0.45.4 and submit a software upgrade proposal with the name v045-to-v046.
• Once the node stops at the upgrade height, build the binary from v0.46.0-rc1 and restart the node.

---

For Admin Use

• Not duplicate issue
• Appropriate labels applied
• Appropriate contributors tagged
• Contributor assigned/self-assigned

[amaury1093]

I confirm I ran into the same issue while doing a localnet upgrade.

[amaury1093]

@kaustubhkapatral See https://github.com/tendermint/tendermint/blob/master/UPGRADING.md#database-key-format-changes, you need to migrate the TM db keys first.

I created #12028 to add this command to the SDK.