[Bug]: Panic when upgrading to v0.50 from v0.47.x

[aeryz]

Is there an existing issue for this?

• I have searched the existing issues

What happened?

Hey, we are testing our node that we recently upgraded to sdk version v0.50. When starting from scratch, the node is running without an issue but when we upgrade from 0.47.x to 0.50, our node panics with the following message:

panic: consensus key is nil: collections: not found: key 'no_key' of type github.com/cosmos/gogoproto/tendermint.types.ConsensusParams


 goroutine 1 [running]:
 github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).GetConsensusParams(_, {{0x432aa88, 0x5ff37e0}, {0x4345670, 0x
  github.com/cosmos/cosmos-sdk@v0.50.1/baseapp/baseapp.go:514 +0x147
 github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).internalFinalizeBlock(0xc00061e900, {0x432aa88, 0x5ff37e0}, 0
  github.com/cosmos/cosmos-sdk@v0.50.1/baseapp/abci.go:737 +0x747
 github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).FinalizeBlock(0xc00061e900, 0xc0016c3440)
  github.com/cosmos/cosmos-sdk@v0.50.1/baseapp/abci.go:871 +0xeb
 union/app.(*UnionApp).FinalizeBlock(0x43a2c5?, 0xc000da7d40?)
  union/app/app.go:1009 +0x16
 github.com/cosmos/cosmos-sdk/server.cometABCIWrapper.FinalizeBlock(...)
  github.com/cosmos/cosmos-sdk@v0.50.1/server/cmt_abci.go:44
 github.com/cometbft/cometbft/abci/client.(*localClient).FinalizeBlock(0xc001fd1fd8?, {0x432ae40?, 0x5ff37e0?}
  github.com/cometbft/cometbft@v0.38.1/abci/client/local_client.go:185 +0xd0
 github.com/cometbft/cometbft/proxy.(*appConnConsensus).FinalizeBlock(0xc0019afb78, {0x432ae40, 0x5ff37e0}, 0x
  github.com/cometbft/cometbft@v0.38.1/proxy/app_conn.go:104 +0x170
 github.com/cometbft/cometbft/state.(*BlockExecutor).ApplyBlock(_, {{{0xb, 0x0}, {0xc0015cfb38, 0x6}}, {0xc001
  github.com/cometbft/cometbft@v0.38.1/state/execution.go:213 +0x622
 github.com/cometbft/cometbft/consensus.(*Handshaker).replayBlock(_, {{{0xb, 0x0}, {0xc0015cfb38, 0x6}}, {0xc0
  github.com/cometbft/cometbft@v0.38.1/consensus/replay.go:534 +0x257
 github.com/cometbft/cometbft/consensus.(*Handshaker).ReplayBlocksWithContext(_, {_, _}, {{{0xb, 0x0}, {0xc001
  github.com/cometbft/cometbft@v0.38.1/consensus/replay.go:433 +0x79a
 github.com/cometbft/cometbft/consensus.(*Handshaker).HandshakeWithContext(0xc001fe5498, {0x432ad98, 0xc001976
  github.com/cometbft/cometbft@v0.38.1/consensus/replay.go:274 +0x478
 github.com/cometbft/cometbft/node.doHandshake({_, _}, {_, _}, {{{0xb, 0x0}, {0xc0015cfb38, 0x6}}, {0xc00161e8
  github.com/cometbft/cometbft@v0.38.1/node/setup.go:182 +0x1a9
 github.com/cometbft/cometbft/node.NewNodeWithContext({0x432ad98, 0xc001976870}, 0xc0011e0640, {0x430a710, 0xc
  github.com/cometbft/cometbft@v0.38.1/node/node.go:350 +0x5b7
 github.com/cosmos/cosmos-sdk/server.startCmtNode({0x432ad98, 0xc001976870}, 0xc0011e0640, {0x4368768?, 0xc000
  github.com/cosmos/cosmos-sdk@v0.50.1/server/start.go:389 +0x43f
 github.com/cosmos/cosmos-sdk/server.startInProcess(_, {{{0xc000c6f4b4, 0x6}, 0x0, {0xc000d5e570, 0x7}, {0x2fc
  github.com/cosmos/cosmos-sdk@v0.50.1/server/start.go:335 +0x188
 github.com/cosmos/cosmos-sdk/server.start(_, {{0x0, 0x0, 0x0}, {0x434c2b0, 0xc0007955c0}, 0x0, {0x0, 0x0}, {0
  github.com/cosmos/cosmos-sdk@v0.50.1/server/start.go:253 +0x2f3
 github.com/cosmos/cosmos-sdk/server.StartCmdWithOptions.func2.1()
  github.com/cosmos/cosmos-sdk@v0.50.1/server/start.go:173 +0x5e
 github.com/cosmos/cosmos-sdk/server.wrapCPUProfile(0xc001bc0c60, 0xc000d9f818)
  github.com/cosmos/cosmos-sdk@v0.50.1/server/start.go:587 +0x1b3
 github.com/cosmos/cosmos-sdk/server.StartCmdWithOptions.func2(0xc0019e0200?, {0xc000e1e9a0?, 0x0?, 0xe?})
  github.com/cosmos/cosmos-sdk@v0.50.1/server/start.go:172 +0x212
 github.com/spf13/cobra.(*Command).execute(0xc0019e3800, {0xc000e1e8c0, 0xe, 0xe})
  github.com/spf13/cobra@v1.8.0/command.go:983 +0xabc
 github.com/spf13/cobra.(*Command).ExecuteC(0xc0016cf800)
  github.com/spf13/cobra@v1.8.0/command.go:1115 +0x3ff
 github.com/spf13/cobra.(*Command).Execute(...)
  github.com/spf13/cobra@v1.8.0/command.go:1039
 github.com/spf13/cobra.(*Command).ExecuteContext(...)
  github.com/spf13/cobra@v1.8.0/command.go:1032
 github.com/cosmos/cosmos-sdk/server/cmd.Execute(0xc0016cf800?, {0x0, 0x0}, {0xc00115cdc0, 0x6})
  github.com/cosmos/cosmos-sdk@v0.50.1/server/cmd/execute.go:33 +0x12a

We followed the migration guide. Also we have fixed the problem by setting the ConsensusParam in FinalizeBlock in a very hacky way but this shouldn't be the solution. I'm thinking that we are missing something but couldn't find the cause.

Cosmos SDK Version

0.50

How to reproduce?

Run a v0.47 chain, then upgrade to v0.50.

[julienrbrt]

Can you try with the latest commit from release/v0.50.x? We have a fix supporting for empty consensus params for v0.50.2.

[aeryz]

Oh, based on the change that I see, it will fix our issue but will try and get back to you on that. Thanks!

[aeryz]

Btw, I see this comment in the fix:

		// This could happen while migrating from v0.45/v0.46 to v0.50, we should
		// allow it to happen so during preblock the upgrade plan can be executed
		// and the consensus params set for the first time in the new format.

The comment says that this could happen when migrating from v0.45/v0.46 but we were previously on v0.47. Is it normal to have this issue?

[julienrbrt]

If you didn't migrate your consensus params in v0.47 (https://github.com/cosmos/cosmos-sdk/blob/release/v0.47.x/UPGRADING.md#xconsensus), then yes 😅.
Otherwise, I'd have to see your code to be able to tell.

[aeryz]

It worked! Thanks again :)