[Bug]: Chain upgrade from v0.52 fails in system tests

[alpe]

Is there an existing issue for this?

• I have searched the existing issues

What happened?

panic: failed to load latest version: failed to load store: initial version set to 22, but found earlier version 1 [cosmossdk.io/store@v1.1.1/rootmulti/store.go:256]

full stacktrace

cosmossdk.io/simapp.NewSimApp({0x10397a2f0, 0x14001418380}, {0x10c8c2a18, 0x14001090010}, {0x0, 0x0}, 0x1, {0x103946238, 0x140016ac380}, {0x14001418000, ...})
	cosmossdk.io/simapp/app_di.go:285 +0x9c4
cosmossdk.io/simapp/simd/cmd.newApp({0x10397a2f0, 0x14001418380}, {0x10c8c2a18, 0x14001090010}, {0x0, 0x0}, {0x103946238, 0x140016ac380})
	cosmossdk.io/simapp/simd/cmd/commands.go:124 +0x78
github.com/cosmos/cosmos-sdk/server.startApp[...](0x140016923a0, 0x10391e2e8, {0x10391e338, 0x0, 0x0, 0x0, 0x140013691d0})
	github.com/cosmos/cosmos-sdk@v0.53.0/server/start.go:650 +0x250
github.com/cosmos/cosmos-sdk/server.start[...](_, {{0x10399a0f0, 0x14001696600}, 0x0, {0x0, 0x0}, {0x1039ba128, 0x1400124b4a0}, {0x1039935e0, 0x140011a7700}, ...}, ...)
	github.com/cosmos/cosmos-sdk@v0.53.0/server/start.go:226 +0xd8
github.com/cosmos/cosmos-sdk/server.StartCmdWithOptions[...].func2.1()
	github.com/cosmos/cosmos-sdk@v0.53.0/server/start.go:201 +0x68
github.com/cosmos/cosmos-sdk/server.wrapCPUProfile(0x140016923a0, 0x14000b11920)
	github.com/cosmos/cosmos-sdk@v0.53.0/server/start.go:600 +0x16c
github.com/cosmos/cosmos-sdk/server.StartCmdWithOptions[...].func2({0x1027b3650, 0x4?, 0x1027b3464?})
	github.com/cosmos/cosmos-sdk@v0.53.0/server/start.go:200 +0x1ac
github.com/spf13/cobra.(*Command).execute(0x14001384908, {0x14001696240, 0x3, 0x3})
	github.com/spf13/cobra@v1.8.1/command.go:985 +0x834
github.com/spf13/cobra.(*Command).ExecuteC(0x14001249b08)
	github.com/spf13/cobra@v1.8.1/command.go:1117 +0x344
github.com/spf13/cobra.(*Command).Execute(...)
	github.com/spf13/cobra@v1.8.1/command.go:1041
github.com/spf13/cobra.(*Command).ExecuteContext(...)
	github.com/spf13/cobra@v1.8.1/command.go:1034
github.com/cosmos/cosmos-sdk/server/cmd.Execute(0x14001249b08, {0x0, 0x0}, {0x14001121800, 0x12})
	github.com/cosmos/cosmos-sdk@v0.53.0/server/cmd/execute.go:40 +0x31c
main.main()
	cosmossdk.io/simapp/simd/main.go:16 +0x44

Cosmos SDK Version

main

How to reproduce?

Use TestChainUpgrade with a local v0.52 binary

git checkout release/v0.52.x
make build
cp build/simd tests/systemtests/binaries/simd_v0.52

git checkout alex/system_TestQueryLatestValidatorSet # or main when merged
# activate test

cd tests/systemtests
go test -v -mod=readonly -failfast -tags='system_test' --run TestChainUpgrade     ./...

[julienrbrt]

Additionally there's a issue with the docker image for 0.50 upgrade to 0.52 in system tests: https://github.com/cosmos/cosmos-sdk/actions/runs/11271625178/job/31344832950

[julienrbrt]

Additionally there's a issue with the docker image for 0.50 upgrade to 0.52 in system tests: https://github.com/cosmos/cosmos-sdk/actions/runs/11271625178/job/31344832950

This part looks fixed by #22433