Nil Panic in isNextProposer During ReplayBlocks with evm-build-optimistic Enabled

[zsystm]

Description and context

A panic occurs due to cmtAPI being nil in the isNextProposer function during the ReplayBlocks process. This issue arises for any node that is behind the latest state and has the evm-build-optimistic option set to true.

panic flow:ReplayBlocksWithContext --> FinalizeBlock --> PostFinalize

how cmtAPI can be nil?

• cmtAPI is set after newCometNode is called (ref. code)
• ReplayBlocks is done during newCometNode

panic log:

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0x1ea80ea]
goroutine 1 [running]:
github.com/piplabs/story/client/x/evmengine/keeper.(*Keeper).isNextProposer(0xc00077eb40, {0x37403d0?, 0xc001189888?}, {0xc005c074a0, 0x14, 0x748594f01a68?}, 0x20?)
        /home/ubuntu/story/client/x/evmengine/keeper/keeper.go:175 +0x4a
github.com/piplabs/story/client/x/evmengine/keeper.(*Keeper).PostFinalize(_, {{0x3740210, 0x51c88e0}, {0x3758270, 0xc00889cc00}, {{0x0, 0x0}, {0x0, 0x0}, 0x98f8f, ...}, ...})
        /home/ubuntu/story/client/x/evmengine/keeper/abci.go:177 +0x365
github.com/piplabs/story/client/app.abciWrapper.FinalizeBlock({{0x375c738?, 0xc002f322d0?}, 0xc002f322e0?, 0xc002f322f0?}, {0x3740328, 0x51c88e0}, 0xc000936480)
        /home/ubuntu/story/client/app/abci.go:110 +0x2fa
github.com/cometbft/cometbft/abci/client.(*localClient).FinalizeBlock(0x3758be8?, {0x3740328?, 0x51c88e0?}, 0x74854d6ff828?)
        /home/ubuntu/goApps/pkg/mod/github.com/cometbft/cometbft@v0.38.9/abci/client/local_client.go:185 +0xcd
github.com/cometbft/cometbft/proxy.(*appConnConsensus).FinalizeBlock(0xc0020f9080, {0x3740328, 0x51c88e0}, 0xc000936480)
        /home/ubuntu/goApps/pkg/mod/github.com/cometbft/cometbft@v0.38.9/proxy/app_conn.go:104 +0x170
github.com/cometbft/cometbft/state.(*BlockExecutor).applyBlock(_, {{{0xb, 0x0}, {0xc00577a0d8, 0x6}}, {0xc00577a110, 0x7}, 0x1, 0x98f8e, {{0xc004b36440, ...}, ...}, ...}, ...)
        /home/ubuntu/goApps/pkg/mod/github.com/cometbft/cometbft@v0.38.9/state/execution.go:224 +0x525
github.com/cometbft/cometbft/state.(*BlockExecutor).ApplyBlock(_, {{{0xb, 0x0}, {0xc00577a0d8, 0x6}}, {0xc00577a110, 0x7}, 0x1, 0x98f8e, {{0xc004b36440, ...}, ...}, ...}, ...)
        /home/ubuntu/goApps/pkg/mod/github.com/cometbft/cometbft@v0.38.9/state/execution.go:219 +0x1d9
github.com/cometbft/cometbft/consensus.(*Handshaker).replayBlock(_, {{{0xb, 0x0}, {0xc00577a0d8, 0x6}}, {0xc00577a110, 0x7}, 0x1, 0x98f8e, {{0xc004b36440, ...}, ...}, ...}, ...)
        /home/ubuntu/goApps/pkg/mod/github.com/cometbft/cometbft@v0.38.9/consensus/replay.go:534 +0x24e
github.com/cometbft/cometbft/consensus.(*Handshaker).ReplayBlocksWithContext(_, {_, _}, {{{0xb, 0x0}, {0xc00577a0d8, 0x6}}, {0xc00577a110, 0x7}, 0x1, ...}, ...)
        /home/ubuntu/goApps/pkg/mod/github.com/cometbft/cometbft@v0.38.9/consensus/replay.go:433 +0x77a
github.com/cometbft/cometbft/consensus.(*Handshaker).HandshakeWithContext(0xc00b90d988, {0x3740328, 0x51c88e0}, {0x375ccb8, 0xc000c57b20})
        /home/ubuntu/goApps/pkg/mod/github.com/cometbft/cometbft@v0.38.9/consensus/replay.go:274 +0x425
github.com/cometbft/cometbft/node.doHandshake({_, _}, {_, _}, {{{0xb, 0x0}, {0xc00577a0d8, 0x6}}, {0xc00577a110, 0x7}, ...}, ...)
        /home/ubuntu/goApps/pkg/mod/github.com/cometbft/cometbft@v0.38.9/node/setup.go:182 +0x1a9
github.com/cometbft/cometbft/node.NewNodeWithContext({0x3740328, 0x51c88e0}, 0xc0011b4a10, {0x3728d40, 0xc000c5c8c0}, 0xc002f32290, {0x3712ea0, 0xc002e9acc0}, 0xc001c0e7b8, 0x30ca3f8, ...)
        /home/ubuntu/goApps/pkg/mod/github.com/cometbft/cometbft@v0.38.9/node/node.go:359 +0x5b7
github.com/cometbft/cometbft/node.NewNode(0xa4ece6ec5fae1e46?, {0x3728d40?, 0xc000c5c8c0?}, 0xb98d0bca04934454?, {0x3712ea0?, 0xc002e9acc0?}, 0xdc660575d42bd527?, 0xaf20fe9141232677?, 0x9e06ade9e6bddb00?, {0x3740408, ...}, ...)
        /home/ubuntu/goApps/pkg/mod/github.com/cometbft/cometbft@v0.38.9/node/node.go:279 +0xa9
github.com/piplabs/story/client/app.newCometNode({0x3740248, 0xc000a08ae0}, 0xc0011b4a10, 0xc000577408, {0x3728d40, 0xc000c5c8c0})
        /home/ubuntu/story/client/app/start.go:207 +0x49f
github.com/piplabs/story/client/app.Start({_, _}, {{{0xc001116380, 0x19}, {0xc000e7bda0, 0x5}, {0x0, 0x0}, {0xc0009146c0, 0x2d}, ...}, ...})
        /home/ubuntu/story/client/app/start.go:133 +0x625
github.com/piplabs/story/client/app.Run({_, _}, {{{0xc001116380, 0x19}, {0xc000e7bda0, 0x5}, {0x0, 0x0}, {0xc0009146c0, 0x2d}, ...}, ...})
        /home/ubuntu/story/client/app/start.go:56 +0x79
github.com/piplabs/story/client/cmd.newRunCmd.func1(0xc001114308, {0x2d00314?, 0x4?, 0x2d00330?})
        /home/ubuntu/story/client/cmd/cmd.go:51 +0x225
github.com/spf13/cobra.(*Command).execute(0xc001114308, {0x51c88e0, 0x0, 0x0})
        /home/ubuntu/goApps/pkg/mod/github.com/spf13/cobra@v1.8.0/command.go:983 +0xaca
github.com/spf13/cobra.(*Command).ExecuteC(0xc00113cf08)
        /home/ubuntu/goApps/pkg/mod/github.com/spf13/cobra@v1.8.0/command.go:1115 +0x3ff
github.com/spf13/cobra.(*Command).Execute(...)
        /home/ubuntu/goApps/pkg/mod/github.com/spf13/cobra@v1.8.0/command.go:1039
github.com/spf13/cobra.(*Command).ExecuteContext(...)
        /home/ubuntu/goApps/pkg/mod/github.com/spf13/cobra@v1.8.0/command.go:1032
github.com/piplabs/story/lib/cmd.Main(0xc00113cf08)
        /home/ubuntu/story/lib/cmd/cmd.go:34 +0xcf
main.main()
        /home/ubuntu/story/client/main.go:10 +0x18
iliad.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
iliad.service: Failed with result 'exit-code'.
iliad.service: Consumed 1.145s CPU time.
iliad.service: Scheduled restart job, restart counter is at 8.

Steps to reproduce

1. Stop a node for a period of time.
2. Set evm-build-optimistic to true
3. Restart the node to allow it to fetch blocks from other peers and replay them to sync to the latest state.

Experienced behavior

A nil panic occurs during block synchronization and replay when the evm-build-optimistic option is enabled.

Expected behavior

No nil panic should occur during block synchronization and replay, even when the evm-build-optimistic option is enabled.

Solution recommendation

Implement a nil check similar to the approach used in Omni.