upgrades: make sure that relaying works across upgrades

[erwanor]

Is your feature request related to a problem? Please describe.
We should test that ordinary chain state upgrades are not light-client breaking. This should be tested by pointing a relayer to a chain that undergoes a simple upgrade and make sure that there are no errors across the upgrade boundary.

Context: Upgrades that are light-client breaking should be rare, and require a special IBC mechanism (#3356). We want to make sure that we haven't missed something that would make ICS07 clients incapable of verifying our chain state post-upgrade.

[conorsch]

See #4087: we plan to exercise a chain upgrade Soon™. As long as we were careful to ensure that Hermes is running on the fresh chain pre-upgrade, we'll get clarity on whether we already have upgrade support for IBC by testing transfers after the upgrade, without making new channels.

[erwanor]

Great news, last friday, @avahowell and I paired on this. After a couple tries, we were able to perform a state breaking upgrade and relay UpdateClient messages across the migration boundary. This is an important milestone because it establish that our "ordinary" upgrade path is not IBC breaking (like we conjectured).

Hermes is able to relay across upgrades using a genesis-restart mode that requires running two full nodes: one for the legacy chain, and another for the upgraded chain: the procedure is outlined in this document and implemented via an internal load balancer for lightblock queries via AnyIo.

I will close this ticket in favor of two new ones:

• pd: implement a "stasis" mode to only serve RPCs
• upgrades: explore folding a lightblock window into the app state

The first one is about adding the ability to pause block production while keeping both pd and comet running (to serve RPCs). The second one is about brainstorming ways to reduce the operational overhead on relayers and make our upgrade process leaner.

[hdevalence]

pd: implement a "stasis" mode to only serve RPCs

This is already the case, no? pd will happily sit there and serve RPC requests without block production, we don't need to do anything?

upgrades: explore folding a lightblock window into the app state

I don't think we should do this, the cost/benefit of adding this additional scope is unclear. I think we can be comfortable with the status quo for other cosmos chains.

[avahowell]

This is already the case, no? pd will happily sit there and serve RPC requests without block production, we don't need to do anything?

Hermes needs the comet RPC for getting raw block headers, I don't think there's a way currently to run both comet and pd without producing blocks

[erwanor]

I think this can be done completely via comet config options, or so I hope. I tagged it as a pd matter as a loose category. Ideally, we would have pointers to safe config options that completely disable things like peer exchange, mempool, and producing/gossiping new blocks, but keeping the RPC and maybe the Info requests.

[hdevalence]

Ideally, we would have pointers to safe config options that completely disable things like peer exchange, mempool, and producing/gossiping new blocks,

But in the event of an upgrade, we know there won't be new blocks, or else the chain has faulted, so this seems like low benefit vs cost to configure/support.

[erwanor]

That's a good point, it wasn't the case in our testbed setup where we were effectively running a contentious fork to test this. Another reason to do this was to minimize network layer bandwidth churn, because for some reason, the p2p shell in comet does not penalize peers that repeatedly broadcast invalid blocks. But this hasn't been pinned to be an actual problem yet, so I agree we should default to not do this.