pd: test bootstrapping a new node using an out-of-band snapshot

[erwanor]

After a chain upgrade, cometbft resets its block store and starts a new chain beginning at the height specified in the genesis file (see #3451 for context). This is problematic because penumbra full nodes use cometbft's block sync as a replay-log. Nodes bootstrap by reconstructing the chain state since genesis, executing state transitions, until they have reached the tip of the chain.

We need to design around this limitation. There are two strategies that we could employ:

1. rely entirely on off-chain db snapshot distributions

Post-migration, validators and node operators make a snapshot of the state available via public channels. This is far from ideal, but match the reality of how production cosmos chains handle post-upgrade bootstrapping. For example, see quicksync.io, osmosis snapshots, polkachu's snapshot page, cosmos snapshot directory, or lavenderfive snapshots etc.

2. build an in-band state sync mechanism

We're not going to do that for the foreseeable future.

We need to test that a node is able to join a post-upgrade network using a hosted state snapshot.

[conorsch]

See details in #3081: we tested this implicitly by performing the upgrade procedure, and successfully restarting a node using the migrated local state. We didn't explicitly write a plan or tooling about how or where such snapshots should be hosted, but that's not a problem we need to resolve now. We've verified that the upgrade tooling is adequate to allow the network to continue post-upgrade.