Snapshot generated during a ledger rekey is invalid

[jumaffre]

I have been chasing this bug for a while in #3573.

The generation of a snapshot is two-step. First, a snapshot of the store at the snapshot seqno is taken and the snapshot generation is actually done later, asynchronously. However, if a ledger rekey happens in between the two, the ledger secret used to encrypt the snapshot will be the one of after the rekey. When a new node joins from this snapshot, it will try to (correctly!) use the ledger secret applicable at the seqno at which the snapshot was generated, but the decryption of the snapshot will fail with the following error:

2022-04-27T13:29:01.373724Z -0.007 0   [info ] ../src/node/snapshot_serdes.h:192    | Deserialising snapshot (size: 89488, public only: false)
2022-04-27T13:29:01.376271Z -0.010 0   [fail ] ../src/kv/store.h:416                | Initialisation of deserialise object failed
2022-04-27T13:29:01.376373Z -0.010 0   [debug] ../src/http/http_endpoint.h:92       | Error parsing HTTP request: Failed to apply snapshot: 10

The issue is due to the fast path on the selection of a ledger secret used to encrypt ledger entries: we assume that entries are always encrypted in the global order on the primary node, but it isn't the case for snapshots that are encrypted "retroactively".

[achamayou]

While this is an annoying bug we want to fix, it should not require an API change and so doesn't need to block 2.0 final.