state sync spike to put swing-store data in IAVL tree on periodic basis

[warner]

In the last meeting we sketched out some options for state-sync experiments (#5542, #3769). This ticket is to track the spike that we talked about, where we'd put the swing-store contents into the IAVL tree, one of:

• at the end of every block
• after every N'th block (maybe targetting about once a day), with some durable swing-store -side tracking of what changes need to be applied at time N
• after every N'th block we just overwrite the whole IAVL sub-space with a fresh copy of the swing-store data (erasing all the old bits)

[mhofman]

My understanding of state-sync is that the state needs to be potentially available for every block, requiring a write at the end of each block. Any side tracking + diff would likely be more appropriate for an approach where only the hash of external blobs are saved in the IAVL tree, as potentially described in #5542 (comment).

Given the potential performance impact of writing all Swingset state to the IAVL tree, we first need to measure with good granularity the impact it would have.

The plan is to:

1. Implement a mechanism to generate a dump of the changes made to SwingSet state during the block: API to get summary of swingstore block changes #6562. This provides a baseline of the performance.
2. Add an after commit event / action to measure time spend by go to commit the block: After-commit block action #6563. This allows to measure our baseline (time from endblock-finish to after-commit)
3. Plumb the dump of changes crudely as an end_block result field (alongside queue depth data), and replicate each write/delete into the IAVL tree on the go side.
4. Measure impact and make decision on path forward. We'll want to make sure this includes a variety of blocks, including ones where XS snapshots are generated / deleted. We'll likely need to average this measurement over multiple runs and/or blocks as we've previously noticed high variability in the performance of the nodes.

One thing to take into consideration is the planned move of cosmos to a new IAVL implementation, which would likely impact performance.

If we continue with the IAVL tree approach, we'll want to consider writing this swingset state into a different module than the current swingset cosmos module to provide some separation (e.g. in case we want to directly use the IAVL tree from SwingSet in the future).

[mhofman]

Closing as we had spiked this and decided to take this approach for #7225