maybe: HostDB.publishStore: public/provable merkle-tree storage, liveslots-supported Notifiers

[warner]

What is the Problem Being Solved?

#4558 uses application-managed device nodes to distribute the authority to write to various subtrees of a chain-validated hierarchical data structure (the chain's merkle tree, plus Event publishing). This provides the tools to perform cheap publishing of state changes without using (paid) transactional messages. In that approach, the kernel is unaware of anything special going on: the host app (cosmic-swingset) provides a device to perform the writes, and the host-provided bootstrap vat divides up the namespace and doles out writer authority to each vat.

An alternate approach would elevate this data structure to be kernel-visible. It would also elevate the usage of it (the Notifier/Updater pattern) to be part of liveslots.

In this scheme, swingset kernels would need to be provided with a way to write into this provable data tree.

Swingset requires that the host application provide a "HostDB" object, with subcomponents for kvStore, streamStore, snapStore, etc, and certain rules about when/if changes are committed. This keeps the kernel unaware of the host's transactional lifecycle, which is important for managing consensus within a chain, as well as preventing "hangover inconsistency". The general rule is that any changes the kernel makes to its HostDB must be committed/abandoned at the same time as the host application's internal data.

The kvStore is not currently exposed to the outside world. There are portions of it that are not part of consensus (snapshot identifiers, at least). We might change this as part of #3769, to enable new validators to copy the kvStore from an existing validator (along with the rest of HostDB, to catch up faster). But even then, we would probably treat the kernel's kvStore schema as an internal detail, and not use it for userspace publishing of vat data. And we certainly wouldn't want the chain to publish an Event into the block results for every change to kvStore.

So this scheme would add a new component to HostDB that expressly is published into the chain's state vector, and where every change does add an Event into the block results. This new HostDB component should expose a tree-shaped data structure with separator-joined string pathnames, and string values.

Once added, the kernel would subdivide the tree by vat, and (somehow) automatically give each vat the authority to manipulate its given subtree. One approach might be to add a variant of vatPowers.vatstore.(get|set|delete), perhaps with an additional method to learn the externally-visible pathname of that vat's subtree.

A further enhancement might be to make Notifiers first-class within liveslots. In this scheme, usersapce doesn't get to write directly, but instead it receives vatPowers.makeNotifier() to make a new one. Each Notifier would automatically allocate a child path within the vat's subtree. The notifier's updater could be invoked with plain-JSON-serializable data, and liveslots would make a new syscall.publishStore.write(path, data) or something to get the data into the kernel. The kernel would then write it into the new HostDB component (scoped by the vatID), and the host application would react to that by writing it into the merkle tree and adding an Event to the block.

These Notifiers could also be used as normal objects, with getUpdateSince. If we didn't offer that, or maybe if you never call it, the Notifier would be Durable (since it wouldn't need to track any Promises). The Notifier would also need a way to learn its pathname, so you could tell someone outside the chain what to look for.

I'm slightly against this approach, because it introduces a new kernel concept that is only relevant to chain-hosted kernels, not solos. If Swingset were its own product, we'd want to draw a line between what the kernel knows about and what the host application knows about, and the idea of an automatically-published +provable +Event-announcing data structure only makes sense on a chain. (@dtribble briefly mentioned putting metrics or stats here, but I think I convinced him that it's too expensive to use for things like that, and that we should push developers who care to run an instrumented follower node to extract that data). The HostDB-gets-committed-by-host-app behavior is critical for chains, but it's also super-relevant for solo nodes too, because it needs to be integrated with IO, to avoid hangover inconsistency. Most other kernel features are coherent for all kinds of deployment shapes. But this new form of storage would be awfully weird to see in a solo machine (who needs to get merkle-tree proofs out of a solo machine? why would anyone believe them anyways?), so having native support for it in Swingset feels awkward, like "your blockchain is showing".

But, the chain is the primary use case for Swingset, at least so far. If it makes things significantly cleaner for userspace, I'm willing to consider it.

Description of the Design

• HostDB.publishStore (name TBD) with a tree-shaped write-only string-keyed string-valued publish pathway
  • cosmic-swingset provides one that 1: writes to IAVL tree, and 2: adds Events to each block with the changes
• syscall.publishStore.write(), maybe delete
• vatPowers.publishStore or vatPowers.makeNotifier

Questions:

• should the ability to publish be ambient within a vat? (if it's write-only, this isn't a communication channel, at least not within the vat.. more like console.log)
• should the ability to make published notifiers be ambient within a vat?
• should this notifier publish strings? JSON-encodable data? Some sort of marshal() call? (and if so, how are slotToVal/valToSlot handled??)

Security Considerations

Vats that do a lot of publishStore writes, especially to a large number of keys/subpaths, could consume a lot of space. IAVL writes on Cosmos are really expensive (although I bet not as expensive as running JS). This might be a DoS vector. Vats should probably be charged/metered for their use.

publishStore writes need to be buffered until the host app commits the block. They should not be visible to the outside world unless+until the rest of the state changes are committed, else you get hangover inconsistency.

Since kvStore lives in a separate DB than the Cosmos-SDK IAVL tree, committed at different times, there is a window when the kernel kvStore is committed but the host IAVL DB is not. If the system crashes in this window, the next restart is handled specially (cosmic-swingset realizes it does not need to make certain deliveries into the kernel, because the kernel state has seen them already). This interaction is fragile and needs to be considered carefully each time we think about introducing a new HostDB component or change anything about the order of commit calls. These considerations would apply to the new publishStore. (although I suspect the considerations are trivial, since the data will go into the same IAVL tree that the rest of cosmos-sdk uses).

Test Plan

Unit tests within SwingSet to confirm that vat action gets turned into publishStore writes as expected.

Unit tests within cosmic-swingset to confirm that publishStore writes turn into IAVL writes and Events as expected.

cc @dtribble @gibson042 @michaelfig @erights