swingset bulk-storage API for vats

[warner]

To move large Vat's storage needs from RAM to disk, it would be nice if Vats had some sort of storage API to the kernel. This could be a per-vat disk-resident key-value store provided directly by the kernel, or implemented in some device that vats can reach with existing syscalls.

(it's also worth considering simply running each vat in a separate process, and let the normal OS virtual-memory paging system take care of the problem, however Node.js frequently enforces a 1.5GB maximum heap size, which will limit our ability to use it)

I'm personally inclined to make this be a get/set/delete syscall API, specified to be persisted coherently with cranks/blocks by the kernel and host loops. But I know @dtribble felt it should be managed by a distinct device, rather than the kernel.

In either case, the get API requires a synchronous return value in the syscall, which is a drag.

[warner]

@FUDCo added syscall.vatstore{Get,Set,Delete} a while ago, to support "virtual objects". The user-facing API for bulk storage should export these calls, except with a prefix on the key (so userspace cannot interfere with the keys owned by the virtual object manager, which would be an authority violation).

These should be methods on the vatPowers object, only available to the root object (and any other object it chooses to share them with, of course), since they represent a shared-mutable-state communication channel.

[warner]

Strawman API:

function buildRootObject(vatPowers) {
  const { vatStore } = vatPowers;
  function method(args) {
    vatStore.set('key', 'value');
    vatStore.delete('otherkey');
    return vatStore.get('key2');
  }
}

strawman implementation (code added to liveSlots.js):

  // give user-level code access to a prefixed subset of the vatstore
  function userKey(key) {
    return `u.${key}`;
  }
  const vatStore = {
    get: (key) => syscall.vatstoreGet(userKey(key)),
    set: (key, value) => syscall.vatstoreSet(userKey(key), value),
    delete: (key) => syscall.vatstoreDelete(userKey(key)),
  };

    const rootObject = buildRootObject(
      harden({ D, exitVat, exitVatWithFailure, vatStore, ...vatPowers }),
      harden(vatParameters),
    );

Questions:

• should liveslots just go ahead and marshal.serialize() the data userspace puts into the store? i.e. should we store caps, in addition to inert data?
  • (this would add to the GC task, we'd need to chase decrefs when the data is changed or deleted, just like we'll need to do with the contents of virtual objects)
  • this might also make it easier to accept binary data in the future, since we'd be serializing it with whatever mechanism we decide to use on remote-message arguments
• if not, do we restrict callers to using just strings (since that's all the syscall API accepts), or do a basic JSON.stringify first?

[warner]

This is made visible to (some) vats on vatPowers.vatstore, which has get/set/delete methods. To enable this, the vat's managerOptions.enableVatstore must be true, which leaves it up to the host application (when it configures static vats), or the caller of vatAdmin.createVat().

I think that's enough to close this. Changes we might want to make in the future:

• enable this for all vats, removing the enableVatStore switch
• add an enumeration/iteration API, probably involving Vatstore syscall API needs some kind of "enumerate keys" operation #3734