durable Far methods/functions

[warner]

What is the Problem Being Solved?

Our "Durable Objects" enable each version of a vat to supply the runtime behavior of a whole category ("Kind") of object, while their state is kept on disk and survives a version upgrade. Remote references arrive at the importing vat as a Presence, which can also be stored in virtual or durable data in their own disk DB. This is sufficient to allow "callback object" patterns to persist in durable storage.

However, remembering a whole object is a bit annoying when the functionality being exposed is limited to a single function. It requires selecting a name for the method to be invoked, and the best name is usually the same as the variable you'd be holding the reference in.

Also, it become awkward when used to "durable-ize" an existing API convention like #3787 virtual/durable promises. The creation side might not be so bad ({ promise, resolver } = makeDurablePromiseKit(); doReject ? resolver.resolve(foo) : resolver.reject(bar)), because usually we bundle both the positive resolve() authority and the negative reject() authority together. So retaining a single object with two methods isn't a stretch.

But on the subscriber side, where a durable pattern (trying to attach a callback that might not get executed until after the entire subscribing vat has been upgraded) might start like this:

const stuff = await E(remote).foo();
const { durablePromise } = stuff;
const makeDO = defineKind(..)
const durableObj = makeDO();
// now attach the durable object's methods to the durable promise

we must then pick a syntax for our .then replacement. One option is to provide a single callback object with assumed (fixed) method names:

durablePromise.durableThen(durableObj); // calls durableObj.resolved or .rejected

a variant would allow alternate method names

durablePromise.durableThen(durableObj, 'otherNameForResolved', 'rejName');

a deeper variant would allow two distinct objects

durablePromise.durableThen({ resolveTarget: durableObj,
                             resolveMethodName: 'otherNameForResolved',
			     rejectTArget: otherDurableObj,
			     rejectMethodName: 'rejName' });

The first option would be the easiest to implement, but I think it would be a hard sell for programmers used to plain Promises and .then. Especially because the call to .then/etc is usually occurs at an interface between two systems: the Promise provider doesn't know who will be subscribing to it, and the subscriber doesn't know why/how the promises is being resolved. The boundary knows enough about the two sides to glue them together. But if the subscriber has to name their methods in a particular way to satisfy the .then API, the abstraction boundary starts to leak, and the promise pattern loses some of its convenience.

If we could get an object that was as detail-hiding as a plain Function, but internally could be mapped to something durable, then we could maintain a very .then-like API. We might be able to get away with a syntax arc from p.then(res,rej) to E.when(p, res, rej) to VatData.when(durableP, durableRes, durableRej).

Description of the Design

A "Durable Method" would be a method of a Durable Object. It can be identified by the object's vref plus the method name. We've implement durable objects, and our serialization system will recognize them (during export or when storing into virtualized data), but we don't currently do anything to recognize their methods. We could add a WeakMap that maps each method to the (vref, methodname) pair, or we could attach a private Symbol-named property to each to remember the pair. And we could define a vref for the method: the object would have a vref of o+${kindID}/${instanceID}:${facetID}, and we could attach a .${methodName} suffix to build a vref for the method itself.

The slotToVal table would be unchanged: it maps baseref to the cohort of facets. convertSlotToVal would be augmented to recognize the longer vref and extract the named method from the facet when necessary. The valToSlot table would be expanded to map each method to its vref, in addition to the facets.

When this vref is exported to the kernel, the c-list will allocate a koNN kref for it, which can be sent to other vats just like any reference. Neither the kernel nor the other vats are aware of the baked-in method name. The "export status" key would grow from an encoded array of one Reachable/Recognizable/Neither flag per facet, to a table that also includes a flag for each exported method name. When stored in virtualized data, the Virtual Reference Manager would refcount the durable method by using the baseref (o+${kindID}/${instanceID}) as the index, just like it does for facets.

The importing vat would receive a Presence as usual. The one difference is that they must invoke this with a nameless eventual-send:

E(durableMethod)(args);

Trying to use E(durableMethod).methodname(args) on the durable method would result in an error on the exporting side, just like you'd get if you invoked a missing method.

We could also define "Durable Functions", which would be created from a new sort of Kind definition function, maybe something like:

const makeFooFunc = defineDurableFunctionKind('iface',
  (args) => state,
  (state) => result,
  finish);
const durableFoo = makeFooFunc(args);

where the third argument is a stripped-down form of the behavior definition in #4905. This form defines a single function per Kind. For each instance, we create a copy that is bound to the state of that one instance and return that durableFoo to the creator. This durableFoo has a vref that includes the kindID and instanceID (but needs no other distinguishers). The caller can drop the in-RAM durableFoo Representative at any time, and we can reconstruct it if/when the vref gets deserialized later.

The most general form of this would allow multiple units of behavior to be built around the same state object. You might have 0 or more bare functions and 0 or more facets (each being a full object with a variety of methods in them). The assessFacetiousness code that analyzes the behavior argument would need to make some decisions about how the different varieties should be shaped. In particular, a record of named functions would look the same as a single facet with named methods. Ideally we'd come up with an internal representation that didn't need to distinguish between the two.

Related

#61 talks about how to pipeline these nameless calls (mostly in the kernel), but that doesn't overlap much with the durable definition of their targets.

#3787 provides some motivation, because the most natural syntax for the subscriber of a received durable promise would be to provide durable methods/functions as callbacks. We could define VatData.when() to only accept durable (nameless) methods/functions as the second and third arguments, and the syntax would look just like the usual p.then().

Security Considerations

We should make it clear that getting access to a method foo of some durable object does not provide access to any other methods of that object, or to the object itself. I don't know if normal JS has some mechanism for this (I believe x = target.foo; x() does something different than target.foo(), but e.g. in Python they're the same thing and the binding to x happens during the .foo lookup, but in neither case does x give you an obvious way to get back to target). But if it does, we should probably document the fact that you can rely upon this as a security boundary.

Holding onto a method allow a remote party to keep your entire cohort of facets alive (and the state they access), which might be a larger handle than you intended to provide.

Test Plan

lots of unit tests

[warner]

If we didn't want to register every method of every facet at actualization time, we could also provide something like dff = VatData.DurableFarFunction(obj, methodname). This would throw if obj was not a durable object/facet and methodname was not a known method of that object. It would return the bound method (=== obj[methodname], since we're using objects-as-closures), but it would also register that method object in valToSlot, making it eligible for use as a durable far function/method.

Hm, although then we'd have to think about what happens if you extract obj[methodname] without registering it this way (and then serialize it somehow). From liveslot's point of view, you've just given it a precious in-RAM Function with no other context. We don't yet support these in marshal, but it'd make sense to think of them as "ephemeral Far functions", and assign them a new o+NN vref just like with ephemeral "Remotables" (ones that userspace creates with Far(iface, { ...methods })). These would be prohibited from use in durable data, but they could be exported from the vat or stored in virtual data without hassle (they just consume RAM until dropped). Their vref would depend on whether they were exported/serialized before or after registration. OT3H if we continue to implement Far by hoisting the prototype and inserting a new one, then you can't Far a hardened object, which means you can't Far a function that you've already DurableFar'ed, and vice versa. And you can't serialize a non-hardened object (although a lot of our serialization pathways do the hardening for you). And you can't pass a non-Far'ed Function (it isn't pass-by-anything).

So you must choose between Far(function) and DurableFar(function) before it will serialize. If you choose Far, you'll get an ephemeral vref that keeps the function (and therefore the facet, and the entire cohort) in RAM until dropped. If you choose DurableFar, you get a durable vref that embeds the facet and methodname, which doesn't keep anything in RAM, and can survive an upgrade. In both cases you get back the original Function object (modified and hardened, but still ===).

On the whole, I suspect we'll cause fewer surprises if we register the full vref for every method of every facet, making them all durable without requiring additional user effort.

[warner]

Our current call to registerValue will need to be extended to help it register all the methods in valToSlot, not just the facets. It might make sense to grow the API to accept a record that maps vref extension (everything past the baseref) to the object being registered. The VOM could then submit a table with all the facets and all their methods, so registerValue doesn't need to know as many of the details.

[warner]

@FUDCo says this probably wouldn't be too hard to implement (a special Kind that holds single functions instead of named methods). But we aren't trying to build #3787 for MN-1 (we decided the trick in #4567 (comment) is sufficient), and I didn't hear a strong demand for unnamed functions when I asked about this in yesterday's meeting. So I think we can defer this work until at least after MN-1, possibly forever.

[warner]

This came up again today, the Zoe/ZCF durabilization work had assumed that it already existed, and was trying to do:

const maker = defineDurableKind(handle, init, arg => result);

where arg => result was provided instead of the usual behavior record:

const maker = defineDurableKind(handle, init, { foo: arg => result, bar: arg => otherresult });

This provoked an error as assessFacetiousness(behavior) reported not instead of the required one or many. So the code reacted just fine, but the invalid attempt at least suggests that using plain defineDurableKind with a plain function is the desired API.

@erights agreed that this is still not a MN-1 priority.

[dckc]

I wonder to what extent this is addressed by internal/src/callback.js

cc @michaelfig

[michaelfig]

I wonder to what extent this is addressed by internal/src/callback.js

Probably not addressed; callback.js is an adapter to apply a caller's remotable object+method to a callee's arguments. It would only work with caller functions if they were already remotable (either Far or durable), nor does it encapsulate the combination of the remotable object+method (just passes them around in a record).

[michaelfig]

Oh, it looks like I only (incorrectly) skimmed the issue, and answered only one of the orthogonal questions above (wrt Far functions). Yes! internal/src/callback.js puts the object+method in a durable-compatible callback descriptor that can later be invoked with callE(callback, ...args) or callSync(callback, ...args).

The encapsulation I was talking about will be achievable via #7586 when it lands. That allows creating an exoClass that makes its methods dispatch to a set of reified callbacks, potentially different for every instance. Such an exo can be a perfect stand-in for any other object of the same interface, without any special considerations for how to invoke its methods.

When we've leaned into durable functions, it would be worthwhile to create a similar encapsulation so that a callback descriptor can be wrapped in a durable function and invoked just like a normal function.