kernel-on-new-SES

[warner]

(capturing today's meeting)

@jfparadis has made great progress with the new SES implementation (the one with Compartment). The current code lets you do:

import { lockdown } from 'SES';
// that adds, `Compartment` and `harden` to our globals
maybe_do_stuff_to_global();
lockdown();
// our global is still there, with platform-provided powers like Request

// but all primordials are now safe to share: their .constructor references
// to Function have been tamed

const endowments = { console: harden(tamedConsole), stuff };
const c = new Compartment(endowments);
const attacker = c.evaluate(attackerCode);
attacker(guess => "wrong!");

The endowments are added to the new compartment's global (rather than merely being visible in the top-level global scope).

c.evaluate() currently takes a second argument, which might be options, or might be per-eval endowments, or might be options which can include endowments. It seems likely that we'll get rid of per-eval endowments because they're annoying. From within a Compartment, plain eval() behaves the same way as a one-argument call to c.evaluate from outside the compartment. If/when we wanted to provide a two-argument evaluate() inside, we could provide an endowment that wraps c.evaluate (probably taming some of the options for safety).

Eventually the new Compartment API will grow to include a module map, and c.import(module_name). But for now we just get strings.

We're going to assume we can get full/enough debugging information despite using SES, and drop support for --no-ses because it would interfere with this plan.

First Step

The swingset plan (first step) is:

• host application doesn't know about SES
• swingset exports a function to build a new Swingset instance, which takes a config object and a hostDB
• inside that function, we do something like:

 const kernel_source = bundle_source(require.resolve('./kernel'));
 const genesis_vat_sources = bundle_source(config.something.filenames);
 const builtin_vat_sources = bundle_source(timerVatSource); // same with builtin devices
 c = new Compartment({ console: x, maybeothers });
 const kernel = c.evaluate(kernel_source)(kernelEndowments);

So we're using rollup at runtime, rather than having an npm build step to flatten the kernel code. (actually the zeroth step might be to flatten at build time, but removing that build step would be nice).

We have to bundle the builtin and genesis vats ahead of time, because the kernel still doesn't get access to the filesystem and cannot run rollup itself. This matches what we're currently doing in controller.js.

By removing --no-ses, we can get rid of @agoric/evaluate, and change both the kernel (in vat-creation code) and the contract-host vat (in the contract-evaluation code) to create a new Compartment and then call c.evaluate. The new SES puts harden into the globals, so we can remove require('harden'), and Nat can be flattened by rollup. So we remove makeRequire and all instances of require() from the flattened/rolluped vat source string. No holes.

Second Step: XS

XS currently gives us a Compartment API whose first argument is a module name, and whose second argument is a module map. The map can only point to module objects that were available at compile time (no dynamic modules right now). The XS crew is hard at work implementing the new API, but in the meantime we can shim it by creating an empty module (and compiling it into the app), then calling c = new Compartment('empty', modulemap); const c_eval = harden(c.global.eval); c_eval(newcode);. We'll need to add harden to the global first, and freeze/stash things before evaluating any untrusted code.

We'll need a build step that crawls the kernel source tree and builds a manifest file, which is fed into the XS build process to create the baked-in module map. We don't need to use rollup on these static sources (kernel, built-in vats/devices, probably genesis vats) because XS's compartment has a module loader. We just need to make sure the import statements in the kernel sources match the module specifiers that make it into the module map. We should probably make sure that the static vats get a map that only gives them access to their own vat code (and shared dependencies like Nat), not for safety reasons (each Compartment gets a separate instance of each module), but for clarity.

Then we'll need some startup code that creates a hostDB and a config object. This config object needs to point at module names, rather than source code, since we'll be loading them as modules. Then inside buildSwingset we'll do e.g. new Compartment('kernel') and new Compartment('genesisvat1') instead of evaluating rollup'ed strings.

Later Step: node + modules

Eventually, the SES-on-Node API will grow module support. Each Compartment will get a module map (passed as an argument to new Compartment), and the new compartment object will get a c.import(module_name) method. The module map answers the question "if X says import Y, give it Z", as well as information about how to build Z (perhaps source code that should be evaluated, perhaps a function that acquires it, maybe a hash of the expected source code and some sort of implicit loader, or something). Calling c.import('first') causes first to be looked up in the module map used to create c. The Y argument might be relative (../other) or absolute (moment) or whatever: the semantics of Y are left up to whoever created the module map. All the Compartment does is extract the foo in import 'foo' and look it up in the Y column of the module map.

The Z argument names a module, which is looked up in the parent Compartment's module map. Compartments can only grant access to a module if they themselves were given access to it. (Modulo dynamically-created modules which is still being worked out). But we can still do multi-level dependency injection: if first says import 'second', and second says import 'third', we can put third -> pseudothird in the child Compartment's module map and it will get pseudothird.

With modules working on Node, we can stop rollup-ing the kernel source (and the static vat/device sources). We need a build step that constructs a manifest somehow, and uses that manifest to create a module map, which is used to create the kernel's compartment.

dynamic vats

For now, the source code for dynamic vats must be delivered as a single string. The kernel's create-dynamic-vat function will make a new Compartment and c.evaluate the source code inside it.

Eventually, dynamic vats will be delivered as a bundle (maybe a zipfile) of modules, with enough metadata to build a module map for the new compartment. @michaelfig has a tool (a module translator) that converts a tree full of ES6 module files into a set of metadata+sourcecode objects, one at a time, and a second tool (a module loader) which takes this set and evaluates them and wires them together correctly. We can run the first tool on the developer's machine, send the zipfile over to the chain/server, and run the second tool inside the chain to load the new vat. The first tool involves babel and is too big to safely run on the chain, but the second tool is much smaller.

On XS, we either need to shim dynamic modules in (using some portion of these tools), or get dynamic module support added to the core. The latter sounds hard: XS currently compiles modules into C code, so building them dynamically might involve spawning a copy of gcc and then somehow using dlopen() to load the resulting ELF file.

[dckc]

Re dynamic module loading in xs, see https://gist.github.com/dckc/cbbd3e8469723b342cc90799ace7a287

[warner]

The branch 477-new-ses-2 succeeds at running a basic demo. I haven't tried running any unit tests yet.

$ yarn
$ yarn build
$ cd packages/SwingSet
$ node bin/vat demo/encouragementBot

I disabled a lot of code: tildot expansion, default-evaluate-options, probably others. All the lockdown safety options are disabled.

[dckc]

your journey is remarkably similar to my path getting this stuff to run on xs :)

[warner]

reminder to self: remove the disable-the-SES-warning code that was added to packages/cosmic-swingset/lib/ag-solo/entrypoint.js (the globalThis.harden = null) in #973 when we move everything to new-SES

[dckc]

Yesterday I checked in with @warner about this, which is blocking SwingSet-on-xs #47; I learned that that modulo metering, "take 6" i.e. #1078 on the 477-new-ses-6 is working pretty well.

For example, as of a7eb63d:
https://github.com/Agoric/agoric-sdk/blob/477-new-ses-6/packages/SwingSet/test/test-liveslots.js

I think this is far enough along to unblock XS work; metering can come along later.

[warner]

The first phase of this was completed yesterday with the landing of #1201. The kernel is now all-SES-all-the-time, and the host application is responsible for providing a SES environment (by importing the new @agoric/install-ses module, which calls lockdown()).

[warner]

This is sufficiently done. We'll add module support in some later ticket.