refactor swingset source code to match runtime artifacts

[warner]

What is the Problem Being Solved?

It's not particularly obvious how the swingset source code maps to the various Compartments and bundles and workers that get used at runtime. We've mused about a better layout, but now I wanted to write up a proposal and implement it.

Swingset is a library, meant to be imported into a host application, which interacts with a "controller". The library consists of several pieces:

• "controller": this code is imported into the host application, and thus runs in the "Start Compartment" (where it has access to the standard Node.js facilities like fs and crypto)
• "kernel": the controller creates a bundle of all the kernel code and stores it in the database, so a given swingset instance will keep using the same kernel code until deliberately upgraded (upgrading the kernel itself: controller.setKernelBundleID() #4375). It uses importBundle to create a new Compartment and evaluate the kernel bundle inside it.
• "worker": most vats run inside an "xsnap worker", which is a separate (unix) process launched from the xsnap binary. The kernel/host-app process sends commands over a pipe to evaluate code bundles and process commands. The first bundle evaluated inside the worker is the lockdown bundle, which establishes a SES environment. The second is the supervisor bundle, which installs the command handler. The command handler currently imports liveslots (which is therefore incorporated into the supervisor bundle), but we intend (upgrading liveslots: controller.setLiveslotsBundleID() #4376) to make that a separate bundle and deliver it over the command pipe. Finally the supervisor is instructed to evaluate the vat source bundle, to personalize the worker into a specific vat.
• "devices"/"vats": swingset defines a number of vats and devices for internal use (comms, vatAdmin, bundle), which are present in all swingset instances, and need no external endowments. Their source code lives inside the swingset tree, and are bundled by the controller, then stored in the DB. Swingset also provides the source code for some common devices (timer, mailbox, network) that are used by most host applications. These common devices are only enabled if the host configures them (and provides the necessary endowments like a clock). If configured, their sources are bundled by the controler and stored in the DB.
• "development tools": we have a few exports that are intended to help write unit tests of code that expects to run inside a vat environment, without actually starting a whole swingset kernel. We have a debug flavor and a non-debug flavor. By importing one of these files from your test-*.js file, before importing anything else, the test program is provided with a suitable SES environment that includes adequate mocks for the special vat facilities like VatData.makeKind (for virtual objects).

Bundles include a copy of all the files transitively imported by their entry-point file. The bundler (Endo) will refuse to follow imports to non-source modules, like Node's builtins (fs) or modules that include C/C++ -compiled .so binary libraries. So if two components reference a common source file, and one component is included in a bundle, that common source file must not import any builtins or compiled modules.

Description of the Design

I'm thinking of the following layout:

• src/lib/*.js : small files which are used by multiple places, like parseVatSlot.js. These should not import anything (at least not outside of lib/), and must not import builtins or binary modules. These are safe to use from bundled code.
• src/kernel/*.js: files which get included in the kernel bundle. Excludes liveslots and vat-supervisor code. Includes vat-manager code. Includes vat-warehouse.
• src/supervisor/*.js: files which get included in the supervisor bundle. Maybe break this out into subdirs for the different worker types.
• src/liveslots/*.js: files which get included in the liveslots bundle. References lib/ but nothing else. Includes all the virtual object/collection code.
• src/builtin-vats-devices/*.js: code to be bundled into built-in devices and vats: comms, vat-tp
• src/devices/: code for devices that userspace must enable and configure, not included automatically

I need to understand src/vats/network/*.js and the "plugin manager" to see where they fit. We might need a src/vats/ if swingset provides source code for vats that must be enabled by the host application.

Security Considerations

none I can can think of, probably makes things easier to audit

Test Plan

If the unit tests (which will need a lot of import-path surgery) still work, we'll know this was successful.

cc @FUDCo is there any other housekeeping you'd like to see done while we're tidying up?

[warner]

here's my current plan:

target	now
DELETE	src/kernel/json-stable-stringify.js
controller	src/controller.js
controller	src/hostStorage.js
controller	src/initializeSwingset.js
controller	src/kernel/initializeKernel.js
controller	src/spawnSubprocessWorker.js
controller	src/validate-archive.js
devices	src/devices/index.js
devices/bridge	src/devices/bridge-src.js
devices/bridge	src/devices/bridge.js
devices/bundle	src/devices/bundle.js
devices/command	src/devices/command-src.js
devices/command	src/devices/command.js
devices/lib	src/deviceTools.js
devices/loopbox	src/devices/loopbox-src.js
devices/loopbox	src/devices/loopbox.js
devices/mailbox	src/devices/mailbox-src.js
devices/mailbox	src/devices/mailbox.js
devices/plugin	src/devices/plugin-src.js
devices/plugin?	src/devices/plugin.js
devices/timer	src/devices/timer-src.js
devices/timer	src/devices/timer.js
devices/vatadmin	src/kernel/vatAdmin/vatAdmin-src.js
kernel	src/kernel/cleanup.js
kernel	src/kernel/deviceManager.js
kernel	src/kernel/deviceSlots.js
kernel	src/kernel/deviceTranslator.js
kernel	src/kernel/gc-actions.js
kernel	src/kernel/id.js
kernel	src/kernel/kdebug.js
kernel	src/kernel/kernel.js
kernel	src/kernel/kernelQueue.js
kernel	src/kernel/kernelSyscall.js
kernel	src/kernel/loadVat.js
kernel	src/kernel/metrics.js
kernel	src/kernel/notifyTermination.js
kernel	src/kernel/parseKernelSlots.js
kernel	src/kernel/slogger.js
kernel	src/kernel/vatTranslator.js
kernel	src/kernel/index.js
kernel/state	src/kernel/state/deviceKeeper.js
kernel/state	src/kernel/state/kernelKeeper.js
kernel/state	src/kernel/state/reachable.js
kernel/state	src/kernel/state/storageWrapper.js
kernel/state	src/kernel/state/vatKeeper.js
kernel/vm	src/kernel/vatManager/factory.js
kernel/vm	src/kernel/vatManager/manager-helper.js
kernel/vm	src/kernel/vatManager/manager-local.js
kernel/vm	src/kernel/vatManager/manager-nodeworker.js
kernel/vm	src/kernel/vatManager/manager-subprocess-node.js
kernel/vm	src/kernel/vatManager/manager-subprocess-xsnap.js
kernel/vm	src/kernel/vatManager/transcript.js
kernel/vm	src/kernel/vatManager/types.js
kernel/vm	src/kernel/vatManager/vat-warehouse.js
lib	src/assertOptions.js
lib	src/capdata.js
lib	src/kernel/djson.js
lib	src/kernel/dummyMeterControl.js
lib	src/makeUndeliverableError.js
lib	src/message.js
lib	src/netstring.js
lib	src/parseVatSlots.js
lib	src/runPolicies.js
lib	src/storageAPI.js
lib-impure	src/engine-gc.js
lib-impure	src/gc-and-finalize.js
lib-impure	src/hasher.js
lib-impure	src/waitUntilQuiescent.js
lib-impure	src/worker-protocol.js
liveslots	src/kernel/collectionManager.js
liveslots	src/kernel/liveSlots.js
liveslots	src/kernel/virtualObjectManager.js
liveslots	src/kernel/virtualReferences.js
src	src/index.js
src/??	src/types.js
src/?? ext	src/storeModule.js
src/?? ext	src/types-exported.js
supervisor/lib	src/kernel/vatManager/supervisor-helper.js
supervisor/nw	src/kernel/vatManager/supervisor-nodeworker.js
supervisor/sn	src/kernel/vatManager/supervisor-subprocess-node.js
supervisor/sx	src/kernel/vatManager/lockdown-subprocess-xsnap.js
supervisor/sx	src/kernel/vatManager/supervisor-subprocess-xsnap.js
vats/comms	src/vats/comms/cdebug.js
vats/comms	src/vats/comms/clist-inbound.js
vats/comms	src/vats/comms/clist-kernel.js
vats/comms	src/vats/comms/clist-outbound.js
vats/comms	src/vats/comms/clist-xgress.js
vats/comms	src/vats/comms/clist.js
vats/comms	src/vats/comms/controller.js
vats/comms	src/vats/comms/delivery.js
vats/comms	src/vats/comms/dispatch.js
vats/comms	src/vats/comms/gc-comms.js
vats/comms	src/vats/comms/index.js
vats/comms	src/vats/comms/parseLocalSlots.js
vats/comms	src/vats/comms/parseRemoteSlot.js
vats/comms	src/vats/comms/remote.js
vats/comms	src/vats/comms/state.js
vats/network	src/vats/network/bytes.js
vats/network	src/vats/network/index.js
vats/network	src/vats/network/multiaddr.js
vats/network	src/vats/network/network.js
vats/network	src/vats/network/router.js
vats/network	src/vats/network/types.js
vats/plugin	src/vats/plugin-manager.js
vats/timer	src/vats/types.js
vats/timer	src/vats/vat-timerWrapper.js
vats/timer?	src/vats/timed-iteration.js (needs types.js Timer)
vats/vat-tp	src/vats/vat-tp.js
vats/vatadmin	src/kernel/vatAdmin/vatAdminWrapper.js

[FUDCo]

That all seems rational. The one thing that looks weirdly non-orthogonal is moving liveSlots stuff to its own directory but leaving devSlots in kernel. Not that there's really that much in devSlots to justify a separate directory, but the non-parallel structure tugs at my OCD. (And I can imagine devSlots growing. Say, if we add some flavor of promise support to devices).

[warner]

Hm, yeah, that's kinda weird, but I think it lines up. The important distinction is what bundle each file gets included in: xsnap workers do both importBundle(liveslotsBundle) and importBundle(vatBundle), so liveslots can't include anything from e.g. kernel/. Device code gets bundled and imported into a new Compartment, but deviceSlots.js gets included in the kernel bundle (not the device bundle), and devices always run in the kernel process. So I think deviceSlots.js wants to be closer to the kernel than the devices. My goal is that each bundle corresponds to a directory.

The really weird part is that the kernel actually uses a copy of liveslots for worker='local' vats. Those vats bundles are loaded into their own Compartment within the kernel process, but they all talk to a single evaluation of liveSlots.js as imported by manager-local.js. It would be possible to create a new kernel-side Compartment to hold a copy of liveslots for each local vat, which would parallel the subprocess-worker more, but it'd be a waste of memory.