-
Notifications
You must be signed in to change notification settings - Fork 212
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
feat: make new swingset-worker-xsnap-v1 package
This moves the xsnap-worker-creation code out of SwingSet and into a new package, whose NPM name is `@agoric/swingset-worker-xsnap-v1`, and lives (for now) in packages/swingset-worker-xsnap-v1 . This new package encapsulates: * the choice of `xsnap` executable, hence the XS engine version inside * the choice of `@agoric/xsnap-lockdown`, hence the Endo/SES version * the choice of `@agoric/swingset-xsnap-supervisor`, hence liveslots We roughly expect there will be exactly one version of `@agoric/swingset-worker-xsnap-v1` published. Any major changes to XS/xsnap/SES/liveslots will be published to a new `@agoric/swingset-worker-xsnap-v2` package, and a single kernel can depend upon both -v1 and -v2 at the same time, to support older vats (started with -v1) as well as newer vats (started with, or upgraded to, -v2). Minor (and sufficiently backwards-compatible) changes of the components may be possible. If so, we can publish new versions of -v1, and deployed systems can perform orchestrated (within-consensus) upgrades from e.g. -v1@1.0 to -v1@1.1 . We still need to think carefully about potential differences in behavior between the first-run deliveries (using 1.0) and the post-upgrade replayed deliveries (using 1.1). SwingSet now has a primary dependency on the new `@agoric/swingset-worker-xsnap-v1` package, and only `devDependencies` on the individual components (`@agoric/swingset-xsnap-supervisor` and `@agoric/xsnap-lockdown`). Hopefully these dev-dependencies will go away once the dust settles. refs #6596
- Loading branch information
Showing
13 changed files
with
323 additions
and
73 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
| dist |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,25 @@ | ||
| # swingset-worker-xs-v1 | ||
|
|
||
| This package provides a function to create a "xsnap-v1" SwingSet vat worker. | ||
|
|
||
| This worker will include specific (stable) versions of the following components: | ||
|
|
||
| * the `xsnap` package (@agoric/xsnap), which includes: | ||
| * the `xsnap` executable, a C program that combines a specific version of the XS JavaScript runtime, and a driver program (`xsnap.c`) that accepts command messages over a socket | ||
| * a JS library to launch that program as a child process, and then send/receive messages over the socket | ||
| * a "lockdown bundle" (@agoric/xsnap-lockdown), which can be evaluated inside the xsnap program, to transform the plain (unsecured) JS environment into our preferred (secure) SES/Endo environment, by taming the global constructors, removing ambient authority, and creating the `Compartment` constructor | ||
| * a "supervisor bundle", which can be evaluated after lockdown, to hook into the globally-registered handler function to accept delivery messages | ||
| * this imports a specific version of "@agoric/swingset-liveslots", to create the object-capability / distributed-messaging environment, which can route messages through syscalls, and provide virtual/durable object support | ||
|
|
||
| By importing `@agoric/swingset-worker-xsnap-v1`, the kernel will get a stable behavior from the worker (including consistent heap snapshot contents), regardless of changes to other packages, or behavior-neutral changes to the kernel itself. Any two kernels which use `xsnap-v1` and make the same sequence of deliveries (and syscalls responses) should get the same XS state, the same syscalls, the same metering, and the same XS heap snapshot hash. | ||
|
|
||
| To guard against accidental changes to the `@agoric/swingset-worker-xsnap-v1` package (perhaps yarn.lock being insufficient to lock down the bundle package versions correctly), the API can also return a hash string that describes the contents of the bundles. The kernel code that imports this module can compare this string against hard-coded values, which would only be changed when deliberately switching to a new version of the worker. | ||
|
|
||
| The intention is for the `@agoric/swingset-worker-xsnap-v1` package to be published to NPM only once: we publish version 1.0.0 and then never publish again. Any new changes would go into a new `@agoric/swingset-worker-xsnap-v2` package (of which we'd only ever publish 1.0.0 as well), etc. | ||
|
|
||
| We will figure out a different approach for "dev" development, where downstream clients want to use unstable/recent versions instead. Follow https://github.com/Agoric/agoric-sdk/issues/7056 for details. | ||
|
|
||
| ## API | ||
|
|
||
| The primary export is a function named `makeStartXSnapV1()`. This takes a set of authorities (the snapStore, `env`, and a `spawn` function) and returns a `startXSnapV1()` function. This latter function encapsulates all the components listed above. | ||
|
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,12 @@ | ||
| // This file can contain .js-specific Typescript compiler config. | ||
| { | ||
| "extends": "../../tsconfig.json", | ||
| "include": [ | ||
| "*.js", | ||
| "lib/**/*.js", | ||
| "src/**/*.d.ts", | ||
| "src/**/*.js", | ||
| "test/**/*.js", | ||
| "tools/**/*.js", | ||
| ], | ||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,47 @@ | ||
| { | ||
| "name": "@agoric/swingset-worker-xsnap-v1", | ||
| "version": "0.9.0", | ||
| "description": "make swingset xsnap-v1 workers", | ||
| "author": "Agoric", | ||
| "license": "Apache-2.0", | ||
| "type": "module", | ||
| "main": "./src/index.js", | ||
| "scripts": { | ||
| "build": "exit 0", | ||
| "clean": "exit 0", | ||
| "lint": "run-s --continue-on-error lint:*", | ||
| "lint:js": "eslint 'src/**/*.js' 'test/**/*.js'", | ||
| "lint:types": "tsc -p jsconfig.json", | ||
| "lint-fix": "eslint --fix 'src/**/*.js' 'test/**/*.js'", | ||
| "test": "ava", | ||
| "test:c8": "c8 $C8_OPTIONS ava --config=ava-nesm.config.js", | ||
| "test:xs": "exit 0" | ||
| }, | ||
| "dependencies": { | ||
| "@agoric/assert": "^0.5.1", | ||
| "@agoric/swingset-xsnap-supervisor": "^0.9.0", | ||
| "@agoric/xsnap": "^0.13.2", | ||
| "@agoric/xsnap-lockdown": "^0.13.2" | ||
| }, | ||
| "devDependencies": { | ||
| "@endo/bundle-source": "^2.4.2", | ||
| "@endo/init": "^0.5.52", | ||
| "@endo/marshal": "^0.8.1", | ||
| "ava": "^5.1.0", | ||
| "c8": "^7.12.0" | ||
| }, | ||
| "files": [ | ||
| "LICENSE*", | ||
| "src" | ||
| ], | ||
| "publishConfig": { | ||
| "access": "public" | ||
| }, | ||
| "ava": { | ||
| "files": [ | ||
| "test/**/test-*.js" | ||
| ], | ||
| "timeout": "2m", | ||
| "workerThreads": false | ||
| } | ||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
| export { makeStartXSnapV1 } from './make-v1.js'; |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,108 @@ | ||
| import fs from 'fs'; | ||
| import { Fail } from '@agoric/assert'; | ||
| import { type as osType } from 'os'; | ||
| import { xsnap, recordXSnap } from '@agoric/xsnap'; | ||
| import { getLockdownBundle } from '@agoric/xsnap-lockdown'; | ||
| import { getSupervisorBundle } from '@agoric/swingset-xsnap-supervisor'; | ||
|
|
||
| const NETSTRING_MAX_CHUNK_SIZE = 12_000_000; | ||
|
|
||
| /** | ||
| * SnapStore is defined by swing-store, but we don't import that: we get it | ||
| * from swingset as an option. To avoid an inconvenient dependency graph, | ||
| * we define the salient methods locally. | ||
| * | ||
| * @typedef {<T>(vatID: string, loadRaw: (filePath: string) => Promise<T>) => Promise<T>} LoadSnapshot | ||
| * @typedef {{ loadSnapshot: LoadSnapshot }} SnapStore | ||
| */ | ||
| /** | ||
| * @param {{ | ||
| * snapStore?: SnapStore, | ||
| * spawn: typeof import('child_process').spawn | ||
| * debug?: boolean, | ||
| * makeTraceFile?: () => string, | ||
| * overrideBundles?: { moduleFormat: string, source: string }[], | ||
| * }} options | ||
| */ | ||
| export function makeStartXSnapV1(options) { | ||
| // our job is to simply curry some authorities and settings into the | ||
| // 'startXSnapV1' function we return | ||
| const { snapStore, spawn, debug = false, makeTraceFile } = options; | ||
| const { overrideBundles } = options; | ||
|
|
||
| /** | ||
| * @param {string} vatID | ||
| * @param {string} name | ||
| * @param {(request: Uint8Array) => Promise<Uint8Array>} handleCommand | ||
| * @param {boolean} [metered] | ||
| * @param {boolean} [reload] | ||
| */ | ||
| async function startXSnapV1( | ||
| vatID, | ||
| name, | ||
| handleCommand, | ||
| metered, | ||
| reload = false, | ||
| ) { | ||
| await 0; // empty synchronous prelude | ||
|
|
||
| /** @type { import('@agoric/xsnap/src/xsnap').XSnapOptions } */ | ||
| const xsnapOpts = { | ||
| os: osType(), | ||
| spawn, | ||
| stdout: 'inherit', | ||
| stderr: 'inherit', | ||
| debug, | ||
| netstringMaxChunkSize: NETSTRING_MAX_CHUNK_SIZE, | ||
| }; | ||
|
|
||
| let doXSnap = xsnap; | ||
| if (makeTraceFile) { | ||
| doXSnap = opts => { | ||
| const workerTrace = makeTraceFile(); | ||
| console.log('SwingSet xs-worker tracing:', { workerTrace }); | ||
| fs.mkdirSync(workerTrace, { recursive: true }); | ||
| return recordXSnap(opts, workerTrace, { | ||
| writeFileSync: fs.writeFileSync, | ||
| }); | ||
| }; | ||
| } | ||
|
|
||
| const meterOpts = metered ? {} : { meteringLimit: 0 }; | ||
| if (snapStore && reload) { | ||
| // console.log('startXSnap from', { snapshotHash }); | ||
| return snapStore.loadSnapshot(vatID, async snapshot => { | ||
| const xs = doXSnap({ | ||
| snapshot, | ||
| name, | ||
| handleCommand, | ||
| ...meterOpts, | ||
| ...xsnapOpts, | ||
| }); | ||
| await xs.isReady(); | ||
| return xs; | ||
| }); | ||
| } | ||
| // console.log('fresh xsnap', { snapStore: snapStore }); | ||
| const worker = doXSnap({ handleCommand, name, ...meterOpts, ...xsnapOpts }); | ||
|
|
||
| let bundles = []; | ||
| // eslint-disable-next-line @jessie.js/no-nested-await | ||
| bundles.push(await getLockdownBundle()); | ||
| // eslint-disable-next-line @jessie.js/no-nested-await | ||
| bundles.push(await getSupervisorBundle()); | ||
| if (overrideBundles) { | ||
| bundles = overrideBundles; // replace the usual bundles | ||
| } | ||
|
|
||
| for (const bundle of bundles) { | ||
| bundle.moduleFormat === 'getExport' || | ||
| bundle.moduleFormat === 'nestedEvaluate' || | ||
| Fail`unexpected: ${bundle.moduleFormat}`; | ||
| // eslint-disable-next-line no-await-in-loop, @jessie.js/no-nested-await | ||
| await worker.evaluate(`(${bundle.source}\n)()`.trim()); | ||
| } | ||
| return worker; | ||
| } | ||
| return startXSnapV1; | ||
| } |
Oops, something went wrong.