Skip to content

Commit

Permalink
fix(swingset): stop using a persistent kernel bundle
Browse files Browse the repository at this point in the history 
Previously, the kernel sources were bundled during
`initializeSwingset()`, and stored in the kvStore portion of the
swingstore kernelDB. It was read from that DB each time
makeSwingsetController() launched the application process.

That ensured the kernel behavior would remain consistent (and
deterministic) even if the user's source tree changed. Using a
persistent kernel bundle also speeds up application launch: we save
about 1.8s by not re-running `bundleSource` on each launch.

Once upon a time I thought this consistency was important/useful, but
now I see that it causes more problems than it's worth. In particular,
upgrading a chain requires an awkward "rekernelize" action to replace
the kernel bundle in the database. And we believe we can reclaim about
1.0s per launch by moving to a not-yet-written `importLocation()` Endo
tool that loads module graphs from disk without serializing the
archive in the middle.

This patch changes swingset to stop using a persistent kernel
bundle. Each application launch (i.e. `makeSwingsetController()` call)
re-bundles the kernel sources just before importing that bundle, so
that updated source trees are automatically picked up without
additional API calls or rekernelize steps. The bundle is no longer
stored in the kvStore.

The other bundles (vats, devices, xsnap helpers) are still created
during `initializeSwingset` and stored in the DB, since these become
part of the deterministic history of each vat (#4376 and #5703 will
inform changes to how those bundles are managed). Only the kernel
bundle has become non-persistent.

Bundling is slow enough that many unit tests pre-bundle both the
kernel and the sources of built-in vats, devices, and xsnap helper
bundles. This provides a considerable speedup for tests that build (or
launch) multiple kernels within a single test file. These
`kernelBundles` are passed into `initializationOptions`, which then
bypassed the swingset-bundles-it-for-you code. This saves a minute or
two when running the full swingset test suite.

Unit tests can still use `initializationOptions` to amortize bundling
costs for the non-kernel bundles. `runtimeOptions` now accepts a
`kernelBundle` member to amortize bundling the kernel itself. Tests
which use this trick and also use `makeSwingsetController()` need to
be updated (else they'll run 1.8s slower). All such tests were inside
`packages/SwingSet/test/` and have been updated, which would otherwise
cause the test suite to run about 50-60s slower.

The other agoric-sdk packages that do swingset-style tests are mostly
using the deprecated `buildVatController()`, whose options continue to
accept both kinds of bundles, and do not need changes.

closes #5679
  • Loading branch information
@warner
warner committed Jun 30, 2022
1 parent 1f2c465 commit ad0d304
Show file tree
Hide file tree
Showing 11 changed files with 123 additions and 75 deletions.
8 changes: 6 additions & 2 deletions packages/SwingSet/src/controller/controller.js
View file
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@ import { makeGcAndFinalize } from '../lib-nodejs/gc-and-finalize.js';
import { insistStorageAPI } from '../lib/storageAPI.js';
import { provideHostStorage } from './hostStorage.js';
import {
buildKernelBundle,
swingsetIsInitialized,
initializeSwingset,
} from './initializeSwingset.js';
Expand Down Expand Up @@ -238,6 +239,10 @@ export async function makeSwingsetController(
// see https://github.com/Agoric/SES-shim/issues/292 for details
harden(console);

writeSlogObject({ type: 'bundle-kernel-start' });
const { kernelBundle = await buildKernelBundle() } = runtimeOptions;
writeSlogObject({ type: 'bundle-kernel-finish' });

// FIXME: Put this somewhere better.
const handlers = process.listeners('unhandledRejection');
let haveUnhandledRejectionHandler = false;
Expand All @@ -254,8 +259,6 @@ export async function makeSwingsetController(
function kernelRequire(what) {
assert.fail(X`kernelRequire unprepared to satisfy require(${what})`);
}
// @ts-expect-error assume kernelBundle is set
const kernelBundle = JSON.parse(kvStore.get('kernelBundle'));
writeSlogObject({ type: 'import-kernel-start' });
const kernelNS = await importBundle(kernelBundle, {
filePrefix: 'kernel/...',
Expand Down Expand Up @@ -542,6 +545,7 @@ export async function buildVatController(
slogCallbacks,
warehousePolicy,
slogFile,
kernelBundle: kernelBundles?.kernelBundle,
};
const initializationOptions = { verbose, kernelBundles };
let bootstrapResult;
Expand Down
49 changes: 32 additions & 17 deletions packages/SwingSet/src/controller/initializeSwingset.js
View file
Original file line number Diff line number Diff line change
Expand Up @@ -29,27 +29,29 @@ const allValues = async obj =>
fromEntries(zip(keys(obj), await Promise.all(values(obj))));

/**
* Build the source bundles for the kernel and xsnap vat worker.
*
* @param {object} [options]
* @param {ModuleFormat} [options.bundleFormat]
* Build the source bundles for the kernel. makeSwingsetController()
* calls this on each launch, to get the
* current kernel sources
*/
export async function buildKernelBundles(options = {}) {
// this takes 2.7s on my computer

const { bundleFormat = undefined } = options;
export async function buildKernelBundle() {
// this takes about 1.0s on my computer
const src = rel => bundleSource(new URL(rel, import.meta.url).pathname);
const kernel = await src('../kernel/kernel.js');
return harden(kernel);
}

const src = rel =>
bundleSource(new URL(rel, import.meta.url).pathname, {
format: bundleFormat,
});
/**
* Build the source bundles for built-in vats and devices, and for the
* xsnap vat worker.
*/
export async function buildVatAndDeviceBundles() {
const src = rel => bundleSource(new URL(rel, import.meta.url).pathname);
const srcGE = rel =>
bundleSource(new URL(rel, import.meta.url).pathname, {
format: 'getExport',
});

const bundles = await allValues({
kernel: src('../kernel/kernel.js'),
adminDevice: src('../devices/vat-admin/device-vat-admin.js'),
adminVat: src('../vats/vat-admin/vat-vat-admin.js'),
comms: src('../vats/comms/index.js'),
Expand All @@ -67,6 +69,22 @@ export async function buildKernelBundles(options = {}) {
return harden(bundles);
}

// Unit tests can call this to amortize the bundling costs: pass the
// result to initializeSwingset's initializationOptions.kernelBundles
// (for the vat/device/worker bundles), and you can pass .kernelBundle
// individually to makeSwingsetController's
// runtimeOptions.kernelBundle

// Tests can also pass the whole result to buildVatController's
// runtimeOptions.kernelBundles, which will pass it through to both.

export async function buildKernelBundles() {
const bp = buildVatAndDeviceBundles();
const kp = buildKernelBundle();
const [vdBundles, kernelBundle] = await Promise.all([bp, kp]);
return harden({ kernelBundle, ...vdBundles });
}

function byName(a, b) {
if (a.name < b.name) {
return -1;
Expand Down Expand Up @@ -318,17 +336,14 @@ export async function initializeSwingset(
}

const {
kernelBundles = await buildKernelBundles({
bundleFormat: config.bundleFormat,
}),
kernelBundles = await buildVatAndDeviceBundles(),
verbose,
addVatAdmin = true,
addComms = true,
addVattp = true,
addTimer = true,
} = initializationOptions;

kvStore.set('kernelBundle', JSON.stringify(kernelBundles.kernel));
kvStore.set('lockdownBundle', JSON.stringify(kernelBundles.lockdown));
kvStore.set('supervisorBundle', JSON.stringify(kernelBundles.supervisor));

Expand Down
1 change: 0 additions & 1 deletion packages/SwingSet/src/kernel/state/kernelKeeper.js
View file
Original file line number Diff line number Diff line change
Expand Up @@ -52,7 +52,6 @@ const enableKernelGC = true;
// device.nextID = $NN
// meter.nextID = $NN // used to make m$NN

// kernelBundle = JSON(bundle)
// namedBundleID.$NAME = bundleID
// bundle.$BUNDLEID = JSON(bundle)
//
Expand Down
4 changes: 3 additions & 1 deletion packages/SwingSet/test/device-hooks/test-device-hooks.js
View file
Original file line number Diff line number Diff line change
Expand Up @@ -59,8 +59,10 @@ test('add hook', async t => {
addVattp: false,
addTimer: false,
};
const { kernelBundle } = t.context.data.kernelBundles;
const runtimeOpts = { kernelBundle };
await initializeSwingset(config, [], hostStorage, initOpts);
const c = await makeSwingsetController(hostStorage, {});
const c = await makeSwingsetController(hostStorage, {}, runtimeOpts);

let hookreturn;
function setHookReturn(args, slots = []) {
Expand Down
31 changes: 18 additions & 13 deletions packages/SwingSet/test/device-mailbox/test-device-mailbox.js
View file
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@ import {
buildMailboxStateMap,
buildMailbox,
} from '../../src/devices/mailbox/mailbox.js';
import { bundleOpts } from '../util.js';

test.before(async t => {
const kernelBundles = await buildKernelBundles();
Expand All @@ -37,13 +38,14 @@ test('mailbox outbound', async t => {
},
},
};
const deviceEndowments = {
const devEndows = {
mailbox: { ...mb.endowments },
};

const { initOpts, runtimeOpts } = bundleOpts(t.context.data);
const hostStorage = provideHostStorage();
await initializeSwingset(config, ['mailbox1'], hostStorage, t.context.data);
const c = await makeSwingsetController(hostStorage, deviceEndowments);
await initializeSwingset(config, ['mailbox1'], hostStorage, initOpts);
const c = await makeSwingsetController(hostStorage, devEndows, runtimeOpts);
await c.run();
// exportToData() provides plain Numbers to the host that needs to convey the messages
t.deepEqual(s.exportToData(), {
Expand Down Expand Up @@ -85,13 +87,14 @@ test('mailbox inbound', async t => {
},
},
};
const deviceEndowments = {
const devEndows = {
mailbox: { ...mb.endowments },
};

const { initOpts, runtimeOpts } = bundleOpts(t.context.data);
const hostStorage = provideHostStorage();
await initializeSwingset(config, ['mailbox2'], hostStorage, t.context.data);
const c = await makeSwingsetController(hostStorage, deviceEndowments);
await initializeSwingset(config, ['mailbox2'], hostStorage, initOpts);
const c = await makeSwingsetController(hostStorage, devEndows, runtimeOpts);
await c.run();
const m1 = [1, 'msg1'];
const m2 = [2, 'msg2'];
Expand Down Expand Up @@ -143,23 +146,25 @@ async function initializeMailboxKernel(t) {
},
},
};
const { initOpts } = bundleOpts(t.context.data);
const hostStorage = provideHostStorage();
await initializeSwingset(
config,
['mailbox-determinism'],
hostStorage,
t.context.data,
initOpts,
);
return hostStorage;
}

async function makeMailboxKernel(hostStorage) {
async function makeMailboxKernel(t, hostStorage) {
const s = buildMailboxStateMap();
const mb = buildMailbox(s);
const deviceEndowments = {
const devEndows = {
mailbox: { ...mb.endowments },
};
const c = await makeSwingsetController(hostStorage, deviceEndowments);
const { runtimeOpts } = bundleOpts(t.context.data);
const c = await makeSwingsetController(hostStorage, devEndows, runtimeOpts);
c.pinVatRoot('bootstrap');
await c.run();
return [c, mb];
Expand All @@ -169,8 +174,8 @@ test('mailbox determinism', async t => {
// we run two kernels in parallel
const hostStorage1 = await initializeMailboxKernel(t);
const hostStorage2 = await initializeMailboxKernel(t);
const [c1a, mb1a] = await makeMailboxKernel(hostStorage1);
const [c2, mb2] = await makeMailboxKernel(hostStorage2);
const [c1a, mb1a] = await makeMailboxKernel(t, hostStorage1);
const [c2, mb2] = await makeMailboxKernel(t, hostStorage2);

// they get the same inbound message
const msg1 = [[1, 'msg1']];
Expand All @@ -195,7 +200,7 @@ test('mailbox determinism', async t => {
);

// then one is restarted, but the other keeps running
const [c1b, mb1b] = await makeMailboxKernel(hostStorage1);
const [c1b, mb1b] = await makeMailboxKernel(t, hostStorage1);

// Now we repeat delivery of that message to both. The mailbox should send
// it to vattp, even though it's duplicate, because the mailbox doesn't
Expand Down
57 changes: 34 additions & 23 deletions packages/SwingSet/test/devices/test-devices.js
View file
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@ import {
buildKernelBundles,
} from '../../src/index.js';
import buildCommand from '../../src/devices/command/command.js';
import { bundleOpts } from '../util.js';

function capdata(body, slots = []) {
return harden({ body, slots });
Expand Down Expand Up @@ -61,9 +62,10 @@ test.serial('d0', async t => {
},
},
};
const { initOpts, runtimeOpts } = bundleOpts(t.context.data);
const hostStorage = provideHostStorage();
await initializeSwingset(config, [], hostStorage);
const c = await makeSwingsetController(hostStorage, {});
await initializeSwingset(config, [], hostStorage, initOpts);
const c = await makeSwingsetController(hostStorage, {}, runtimeOpts);
await c.run();

// console.log(util.inspect(c.dump(), { depth: null }));
Expand Down Expand Up @@ -108,15 +110,16 @@ test.serial('d1', async t => {
},
},
};
const deviceEndowments = {
const devEndows = {
d1: {
shared: sharedArray,
},
};

const { initOpts, runtimeOpts } = bundleOpts(t.context.data);
const hostStorage = provideHostStorage();
await initializeSwingset(config, [], hostStorage, t.context.data);
const c = await makeSwingsetController(hostStorage, deviceEndowments);
await initializeSwingset(config, [], hostStorage, initOpts);
const c = await makeSwingsetController(hostStorage, devEndows, runtimeOpts);
c.pinVatRoot('bootstrap');
await c.run();

Expand Down Expand Up @@ -149,9 +152,10 @@ async function test2(t, mode) {
},
},
};
const { initOpts, runtimeOpts } = bundleOpts(t.context.data);
const hostStorage = provideHostStorage();
await initializeSwingset(config, [], hostStorage, t.context.data);
const c = await makeSwingsetController(hostStorage, {});
await initializeSwingset(config, [], hostStorage, initOpts);
const c = await makeSwingsetController(hostStorage, {}, runtimeOpts);
c.pinVatRoot('bootstrap');
await c.run(); // startup

Expand Down Expand Up @@ -238,10 +242,11 @@ test.serial('device state', async t => {
},
};

const { initOpts, runtimeOpts } = bundleOpts(t.context.data);
// The initial state should be missing (null). Then we set it with the call
// from bootstrap, and read it back.
await initializeSwingset(config, ['write+read'], hostStorage, t.context.data);
const c1 = await makeSwingsetController(hostStorage, {});
await initializeSwingset(config, ['write+read'], hostStorage, initOpts);
const c1 = await makeSwingsetController(hostStorage, {}, runtimeOpts);
const d3 = c1.deviceNameToID('d3');
await c1.run();
t.deepEqual(c1.dump().log, ['undefined', 'w+r', 'called', 'got {"s":"new"}']);
Expand All @@ -266,13 +271,14 @@ test.serial('command broadcast', async t => {
},
},
};
const deviceEndowments = {
const devEndows = {
command: { ...cm.endowments },
};

const { initOpts, runtimeOpts } = bundleOpts(t.context.data);
const hostStorage = provideHostStorage();
await initializeSwingset(config, [], hostStorage, t.context.data);
const c = await makeSwingsetController(hostStorage, deviceEndowments);
await initializeSwingset(config, [], hostStorage, initOpts);
const c = await makeSwingsetController(hostStorage, devEndows, runtimeOpts);
c.pinVatRoot('bootstrap');
c.queueToVatRoot('bootstrap', 'doCommand1', [], 'panic');
await c.run();
Expand All @@ -294,13 +300,14 @@ test.serial('command deliver', async t => {
},
},
};
const deviceEndowments = {
const devEndows = {
command: { ...cm.endowments },
};

const { initOpts, runtimeOpts } = bundleOpts(t.context.data);
const hostStorage = provideHostStorage();
await initializeSwingset(config, [], hostStorage, t.context.data);
const c = await makeSwingsetController(hostStorage, deviceEndowments);
await initializeSwingset(config, [], hostStorage, initOpts);
const c = await makeSwingsetController(hostStorage, devEndows, runtimeOpts);
c.pinVatRoot('bootstrap');
c.queueToVatRoot('bootstrap', 'doCommand2', [], 'panic');
await c.run();
Expand Down Expand Up @@ -339,9 +346,10 @@ test.serial('liveslots throws when D() gets promise', async t => {
},
},
};
const { initOpts, runtimeOpts } = bundleOpts(t.context.data);
const hostStorage = provideHostStorage();
await initializeSwingset(config, [], hostStorage, t.context.data);
const c = await makeSwingsetController(hostStorage, {});
await initializeSwingset(config, [], hostStorage, initOpts);
const c = await makeSwingsetController(hostStorage, {}, runtimeOpts);
c.pinVatRoot('bootstrap');

// When liveslots catches an attempt to send a promise into D(), it throws
Expand Down Expand Up @@ -377,9 +385,10 @@ test.serial('syscall.callNow(promise) is vat-fatal', async t => {
},
},
};
const { initOpts, runtimeOpts } = bundleOpts(t.context.data);
const hostStorage = provideHostStorage();
await initializeSwingset(config, [], hostStorage, t.context.data);
const c = await makeSwingsetController(hostStorage, {});
await initializeSwingset(config, [], hostStorage, initOpts);
const c = await makeSwingsetController(hostStorage, {}, runtimeOpts);
c.pinVatRoot('bootstrap');
await c.run();

Expand Down Expand Up @@ -413,13 +422,14 @@ test.serial('device errors cause vat-catchable D error', async t => {
},
};

const { initOpts, runtimeOpts } = bundleOpts(t.context.data);
const bootstrapResult = await initializeSwingset(
config,
[],
hostStorage,
t.context.data,
initOpts,
);
const c = await makeSwingsetController(hostStorage, {});
const c = await makeSwingsetController(hostStorage, {}, runtimeOpts);
await c.run();

t.is(c.kpStatus(bootstrapResult), 'fulfilled'); // not 'rejected'
Expand Down Expand Up @@ -451,13 +461,14 @@ test.serial('foreign device nodes cause a catchable error', async t => {
},
};

const { initOpts, runtimeOpts } = bundleOpts(t.context.data);
const bootstrapResult = await initializeSwingset(
config,
[],
hostStorage,
t.context.data,
initOpts,
);
const c = await makeSwingsetController(hostStorage, {});
const c = await makeSwingsetController(hostStorage, {}, runtimeOpts);
await c.run();

t.is(c.kpStatus(bootstrapResult), 'fulfilled'); // not 'rejected'
Expand Down
Loading

0 comments on commit ad0d304

Please sign in to comment.