handle named bundles in kernel config and core-bootstrap

[warner]

What is the Problem Being Solved?

Once we have bundlecaps (#4372), we'd like the new #4165 / #4247 "core bootstrap" scheme to use them too. Currently this relies upon vatAdminService~.createVatByName(name), rather than a bundlecap or bundleID.

That means changing the kernel config format to somehow handle named bundles properly. We want vats to uniformly defined by a bundleID, so that the future upgrade protocol has a well-defined initial value to upgrade from.

The old code processes a config.bundles section which maps bundle names to either a filename (from which bundleSource can create a bundle) or an actual bundle (used only by tests). It then builds a table of named bundles, which vatAdminService~.createVatByName() can reference. @dckc 's

The new approach should process config.bundles and install each bundle, then build a table mapping name to bundleID. The bootstrap() message should be enhanced to provide this table to the bootstrap vat (perhaps through a third argument, so bootstrap(vats, devices, bundles)), which can use devices.bundle to get bundlecaps, and send them to vatAdminService~.createVat(bundlecap).

When we're done, we can remove createVatByName, and createVat() (which exclusively take a bundlecap) will be the only remaining API.

Description of the Design

Security Considerations

Test Plan

[warner]

#4372 takes config.bundles and installs each one, so bootstrap can do bundlecap = D(devices.bundle).getNamedBundleCap(name) and pass that through to whatever does createVat(). I decided to avoid the third bootstrap() argument.

For now, vatAdminService~.createVatByName() still works, so I think I can wait for the new bootstrap code to land before thinking about how to change it to use bundlecaps.

[warner]

The new bootstrap code landed a while ago. It currently grabs the whole bundle and passes it to the old E(zoe).install(bundle) API:

 const bundleCap = D(vatAdmin).getNamedBundleCap(name); 
 const bundle = D(bundleCap).getBundle(); 
 producer.resolve(E(zoe).install(bundle)); 

To avoid holding the whole bundle, we need to give bootstrap a way to get from a name to a bundleID, so it can use E(zoe).installBundleID(id) instead of install(bundle).

This wasn't straightforward when we first landed the bundle-handling code, because IDs (hashes) weren't always present (there was still a lot of placeholder code). But these days, bundleSource should always be providing the hash. So the task is twofold:

• add E(vatAdminService).getBundleIDByName(name) -> P<BundleID> (or rejects if the name is unknown)
• change bootstrap to use that and E(zoe).installBundleID(id)

An alternative would be to add E(zoe).installBundleCap(bcap), since bootstrap already has a way to get the named bundlecaps. But it's slightly nicer to keep the bundlecaps out of bootstrap entirely, since we don't yet GC them, so they'll just hang out in the bootstrap vat's c-list forever.

[Chris-Hibbert]

basicBehaviors.js now has these lines:

    // TODO (#4374) this should be E(zoe).installBundleID(bundleID);
    const installation = E(zoe).install(bundle);

[Tartuffo]

@dckc or @kriskowal please sync with @warner to see if this really is necessary for MN-1, and whether already done.

[dckc]

My understanding is that all the bundle id / hash bundle stuff is motivated by the goal of setting a max message size around 10M or so; that is: this is blocking #4280 .

• this is not done, as demonstrated by the call to E(zoe).install(bundle) noted above (bundle is likely to be much larger than 10M. p.s. nope... more like 1M)
• this is necessary for MN-1 iff impose configurable hard limit on message sizes #4280 is necessary

[Tartuffo]

We moved #4280 to MN-1.1, so I'm moving this one, too.

[warner]

@dckc and I implemented this today, at least for the statically-installed contract bundles. core-bootstrap installBootContracts is defined with a list of contract names (produce: { centralSupply, mintHolder, walletFactory }), all of which are assumed to be defined in config.bundles. For each one, it asks id = await E(vatAdminSvc).getBundleIDByName(name), and then does E(zoe).installBundleID(id).

That removes the processing of the large contract bundle from both vat-bootstrap and vat-zoe. Zoe converts the BundleID into a BundleCap at installation time, and then sends the BundleCap to ZCF (instead of the full bundle) at instantiation time.

I need to do another pass through basic-behaviors.js > makeVatsFromBundles() to make sure we aren't holding large bundles for the static vats (as opposed to the static contracts that we just fixed). I see teh function is named ..FromBundles, but the function it provides takes bundleName, and calls createVatByName, which I think means it's not handling large bundles anywhere.

Incidentally, when looking at the vat-zoe slogfile for a recent instagoric perf run, I counted 25 bundles being passed to E(zoe).install(). I think three of them were static contracts, fixed here, and the other 22 are from the "install the economy" governance action which doesn't happen during bootstrap, but does happen right away within an instagoric run. Those 22 bundles won't shrink into caps/IDs until we've got client-side bundles-in-txns working.

[warner]

Re-reading my initial comments.. to finish this work, and make all the bootstrap-launched pseudo-static vats use bundlecaps instead of bundle names, we need to change makeVatsFromBundles() from:

      const vatInfo = E(svc).createVatByName(bundleName, { name: bundleName });

to something like:

      const bcap = await E(svc).getNamedBundleCap(bundleName);
      const vatInfo = E(svc).createVat(bcap, { name: bundleName });

Two wrinkles:

• The unit test environment may need to be augmented to support those calls (as it uses a mock VatAdminService)
• This exposes the bundlecaps to vat-bootstrap, which are device nodes, and we don't GC device nodes yet (some day..), which means vat-bootstrap will hold onto these forever. It already holds onto the new vat root objects forever, and vats hold onto their source bundles (at least until upgrade), so we're not actually retaining any additional storage. And some day we might manage to figure out how to drop device nodes during GC, so the user-level code won't need to change.

This second change doesn't save any RAM, because we aren't holding these full bundles in RAM at any point: both names and bundlecaps are tiny. So we don't really need to implement this any time soon. It's just a blocker for cleaning up vat-admin to stop supporting createVatByName.

[mhofman]

Likely a blocker for #6826
cc @gibson042

[warner]

Rereading this, I think we need to apply one of the changes explored here (in #4374 (comment)) to address #6826, but I think we don't need to implement the rest of this ticket (which remains low-priority). Having vatAdminService.createVatByName() is a bit of a wart, but it isn't causing any performance or memory problems, and the implementation internally converts the names to BundleIDs before creation, so my original concern ("We want vats to uniformly defined by a bundleID, so that the future upgrade protocol has a well-defined initial value to upgrade from") doesn't seem significant.

[warner]

We're going to kick this out out of the release.. I want to survey the code and understand how bundles are now being used, but the release isn't threatened by this not being done yet.