controller.installBundle()

[rowgraus]

(by warner):

The task here is for swingset to expose a controller.installBundle() API. This will deliver a code bundle into the kernel, and make the associated bundleID (hash) available for use within the vats.

#4396 is a separate task is for cosmic-swingset to expose this to transactions.

[dckc]

see also #2391 and discussion of blobcaps

[warner]

The idea here is a consensus-visible cosmos-sdk transaction message type which adds a blob of data to a store, indexed by its hash. This txn will require a fee that is independent of the usual "add something to the run-queue" message fee, and possibly substantial: it is intended to both cover the eternal costs of retaining the blob, and to discourage the use of scarce+expensive chain/validator resources as a general-purpose data store.

The blob is meant to hold a JavaScript module source file (or possibly a whole package worth of modules). It would be appropriate (and probably preferable) to impose a requirement that the blob must be parseable as JavaScript, and perhaps even semantically valid, as this would futher discourage its use as a general-purpose data store.

A second message type will define a bundle, as a manifest which references a number of module/package blobs by their hash index. This message will only be accepted if all of the referenced modules/packages already exist. The associated "bundle hash" (hash of the manifest data) will be used as the index.

Once accepted into a block, the bundle index will be eligible for use by the "bundle device": vats can call vatAdmin~.createVat(bundleIndex) to create a new vat from the given bundle identifier. This call will fail if the manifest has not been installed by the time the createVat is delivered.

So far, this only provides a way to install dynamic vat code (i.e. a bundle which exports buildRootObject) into a running chain. The Agoric chain won't really offer this functionality: the only dynamic vats we create all start with the ZCF bundle. Later, after ZCF is running, it loads some particular contract bundle (which exports start) to be differentiated into a specific contract installation.

A second mechanism (not sure how it should work yet) should be used to get the contract's bundle index into a place where it can be efficiently evaluated into the running ZCF vat. That can live in a separate ticket.

[warner]

I'm currently implementing this:

• a "bundle" is an object, with moduleFormat: and some format-specific properties, where endoZipBase64 is the only one accepted
• a "bundleID" is a string, b1- concatenated with the lowercase hex-encoded SHA512 hash of the "compartment map" file
• a valid bundle matches that hash, contains components for everything named in the compartment map, those components match the hash named in the compartment map, and contains no spurious components
• a "bundleCap" is a swingset device node (or more commonly an imported reference to such a device node) with a single D(bundleCap).getBundle() -> bundle method
• controller.installBundle(allegedBundleID, bundle) -> bundleID or throws is how you install a bundle at runtime, from the outside of the kernel
  • this throws an error if the bundle is invalid or doesn't match the alleged bundleID
  • (the first version of our implementation won't perform validation)
• a new "bundle device" will create and host bundleCap device nodes
• D(devices.bundle).getBundleCap(bundleID) -> bundleCap turns the hash into a bundleCap
  • this will throw an error if nobody did controller.installBundle first
• E(vatAdmin).createVat(bundleCap) -> { root, controlFacet } turns bundleCaps into dynamic vats
• importBundle(D(bundleCap).getBundle()) -> Promise<namespace> evaluates the bundle and gives you the resulting namespace, e.g. for when ZCF loads a contract bundle

The way this should integrate with our bootstrap mechanism:

• the config object passed to initializeSwingset can currently contain bundles: { name : spec } (where you might see a spec.bundleSpec filename for pre-generated bundles on disk, spec.sourceSpec for the filename of the entry-point module that will be bundled during initialization, or spec.bundle for the full contents of the bundle already in RAM
  • with this change, all three cases would cause the kernelDB to be updated as if controller.installBundle was called, and the name/bundleID pair is added to a table called namedBundleIDs
• initializeSwingset currently enqueues the bootstrap invocation as E(bootstrapRoot).bootstrap(vats, devices)
  • with this change, we add a third argument, bootstrap(vats, devices, namedBundleIDs), which receives a simple object with bundle names as property names, and bundleIDs as their values
• the "core bootstrap" work (separate production vat bootstrap from testing mechanisms #4165) should have bootstrap() walk namedBundleIDs, convert each bundleID into a bundleCap with D(devices.bundle).getBundleCap(bundleID), and use that bundleCap to create the initial vats
• the ZCF bundleCap should be acquired by bootstrap(), then passed to Zoe in some sort of init() message
  • Zoe should do E(vatAdmin).createVat(zcfBundleCap) instead of createVatByName('zcf')
  • eventually we should pass zcfBundleCap into Zoe via the vatParameters, however these cannot yet contain caps of any sort (vatParameters are limited to JSON-serializable data)

Eventually, as we integrate importBundle into some lower layers of the vat worker, I hope to remove the need for D(bundleCap).getBundle(), and do something closer to importBundle(bundleCap), which would remove the last place where a large bundle string appears in vat transcripts, but that will require more extensive changes.

At the cosmic-swingset layer, we'll need a new cosmos-sdk transaction type to install a bundle. The arguments should include the alleged bundleID (hash) and the serialized bundle itself. The swingset module should call controller.installBundle() and expect a non-error response. Once that transaction is included in a block, it is safe for an ag-solo swingset to send E(zoe).install(bundleID), and Zoe can retrieve the bundleCap. If Zoe gets an error from D(bundles).getBundleCap(bundleID), the install() fails.

@michaelfig and I are still discussing what the contract-author -visible workflow should be. I'm in favor of something like agoric install-contract ./entry.js, which would find the entry-point file, run bundleSource() on it, compute the bundleID, then use a helper program (ag-cosmos-helper?) to build/sign/broadcast/await the txn that includes that bundle and installs it into the chain. That command would print the bundleID, which could then be copied into a deploy script that does the E(zoe).install(). Or, the zoe.install is integrated into the txn handler, along with a board write, so agoric install-contract could finish with the board ID for the installation, after which anybody wanting to deploy an instance could start with the board ID.

I think he's more in favor of having a deploy script drive everything, giving it an authority to invoke bundleSource and also invoke the txn build/sign/broadcast function. I know that contract instantiation really wants to happen from a deploy script, because each instance has unique objects (private owner facets, etc) and contract owners will need to stash references to those objects somewhere safe. But I'm thinking/hoping that mere installation does not, and could be driven from a command-line tool, rather than requiring someone to write code that performs the installation.

cc @dckc and @michaelfig on the "core bootstrap" integration, @rowgraus on how our contract installation pathway might compare to those of other projects, and @erights on general principles.

[michaelfig]

I think he's more in favor of having a deploy script drive everything

I don't presume to say your helper program is unnecessary, just that it's easier to create a composable API and then use it from within a shell command (such as agoric install-contract) than it is to create a shell command and then try to use it from JS. The latter basically guarantees that I can't get what I want: I can write most things as a JS deploy script, but when it comes to agoric install-contract, I'm forced to express everything as exec with command-line options and I/O.

It always bothered me that there were two deploy scripts necessary for nearly every dapp (the first of which was only necessary sometimes) and that they shared state via the filesystem. With the ability to have a bundle installation be idempotent, it is now possible to have just one script to execute when iterating on your dapp.

[michaelfig]

rather than requiring someone to write code that performs the installation

Nobody actually writes code nowadays. They just copy and modify.

[warner]

it's easier to create a composable API and then use it from within a shell command (such as agoric install-contract) than it is to create a shell command and then try to use it from JS.

Ah, I'll totally agree with that. I guess it comes down to what we're asking developers to do at the end (middle?) of their process, when they're ready to deploy. Heroku lets me git push, even though I'm sure there's an API that I could use from some more-sophisticated deployment process that I could build. Definitely in the category of "simple things should be easy, complex things should be possible", and building the CLI tool on top of the API is absolutely the way to enable that.

I guess it also relates to what mental model we want developers to build. Having a CLI tool to install a contract feels (to me) like it would look simpler, that it would give the message "hey, isn't this easy?", and I can imagine a screencast showing the process. Having the developer write a program to use the API feels like more work, to me. But on the other hand, that imaginary video might look simpler just because it ends before the actually-interesting part: instantiating the contract, where there are object references that need to be stashed and used by more code later. A demo which uses the CLI to do the part that doesn't require unique code might be sending the wrong message if the full workflow really does require the developer to think about the program they must write (or copy-paste) to do something with the resulting control facets.

Hm, I guess this is also a wallet question. If we expect some GUI affordances to manage the contract you just instantiated, then we need a way to link up 1: a directory of source code that you've written/edited/cloned, 2: a possible CLI tool invocation that triggers something, 3: an ag-solo instance that can act as your agent, and 4: the wallet GUI. If the problem were simpler, I'd lean towards the CLI tool on top of an API for more complex things. But I don't know how the wallet should be involved.

[Tartuffo]

@warner After various meetings, this ended up with a MN-1 tag AND in the Product Backlog pipeline (not MN-1 Backlog). Please modify appropriately for what it should actually be.

[Tartuffo]

@warner please create a separate ticket for the cosmic-swingset part of this, and remove that label from this.