build(extension): Introduce our own module externalization logic for JavaScript entry points

[rekmarks]

Like most bundlers, vite is extremely clever. It aggressively optimizes everything it touches, which means that cannot let it touch modules like ses and @endo/lockdown. As it turns out, getting vite to ignore a specific module is more difficult than it should be. In #8, I tried relying on importing our "endoify" shim at the top of our TypeScript entry point files. This worked, but in #11 I observed vite "optimizing" the import order such that modules that need to run under lockdown were imported before lockdown was called by our shim.

In #11, I'm taking the approach to manually insert a <script src="endoify.mjs" /> in the correct place in our HTML files, which appears to be the surest (only?) way to accomplish what we want. However, background.ts doesn't have a corresponding HTML file, and it remains at risk of being destructively optimized. Therefore, we should expand on my work in #11 to introduce our own module externalization logic for JavaScript entry points. I recommend taking a similar approach that I took to HTML files, i.e. writing in the import statements at the beginning of the relevant file(s) at the last possible moment.

[grypez]

To clarify the intention of this issue, both for myself and others:

In general, plans (and thus programs) may be plotted on the two dimensions of consistency and progress. While it is possible to design a plan which is inconsistent and makes no progress, it is not possible to attain maximal consistency and maximal progress. Instead, the feasible limits of these two dimensions constitute a pareto frontier. In this repository, we aim for as much progress as possible without sacrificing any consistency, so that our plan should be as close as possible to the intersection of the pareto frontier with the consistency axis.

In particular, we require that our trusted computing base (which includes the ocap kernel) be endoified so that it may be robustly composed with untrusted code without compromising the consistency of our plan (which, in the case of MetaMask, includes maintaining users' control over and privacy of their identities). Our strategy for meeting this requirement is largely encapsulated in the @ocap/shims/endoify shim, but also includes maintaining the following strict import procedure for all javascript entrypoints in the trusted computing base:

1. Import trusted code
2. Import endoify
3. Import untrusted code

We may refer to the proper adherence to this procedure as endoification of the entrypoint. If the endoify shim is properly constructed, endoification defines the boundary between the code we trust (and to which we may one day entrust MetaMask users' security) and the code we merely use. A sufficient characterization of "trusted code" here is "code which must pass a security audit".

Provided the faithful endoification of our entrypoints, our ocap-kernel plans are reliably consistent. There are then many processes we may apply to improve our plan's position along the progress axis--tree-shaking, bundling, minification, etc.--but we must assure those processes do not interfere with endoification.

As exemplified here, external is a notion defined by rollup (and inherited by vite) which indicates that a file should remain untransformed by the bundler and should not be "rolled up" into the bundle. By declaring the endoify shim external, we prevent any transformations which may compromise its trustworthiness.

However, it is still possible for the bundling process to interfere with the endoification of the entrypoints by permuting the order of import statements in their corresponding bundles. Such a permutation could effectively move untrusted code into the trusted computing base, which our plans cannot tolerate. This issue aims to address that possibility with a strategy which is both effective and easily audited.

[grypez]

Broadly, my approach here is as follows.

• Compile the trusted imports of the entrypoint, which collectively form what we may refer to as the trusted header. The trusted header may be either
  • a) systematically identified or
  • b) declared in a separate module and associated to the entrypoint
• Ensure that the trusted header appears untransformed at the beginning of the bundled entrypoint

Regarding the trusted import compilation, option (a) has the advantage of operating on source code which fully declares its intent, an issue which @SMotaal has advocated for and which trivially supports typescript and other integrated development tools. Option (b) has the advantage of obviating the bundling process, and would allow for simple CI detection of changes to trusted headers, since they would be declared in separate files. Both development tool integration and CI detection should be possible with either approach; the core quantities affected are implementation effort and style.

Regarding (2), both options (a) and (b) require that the header be prepended to the bundled entrypoint, but in (1a) this will result in duplicate import statements later in the bundle, a consequence which is at best inelegant and at worst hazardous¹. Thus the removing the duplicate import statements is somewhere between nice and necessary for option (a).

The benefits of approaches (a) and (b) can be combined via the use of a barrel file matching a specific file name². In (c), the trusted header of a file foo.js is encapsulated in a module file foo-trusted-header.js which is declared external, and a vite plugin is written which ensures

1. That a file has no more than one trusted-header import
2. That the trusted-header import is the very first line of the output bundle

Files matching -trusted-header\.[cm]?[jt]s/ can be flagged for special review in the CI, and the source file will fully declare its intent and trivially integrate with existing development tools.

Footnotes

1. Although my understanding is that duplicate module import statements are effectively no-ops, my familiarity with the ECMAScript spec is incomplete and the nearest I can find to a proof of this behavior is the DFS resolution order and uniqueness requirement of the abstract InnerModuleLinking method. ↩

2. The vite documentation explicitly warns against the use of barrel files as they slow down the bundling process, but since the trusted-header barrel file (and its imported dependencies) would be declared as external, I think this is not the anti-pattern they are warning against. ↩