[Proposal] Split execution engine from rest of the runtime using process isolation

[dominic-mulligan-arm]

With Veracruz 2, we may assume that we have some operating system support within the Isolate, under which we are executing. I propose that we make use of this by splitting the Veracruz runtime into components which are isolated from each other using process-level isolation. In particular, the execution engine (the Wasm JIT/interpreter) is an obvious first choice for splitting off, given the potential security implications around JITting, with process-level isolation providing an additional layer of defence in the design of the runtime from an attacker who is able to subvert the JIT somehow.

Moreover, once the execution engine is split off from the rest of the runtime, we can experiment with making the runtime itself distributed. For example:

• We can experiment with a single enclave starting multiple concurrently-executing, but otherwise independent Wasm binaries. Here, the idea is we have a single long-lived Isolate that receives messages from the outside world, and spawns computations on-demand. Users then have a choice, when launching a Veracruz computation, between spawning a new Isolate to run a computation in, or spawning a new process within an existing Isolate. The ability to spawn multiple concurrently-executing computations can be regulated by new policy fields.
• Making computations themselves distributed within a single Isolate, in a similar fashion to the original vision of Google Oak. Now, computations can be split across multiple processes which may communicate with each other across sockets/channels, mediated by the main body of the Veracruz runtime which routes messages between the components that it has spawned.

[nspin]

A note from the perspective of low-TCB isolates (those not based on e.g. Linux).

For some low-TCB isolate solutions, it may be much easier to implement an isolate with a static number of processes, or a small lower bound known at build-time.

This shouldn't stop us from exploring richer structures within isolates, but it's worth keeping in mind.

Do the features you have in mind require dynamically spawning processes within the isolate?

[dominic-mulligan-arm]

I think for the initial split of the execution engine from the main body of the runtime, keeping the one isolate-for-one-computation invariant, a fixed bound (i.e. two) processes is all that is required.

For dynamically spawning concurrent computations, again I think a fixed number of processes could work: this bound would need to be reflected in the policy, and attempting to dynamically launch a new concurrent computation could fail if the pool of available processes is exhausted.

Does that work?

[nspin]

Any level of dynamism would work! My comment was really just about simplicity of implementation.

Choosing a reasonable upper bound at build time, and checking it against the policy early during runtime would work with a simple implementation.

[ShaleXIONG]

Just to add it up. The VFS can be spun out as a separate thread too. The VFS now is a shared and mutable (protected by a mutex) structure between the runtime and execution engine.

Further, the execution engine, native modules, and VFS can be all organised as separate processes and all treated as service on top of runtime.

[dominic-mulligan-arm]

That's a good idea, though as you point out you now need to make sure that the filesystem takes locks appropriately.

[ShaleXIONG]

The file system has a lock even now, just to satisfy the type system since we need a shared and mutable object. So it is ready to be separated out into a thread already. However, It is a big lock rather than a fine-grained lock on an individual inode, for example.

[nspin]

What process would the memory backing the VFS belong to? Are you suggesting that the VFS run as a service over IPC in its own process?

[ShaleXIONG]

Yes, I think VFS can be run as a service. I cannot really justify it is necessary or 100% beneficial tbh apart from that all the incoming data, programs, and results are isolated from the rest. Will need to investigate a bit more.