Consider adding a separate WASI runtime for build.rs isolation

[kazimuth]

This project is brilliant! I've wanted some way to pre-compile build tools like this for a while. I've got a fairly hefty feature request, though.

Currently, Watt is designed to completely sandbox the input. However, I've worked on several non-proc-macro build tools that need to access the filesystem (e.g. to locate non-Rust input files). These projects would still be amenable to pre-compilation; the output binaries don't need to link with them. Unfortunately, Watt currently can't address this use case, since it can only read and write TokenStreams.

It would be neat if there was a way to invoke Watt from a build.rs file. and allow it to access more of the external environment.

One route to implement this would be through WASI. WASI is a wasm "syscall" ABI -- a set of functions implemented in the wasm runtime, which access the external system. (It's defined here.) All that would be needed to support this would be to implement these functions and link them into the Watt runtime.

Rust code can then be compiled with the wasm32-wasi target, and standard library operations will then be routed through these "syscalls" in the compiled wasm module. So build tools could be written using standard Rust APIs, and transparently run through Watt.

You could still retain some sandboxing, since WASI is designed to be explicitly initialized with a set of capabilities -- for example, you could explicitly pass in the paths the build tool is allowed to access in your build.rs file.

You could also allow using WASI syscalls in the proc-macro runtime. I'm not sure if non-deterministic proc-macros are permitted by Rust, though.

Downsides of this approach: it would add some compile time to the runtime, and it adds some complexity.

[dtolnay]

Seems reasonable to me, though I would want that to be a different crate. In this repo the src directory is practically 100% specific to proc macros and you wouldn't want any of that code in a build.rs environment. The runtime directory is not at all specific to proc macros. We embed the runtime in the watt crate by doing:

watt/src/lib.rs

Lines 189 to 190 in c7d0197

	#[path = "../runtime/src/lib.rs"]
	mod watt;

A new crate providing a WASI environment would work the same way; embed the use-case-agnostic runtime and wire it up with whatever endpoints are required by the use case.

[alexcrichton]

I think that this would be a good idea in the sense that proc macros are a very natural fit for wasi, it's a well defined interface and gives proc macros access to things like env vars and stdout/stderr. (the latter one would be quite helpful for debugging)

I think that watt would probably want to just whitelist a small handful of symbols though, for example fd_write and random_get for stdout/stderr and libstd's hash maps. The random_get part could even be deterministic to keep it all deterministic!

When hooking up an optimized runtime (#2) this'll typically be baked in anyway too.

[mystor]

I would love to support basic std{out,err}-writing support in procedural macros as well, as they can be very handy for debugging. The idea of keeping the rest of the inputs deterministic also sounds very appealing here, although it might be good to heavily document that randomness in this environment is not secure.

[Kixunil]

@dtolnay wouldn't embedding runtime like that cause it to compile twice if a crate uses both proc macros and build.rs? Even if not wouldn't it be nicer/cleaner to have the runtime as a separate crate?