port virtual machine from NaCl to WebAssembly aka Wasm

[illwieckz]

May be useful: https://bytecodealliance.org/articles/announcing-the-bytecode-alliance

Wasm runtime list: https://github.com/appcypher/awesome-wasm-runtimes

[illwieckz]

There is also a list of Wasm runtimes there: https://gitlab.com/xonotic/xonotic/issues/244#sandboxing

[illwieckz]

@Kangz do you have any opinion on existing Wasm runtimes?

[illwieckz]

Dropping NaCl would also help us to drop the external deps download.

[DolceTriade]

It would change the external_deps download. The external_deps download downloads the NaCl toolchain and some common NaCl libraries. This would be instead changed to the wasm toolchain and the same libraries, but wasm...

[illwieckz]

I secretly hope we would be able one day to get rid of an SDK download at build time, to rely on system tools or git submodules instead.

To me that external_deps download is part of the NaCl issue.

[DolceTriade]

I mean, you can repackage the SDK as a submodule instead. I prefer having the build system "magically" ensure you have all the requisite dependencies.

[illwieckz]

I would prefer a submodule than what we currently do as I agree with people being concerned by random download done at CMake time. That said, it's better to invest time in Wasm and get rid of NaCl in the process than making NaCl a submodule.

Does using NaCl as a submodule would help Unvanquished to run outside of i386/amd64 or it's not only a limitation of our toolchain but a limitation of NaCl itself?

[ghost]

Nocrogiting: wasm works by way of LLVM, it would probably be more worthwhile to use that in the compile stack and leave wasm out of it. Same goes for rust-to-native I believe.

[Kangz]

After seeing @illwieckz's talk at MiniDebConf I'd be interested in taking this on to avoid relying on deprecated technologies (and eventually forking PNaCl). I haven't done an extensive comparison of runtimes but wasmer and wasmtime both seem fairly mature. I think an important selling point of wasmtime is that it supports debugging the webassembly module which will be quite useful to developers of games using Daemon.

One big difference between WASM and PNaCl will be that the gamelogic won't be in a separate process anymore but in a sandbox inside the engine's process. It's a bit of a security regression but there doesn't seem to be portable sandboxing libraries we can use (Chromium's source code doesn't count and it is very complex). For PNaCl we had to make most syscalls asynchronous, maybe being in process will allow moving back to synchronous calls for performance in some cases?

Another thing is that I don't know how well the WASM runtimes support spawning threads from inside WASM, not that our gamelogic does it at the moment, but it might be useful in the future.

[Kangz]

I also forgot to tag @Amanieu, WDYT about using wasmtime instead of PNaCl in particular security wise?

[illwieckz]

Hi @Kangz! Good to see you around! 🎉

@slipher started to work on this: https://github.com/slipher/Daemon/tree/wasm0 😉

As I remember you said in the past, Wasm was not that ready for our use case and as expected @slipher quickly faced a wall because of Wasm missing required mechanisms. Anyway, I remember @slipher said recently one of the missing feature that was blocking was implemented. I guess the reason why this branch is currently dormant is that we are fully focusing on 0.52 release right now (well, if it was not the reason, I would ask for it 😁).

So the best thing you can do is to get in touch with @slipher and see how you can make together something out of that branch. =)

Maybe we can start by moving this work-in-progress branch from @slipher's own tree to @DaemonEngine organization.

[ghost]

Having gained a lot of experience with WASM this still feels like a waste of energy to me. I am not sure what people think they are gaining by it. There is other talk about pulling in https://github.com/rui314/chibicc as the compiler to replace q3lcc if that is the goal. And using LLVM you get all of the benefit of cross compiling without any of the overhead of interpreters like you would need with WASM.

[illwieckz]

We already ditched LCC and are now doing C++ instead of just C. chibicc looks interesting (thanks for the share!) but seems to be C only. Also it would not be enough to solve the virtual machine problem (unless we revert to Q3VM which is very unlikely).

[Kangz]

We're gaining sandboxing and support for C++ and other languages (seems Xonotic was maybe going to translate qc to Rust?)

[illwieckz]

seems Xonotic was maybe going to translate qc to Rust?

Yes, there is two things experimented: one experiment is a Q1VM on the VM used by Dæmon, the other one is about translating qc to Rust. The second one would be the best one.

[ghost]

"We aren't going to switch back to Q3VM"
Now I understand what I was missing, I was assuming you could use LLVM from multiple languages to Q3VM format. Which is what QuakeJS does. But since you have already moved past it you would need a new "VM" replacement in this case WASM makes sense. Or using some intermediate language as your VM would be interesting like if you could import any .a or .asm or .o object file in to your sandbox, that could be really cool. Maybe even link multiple object files in to the same engine process, like one VM for movement and a separate VM for weapons, a third VM for world model, etc.

[illwieckz]

Ah, I see! I was also wondering if something was missing because I was like… LLVM can't be enough… 🤔 😁

[Amanieu]

I think WASM is the best way to have a portable sandbox for C/C++ code today. NaCl is unfortunately a dead end at this point.

[slipher]

In my prototype I decided to keep VMs as a separate process for a few reasons. For one, it will be possible to kill unresponsive VMs - there is no API for this otherwise. I won't have to worry about Daemon and the Rust/WASM runtimes stepping on each other in any way (for example, one of the WASM engines depends on controlling signal handlers). Finally it provides an extra layer of security with the process boundary preventing e.g. out-of-bounds writes in the engine from affecting the VM or vice versa.

I started off using Wasmer, but ran into a nasty bug. Then I switched to Wasmtime. The Wasmer bug was supposed to be fixed recently.

The biggest missing feature in standalone WASM runtimes was exception handling (it exists in browser WASM by integrating with Javascript exceptions somehow). But we hardly use exceptions in the gamelogic, so that is not a blocker. It was annoying to deal with use of setjmp/longjmp in freetype though.

There was no hard blocker for my previous work, but I lost momentum. I do plan to get back to it after an Unvanquished release.

[Kangz]

Thanks for the summary!

From looking at the wasmtime documentation it seems to be designed to run multiple WASM modules in the same process. I agree that keeping things in a different process would be ideal but it seems a bit more work to start with. What do you think of focusing on putting things in process first, then handle the more difficult cross-process IPC?

Like you said exception handling shouldn't matter. Freetype setjmp longjmp is a bit annoying. I'll ask around to see how important it is and how difficult it is to remove.

Another question I had an IRC, would using the WASI-SDK inject less magic than Emscripten? It seems it would be easier to control what happens.

[slipher]

I actually had it going with separate processes already. I found that the path of least resistance since I could reuse a lot of the NaCl code, and I didn't have to bother with creating a C interface for the Rust code. What still needs to be done is implement shared memory-based communication--I put everything over sockets at first for ease of implementation.

Another question I had an IRC, would using the WASI-SDK inject less magic than Emscripten?

That's a good point. I wanted to try the other toolchain eventually, but maybe sooner would be better than later. If I could just put in a dummy setjmp and longjmp, that would be easier than how it is with Emscripten where there are 20 different cryptically named functions that it expects to import.

[slipher]

Oh I just remembered what the actual biggest issue is - the shared memory block between sgame and server. There is no way to use shared memory directly from a VM (I asked), so that stuff will have to be reworked somehow. Now I remember why I was working on cgame first 😄

[Kangz]

Oh yeah that's a problem. Another reason to start with in-process VMs? This way we could directly get a pointer inside the WASM memory heap. Note that we're asking for similar things for WebGPU gpuweb/gpuweb#747 (comment) and talking to WASM folks it seemed the blocker was that Windows didn't allow unmapping virtual memory while still keeping it committed (so there was no guarantee someone would take the middle of the WASM heap if you unmapped it to mmap shmem into it) VirtualAlloc2 released in 2018 fixes that though. So maybe it will happen eventually?

[Tarek-Hasan]

If you are interested, there is RLBox, a general purpose sandboxing API that can be used to interact with library sandboxed with different backends --- WebAssembly, Native Client, libraries running in a separate process etc.
It's integrated into Firefox from version 95.

[slipher]

I suspect that for VMs we will want the highest possible degree of control, and so won't want to use a wrapper library. RLBox seems interesting for other parts of Daemon though, e.g. poorly tested image decoders

[slipher]

Stack unwinding update: Wasmer and Wasmtime still both lack C++ exception support. We could get by without C++ exceptions, but we cannot live without at least setjmp/longjmp support. This is because Lua uses longjmp whenever there is an error in a Lua script. I can't find any confirmation that either of the two runtimes supports setjmp/longjmp.

Previously I had encountered problems attempting to build Freetype for WASM due to lack of setjmp support, but this seemed surmountable as we could just provide a dummy implementation that would turn a font loading error into a fatal error and terminate the process. Having any Lua error be fatal for the gamelogic is far less palatable.