Wasm hardware support (including 32-bit like Windows i686 and Linux armhf)

[illwieckz]

This is a parent thread to follow outside work on Wasm as we investigate a migration from Native Client (NaCl) to WebAssembly (Wasm).

Platform support:

platform	dæmon	naclsdk	nacl	wasisdk	wasmtime	wamr	wasmtime request
amd64	✅ yes	✅ yes	✅ yes	✅ yes	✅ yes	✅ yes
i686	✅ yes	✅ yes	✅ yes	?	no	✅ yes	bytecodealliance/wasmtime#1980
arm64	✅ yes	no	no	?	✅ yes	✅ yes
armhf	✅ yes	no	✅ yes	?	no	✅ yes	bytecodealliance/wasmtime#1173
riscv64	?	no	no	?	✅ yes	✅ yes
ppc64el	?	no	no	?	no	no	bytecodealliance/wasmtime#1183

Note: the NaCl vm also supports mips (not the naclsdk) and wasmtime supports s390x but we probably have no usage for that.

See this wasmtime label for new targets being discussed:

• https://github.com/bytecodealliance/wasmtime/labels/cranelift%3Anew-target

We may also investigate the WebAssembly micro runtime (WAMR), WAMR runs on amd64, i686, arm64, armhf, riscv64 architectures at least, and on Linux, Windows, macOS, and Android systems at least.

---

This first post focus on 32-bit (what motivated this thread), but other architectures can be discussed.

As WebAssembly is the candidate of choice to replace NaCl, it is interesting to look at what platform it supports. Tracking this is low priority but it's not a bad idea to gather the related knowledge and have a place to track this in a dedicated place.

The NaCl stack we currently use to run the sandboxed game code provides ability to build and run 64-bit amd64 and 32-bit i686, armhf and mips binaries (I doubt there is a single compatible mips platform though). There may be a 64-bit arm64 support in NaCl repository, but a build for it haven't been found yet. So, from that we have working amd64, i386 and armhf.

The Wasmtime runtime provides support for 64-bit amd64, arm64, riscv64 and s390x (I doubt we would have a single s390x user though, and it's too early for riscv64). There is no supported 32-bit platform at all. So from that we may hope to get working amd64 and arm64.

Among existing 32-bit platform around, the major interests are in:

• Windows i686, Windows users who got a 32-bit Windows typically don't replace it with a 64-bit Windows. A lot of computers pre-installed with 32-bit Windows were sold very lately, even if running on 64-bit compatible hardware.
• Linux armhf, usage of 32-bit armhf Linux distributions is very common arm64 boards. For example the default and recommended operating system download for the well-known raspberry pi is 32-bit, even if the CPU is compatible 64-bit and there are 64-bit operating system alternate download available. Some proprietary drivers for some GPUs commonly found on arm64 boards may even be 32-bit only.

As @necessarily-equal wrote in Unvanquished/Unvanquished#1015

https://docs.wasmtime.dev/stability-platform-support.html#jit-compiler-support they seem to say 32 bits support is "planned". The compilation error that you have seen was apparently introduced by bytecodealliance/wasmtime#5226 or a similar PR. It seems the hard part of doing this isn't in wasmtime, but in cranelift.

Adding 32 bits support would be tracked in bytecodealliance/wasmtime#1980 for wasmtime, but no one seems to be working on it at the moment. I honestly doubt someone will feel motivated enough to add support for that, so I'd rather plan the "no 32 bit support" case than the other way around.

Here is the wasmtime ticket to track 32-bit arm support: bytecodealliance/wasmtime#1173
They're welcoming any 32-bit arm implementation but had to remove an half-implemented one because it was unmaintained: bytecodealliance/wasmtime#3721

[illwieckz]

About Wasm and 32-bit, I read a very interesting comment on WebAssembly design tracker (comment from 2021):

It is definitely true that almost all new sold Android devices are 64-bit (although last time I looked in 2020, there were surprisingly still new 32-bit Android devices coming out in India and Pakistan markets), and we do also see this in our analytics. For the overall marketshare, in our analytics we do not reach a 90% prevalence count, and actually not even a 80% bar.

However the actual issue is that Chrome on Android has not yet completed its 64-bit transition. According to this source, the requirements for getting an automatic update to 64-bit Chrome is to have Android 10 with at least 8GB of RAM on the device. […]

The average gaming phones that are sold to the market in 2021 have just 4GB of RAM. […] Based on the above Chromium blog post paired with our analytics, I do not expect 64-bit Chrome overtaking 90% adoption rate within the next three years at least with the current trajectory.

So, while we don't run Android, it gives us a glance on what kind of hardware platforms Wasm is expected to run. The same way there was actually 32-bit versions Windows 10 being distributed (even if hardware can do 64-bit), there are 32-bit Android being distributed, or even 32-bit Chrome (even if hardware can do 64-bit), sometime just to save on memory.

So the same way we need to ship i686 build of our game for 32-bit Windows 10, some others may have to port Wasm to 32-bit arm for running web apps on phones.

[necessarily-equal]

While this is interesting, it kind of suggest the only reason 32 bits wasm is available is to save memory, or cheap android phone. We don't want to support cheap android phones; and running 32 bits wasm executable is actually the wasmtime default anyway bytecodealliance/wasmtime#572, bytecodealliance/wasmtime#3153.

"Runs 32 bits executables" and "Runs on a 32 bits CPU" are very different problems. And for our usecase we don't need to care about 64 bits devices equipped only with 32 bits runtimes because we own the runtime.

[illwieckz]

"Runs 32 bits executables" and "Runs on a 32 bits CPU" are very different problems. And for our usecase we don't need to care about 64 bits devices equipped only with 32 bits runtimes because we own the runtime.

I don't get this. There is no wasmtime runtime to build or download to redistribute, so we cannot distribute a 32-bit wasmtime runtime on Windows i686 (Like on 32-bit Windows-10) or Linux armhf (like on Raspberry Pi default distro), meaning we cannot distribute an engine on those systems.

[illwieckz]

WAMR (WASM micro runtime) may be a better candidate than Wasmtime: it supports all platforms the engine currently run on (and even more), it has more hardware compatibility than both NaCl and Wasmtime.

I found that information there: bytecodealliance/wasmtime#6675 (comment)

Basically WAMR runs on amd64, i686, arm64, armhf, riscv64 architectures at least, and on Linux, Windows, macOS, and Android systems at least.

I updated the table in first post.

[illwieckz]

I built the toolchain and the virtual machine of WAMR, and I modified their benchmark script to also run coremark on NaCl, here is the results, comparing native to NaCl to Wasm (WAMR):

platform	ticks	time	iterations/s	performance loss compared to native
native (x86_64)	13379	13.379000	29897.600718	0%
NaCl nacl_loader (x86_64)	19327	19.327000	20696.435039	-30.78%
WAMR iwasm (aot)	16776	16.776000	23843.586075	-20.25%
WAMR iwasm (aot-segue)	14890	14.890000	26863.666891	-10.15%
WAMR iwasm (interpreter)	214804	214.804000	1862.162716	-93.77%

More iterations per second means the executable runs faster.

Details provided by coremark when running (common to all runs):

2K performance run parameters for coremark.
Iterations       : 400000

When running coremark, Wasm on WAMR is faster than NaCL.