Use WASM commands running in webworker

[ianthomas23]

This is an experiment using POSIX commands compiled to WebAssembly via emscripten instead of the current implementation which uses commands written in bespoke TypeScript. Here is an example:

Screen.Recording.2024-06-24.at.15.04.22.mov

It includes all of coreutils as a single JavaScript module with embedded WASM. The build system for that is based on the marvellous biowasm. It uses its own emscripten filesystem so is not yet integrated with the JupyterLite file browser contents (edit: it is now). There is no support for input or output redirection, pipes, or setting env vars, or many other things yet.

I have temporarily removed the cockle dependency and just put the JS files (fs.js and coreutils.js) in this repo for demo purposes. Eventually they will be in cockle (or an equivalent) and we will need to use much of the existing cockle functionality for command-line editing and so on. The actual JS/WASM files are built by hand at this stage but obviously that needs to be automated.

Because of global state in each command, the code recreates a new instance of the coreutils module for each command. This takes about 30 ms the first time, and less that 20 ms thereafter on my dev system (see log messages in console). The separate fs.js module is just for the emscripten filesystem which lasts the duration of the terminal and is proxied into each command just before they are run.

This reuses the WebWorker code from PR #13.

[jtpio]

Thanks for looking into this!

The actual JS/WASM files are built by hand at this stage but obviously that needs to be automated.

Out of curiosity, what is the process for building these files?

[ianthomas23]

The actual JS/WASM files are built by hand at this stage but obviously that needs to be automated.

Out of curiosity, what is the process for building these files?

It is the classic UNIX configure and make combination but using emscripten. A bit like https://github.com/emscripten-forge/recipes/blob/main/recipes/recipes_emscripten/jpeg/build.sh but with more flags and some manual hacks modifications. And it is done in an Ubuntu docker container to build against the appropriate Linux system headers and with a particular version of the emsdk.

[jtpio]

OK thanks.

Would it be possible to post the commands on this PR, to be able to more easily build the files locally if needed?

[ianthomas23]

Would it be possible to post the commands on this PR, to be able to more easily build the files locally if needed?

Not yet, I need to do more on it before I am happy with it.

[ianthomas23]

This now connects to the file browser using a DriveFS.

Screen.Recording.2024-06-25.at.11.33.08.mov

It breaks some things such as cp creates the new file but doesn't copy the contents, but as I haven't dealt with input/output redirection yet this probably isn't surprising.

[ianthomas23]

I think this is ready now. After merging this there is lots of work to be done on cockle.