Unify xlite kernels

[DerThorsten]

Unify xlite kernels

This issue is just to track the difference in the multiple lite kernels and tried to collect ideas how to improve this.

The Problem

At the moment, the different lite kernels can differ in the following ways:

• with dlopen / side-main module :

  Kernels which need the capability to load shared libraries at runtime, need to be compiled with dlopen. On the emscripten side, this means that the kernel needs to be compiled with the -s MAIN_MODULE=1 flag, and the libraries to load need to be compiled with -s SIDE_MODULE=1 flag.

• with asyncify :

  We use asyncify exclusively to have blocking input requests. The problem with asyncify is that it makes compilation much more complicated, and it did not work with all kernels. In particular, it did not work with xpython.

• needs runtime files :

  Some interpreters need to load files from disk at runtime to work properly. For instance Python needs to load all the standard library modules (besides compiled ones) at runtime from
  the filesystem.

• needs custom initialization :

  For the python case, we not only need the "runtime files" at the right place, but also we
  need to handle all *so files in a special way.
  At the moment This is all very specific to python and we do not have a general API for that.
  This needs to be done before the kernel is used, ie at the initialization time.

Kernel	with dlopen / side-main module	with asyncify	needs runtime files	needs custom initialization
xpython	yes	no	yes	yes
xlua	no (but should be yes)	yes	no	no
xsqlite	no	no	no	no
xwren	no (but should be yes)	no	no	no
xrlang	??	??	??	??

These differences make it difficult to maintain the kernels.

How to Improve (Suggestions)

• we can just make all kernels use dlopen and side-main module. This might make some kernels a bit slower, but I believe it is worth it.

• just remove asyncify. For python, which is the most important one, we cannot use it anyhow. So we should just remove it and not have blocking requests in xeus-lua

• for the runtime files its a bit more complicated. For the Python case we install them via get micromamba, and pack them via empack (And we even allow to change these runtime files after compilation, ie the jupyterlite_xeus_python package allows end users to install custom packages for xpython lite kernels)).
  At the moment we do not have any API for that and just hacked something for the Python case. For python we do the following.
  We have a smth like a files.json file which contains a list of files, which are copied to the emscripten filesystem.
  This json and the actual files need to be served next to the *.wasm file. Then we have a special function in the python kernel which fetches the files from the server and copies them to the emscripten filesystem. This is all very specific to python and we do not have a general API for that.

• The custom initialization can be solved in a simple way. All kernels provide either a init.js with an async function async_initialize_xkernel(){...} or directly implement the async function initialize_xkernel() as part of the kernels Module (ie in some pre.js / post.js). This way, any custom initialization code will hidden behind this async_initialize_xkernel API / function.

[SylvainCorlay]

dlopen / side-main module :

I agree with your suggestion that we should enable this for all kernels. I think that it will become useful for other kernels as well eventually.

with asyncify :

Agree, let's get rid of asyncify.

runtime files :

This is an area where the current solution seems hacky. Why should it be Python specific after all? A Lua kernel could benefit from Conda packages being installed as well.

needs custom initialization :

Providing a hook as suggested may be a good way to kick to touch, until we figure out if more things become common to all kernels.

[DerThorsten]

runtime files :

This is an area where the current solution seems hacky. Why should it be Python specific after all? A Lua kernel could benefit from Conda packages being installed as well.

I did not mean it should be Python specific. Just the things we atm do for Python are not very generic, its a very Python/empack/emscripten-forge specific hack atm the moment.
Once we allow xlua to be compiled with side-modules we will need this also for lua

[SylvainCorlay]

runtime files :

This is an area where the current solution seems hacky. Why should it be Python specific after all? A Lua kernel could benefit from Conda packages being installed as well.

I did not mean it should be Python specific. Just the things we atm do for Python are not very generic, its a very Python specific hack atm the moment.
Once we allow xlua to be compiled with side-modules we will need this also for lua

Yep, we are on the same page 😄

[DerThorsten]

The best prototype so far is using https://github.com/DerThorsten/xeus-lite-kernel-loader-extension/.
It assumes the wasm kernels are installed next to the kernelspec, ie:

• $PREFIX/share/jupyter/kernels/<kernel_name>/<wasm_name>.wasm
• $PREFIX/share/jupyter/kernels/<kernel_name>/<wasm_name>.wasm
• $PREFIX/share/jupyter/kernels/<kernel_name>/kernel.json
• $PREFIX/share/jupyter/kernels/<kernel_name>/logo-32x32.png
• $PREFIX/share/jupyter/kernels/<kernel_name>/logo-64x64.png

We create a kernels.json file to list all kernels and put this file and the kernels into the _output directory of lite
st we can find all kernels in lite at runtime, by fetching the kernels.json file and creating a kernel extension
for each of the kernels.

Change / PRs in other xeus repos needed for that:

• xeus unify wasm options xeus#372
• xeus-lite unify-wasm-kernels #9
• xeus-python unify wasm kernels xeus-python#604
• xeus-lua unify wasm kernels xeus-lua#46
• xeus-sqlite Unify wasm kernels xeus-sqlite#137

For the kernels, the changes are mostly the following:

• adding a wasm_name field to the metadata of the kernel spec (used to load the *.wasm / *.js files)
• using only the simplified options provided by unify wasm options xeus#372

To land this we need to:

• merge all the PRs
• create releases / tags in all repos
• create / update emscripten-forge packages for all repos above
• remove hacks from xeus-lite-kernel-loader-extension and make it user-friendly
• update the xeus-cookiecutter

[DerThorsten]

all needed changed landed into the most relevant kernels. Therefore this can be closed