Numpy doesn't seem to work with wasmtime + VMWare's Python build

[JanPokorny]

Hello! I came across this project and tried to get it working with VMWare's WASM Python from https://github.com/vmware-labs/webassembly-language-runtimes.

I have downloaded python (version without bundled stdlib) and libpython releases from https://github.com/vmware-labs/webassembly-language-runtimes/releases/tag/python%2F3.12.0%2B20231211-040d5a6, and numpy from this repo's dev release, and unpacked it into the following dir structure:

Then I run it with wasmtime --dir=root/::/ python-3.12.0.wasm

However, it does not seem that numpy is importable:

Python 3.12.0 (tags/v3.12.0:0fb18b0, Dec 11 2023, 11:45:15) [Clang 16.0.0 ] on wasi
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy
Traceback (most recent call last):
  File "/numpy/core/__init__.py", line 24, in <module>
    from . import multiarray
  File "/numpy/core/multiarray.py", line 10, in <module>
    from . import overrides
  File "/numpy/core/overrides.py", line 8, in <module>
    from numpy.core._multiarray_umath import (
ModuleNotFoundError: No module named 'numpy.core._multiarray_umath'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/numpy/__init__.py", line 144, in <module>
    from . import core
  File "/numpy/core/__init__.py", line 50, in <module>
    raise ImportError(msg)
ImportError: 

IMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE!

Importing the numpy C-extensions failed. This error can happen for
many reasons, often due to issues with your setup or how NumPy was
installed.

We have compiled some common reasons and troubleshooting tips at:

    https://numpy.org/devdocs/user/troubleshooting-importerror.html

Please note and check the following:

  * The Python version is: Python3.12 from ""
  * The NumPy version is: "1.26.0b1"

and make sure that they are the versions you expect.
Please carefully study the documentation linked above for further help.

Original error was: No module named 'numpy.core._multiarray_umath'

>>>

I have tried different positions of the numpy folder (cwd, PYTHONPATH, site-packages etc.) but nothing seemed to help.

Am I using the numpy build correctly? Is there something more I need to do? Thank you!

[dicej]

Hi @JanPokorny. Thanks for reporting this.

The issue here is that Wasmtime doesn't have any built-in support for dlopen/dlsym, which is the underlying mechanism CPython uses to load native extensions. Part of the reason for that is there's currently no standard host interface for loading .so files at runtime. Emscripten has an ad-hoc interface for it, but there's not yet any equivalent for WASI runtimes like Wasmtime. It's possible that VMWare has rigged something up to support it, but I haven't had a chance to look at what they've done yet.

Consequently, the only way I know of to use these WASI builds is via componentize-py, which uses wit-component to bundle all the .so files into a single component which may be run on e.g. Wasmtime. Here's an example: https://github.com/bytecodealliance/componentize-py/tree/main/examples/matrix-math.

[JanPokorny]

@dicej Oh, I see! That's a crucial bit of info I was missing. I'll look into componentize-py to see if it covers my use case. Thank you for the help!