TA-Lib first problem

[svax974]

A bit new to all this, I'd like to try to get ta-lib-python (and others soon) to wasm.

I'm not sure about the feasibility for that one but I'm getting immediately a first error.

I'm on a fresh ubuntu 22.04 minimal server install, installed python-wasm-sdk 3.11 binary package and of course loaded the wasm32-mvp-emscripten-shell.sh to set all the env stuff.

Here is what I get, pretty long but seems to complain module _posixsubprocess not being found.
Maybe someone knows how to handle this, thanks !

`
[PyDK:wasm] /home/steph/Dev/sources/ta-lib-python $ make
/opt/python-wasm-sdk/python3-wasm setup.py build_ext --inplace
/home/steph/Dev/sources/ta-lib-python/setup.py:77: UserWarning: Cannot find ta-lib library, installation may fail.
warnings.warn('Cannot find ta-lib library, installation may fail.')
/opt/python-wasm-sdk/devices/x86_64/usr/lib/python3.11/site-packages/setuptools/installer.py:27: SetuptoolsDeprecationWarning: setuptools.installer is deprecated. Requirements should be satisfied by a PEP 517 installer.
warnings.warn(
ERROR: Exception:
Traceback (most recent call last):
File "/opt/python-wasm-sdk/devices/x86_64/usr/lib/python3.11/site-packages/pip/_internal/cli/base_command.py", line 160, in exc_logging_wrapper
status = run_func(*args)
^^^^^^^^^^^^^^^
File "/opt/python-wasm-sdk/devices/x86_64/usr/lib/python3.11/site-packages/pip/_internal/cli/req_command.py", line 247, in wrapper
return func(self, options, args)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/python-wasm-sdk/devices/x86_64/usr/lib/python3.11/site-packages/pip/_internal/commands/wheel.py", line 170, in run
requirement_set = resolver.resolve(reqs, check_supported_wheels=True)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/python-wasm-sdk/devices/x86_64/usr/lib/python3.11/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 92, in resolve
result = self._result = resolver.resolve(
^^^^^^^^^^^^^^^^^
File "/opt/python-wasm-sdk/devices/x86_64/usr/lib/python3.11/site-packages/pip/_vendor/resolvelib/resolvers.py", line 481, in resolve
state = resolution.resolve(requirements, max_rounds=max_rounds)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/python-wasm-sdk/devices/x86_64/usr/lib/python3.11/site-packages/pip/_vendor/resolvelib/resolvers.py", line 348, in resolve
self._add_to_criteria(self.state.criteria, r, parent=None)
File "/opt/python-wasm-sdk/devices/x86_64/usr/lib/python3.11/site-packages/pip/_vendor/resolvelib/resolvers.py", line 172, in _add_to_criteria
if not criterion.candidates:
File "/opt/python-wasm-sdk/devices/x86_64/usr/lib/python3.11/site-packages/pip/_vendor/resolvelib/structs.py", line 151, in bool
return bool(self._sequence)
^^^^^^^^^^^^^^^^^^^^
File "/opt/python-wasm-sdk/devices/x86_64/usr/lib/python3.11/site-packages/pip/_internal/resolution/resolvelib/found_candidates.py", line 155, in bool
return any(self)
^^^^^^^^^
File "/opt/python-wasm-sdk/devices/x86_64/usr/lib/python3.11/site-packages/pip/_internal/resolution/resolvelib/found_candidates.py", line 143, in
return (c for c in iterator if id(c) not in self._incompatible_ids)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/python-wasm-sdk/devices/x86_64/usr/lib/python3.11/site-packages/pip/_internal/resolution/resolvelib/found_candidates.py", line 47, in _iter_built
candidate = func()
^^^^^^
File "/opt/python-wasm-sdk/devices/x86_64/usr/lib/python3.11/site-packages/pip/_internal/resolution/resolvelib/factory.py", line 206, in _make_candidate_from_link
self._link_candidate_cache[link] = LinkCandidate(
^^^^^^^^^^^^^^
File "/opt/python-wasm-sdk/devices/x86_64/usr/lib/python3.11/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 297, in init
super().init(
File "/opt/python-wasm-sdk/devices/x86_64/usr/lib/python3.11/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 162, in init
self.dist = self._prepare()
^^^^^^^^^^^^^^^
File "/opt/python-wasm-sdk/devices/x86_64/usr/lib/python3.11/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 231, in _prepare
dist = self._prepare_distribution()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/python-wasm-sdk/devices/x86_64/usr/lib/python3.11/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 308, in _prepare_distribution
return preparer.prepare_linked_requirement(self._ireq, parallel_builds=True)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/python-wasm-sdk/devices/x86_64/usr/lib/python3.11/site-packages/pip/_internal/operations/prepare.py", line 491, in prepare_linked_requirement
return self._prepare_linked_requirement(req, parallel_builds)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/python-wasm-sdk/devices/x86_64/usr/lib/python3.11/site-packages/pip/_internal/operations/prepare.py", line 577, in _prepare_linked_requirement
dist = _get_prepared_distribution(
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/python-wasm-sdk/devices/x86_64/usr/lib/python3.11/site-packages/pip/_internal/operations/prepare.py", line 69, in _get_prepared_distribution
abstract_dist.prepare_distribution_metadata(
File "/opt/python-wasm-sdk/devices/x86_64/usr/lib/python3.11/site-packages/pip/_internal/distributions/sdist.py", line 48, in prepare_distribution_metadata
self._install_build_reqs(finder)
File "/opt/python-wasm-sdk/devices/x86_64/usr/lib/python3.11/site-packages/pip/_internal/distributions/sdist.py", line 118, in _install_build_reqs
build_reqs = self._get_build_requires_wheel()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/python-wasm-sdk/devices/x86_64/usr/lib/python3.11/site-packages/pip/_internal/distributions/sdist.py", line 95, in _get_build_requires_wheel
return backend.get_requires_for_build_wheel()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/python-wasm-sdk/devices/x86_64/usr/lib/python3.11/site-packages/pip/_internal/utils/misc.py", line 701, in get_requires_for_build_wheel
return super().get_requires_for_build_wheel(config_settings=cs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/python-wasm-sdk/devices/x86_64/usr/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_impl.py", line 166, in get_requires_for_build_wheel
return self._call_hook('get_requires_for_build_wheel', {
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/python-wasm-sdk/devices/x86_64/usr/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_impl.py", line 321, in _call_hook
raise BackendUnavailable(data.get('traceback', ''))
pip._vendor.pyproject_hooks._impl.BackendUnavailable: Traceback (most recent call last):
File "/opt/python-wasm-sdk/devices/x86_64/usr/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 77, in _build_backend
obj = import_module(mod_path)
^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/python-wasm-sdk/devices/emsdk/usr/lib/python3.11/importlib/init.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "", line 1206, in _gcd_import
File "", line 1178, in _find_and_load
File "", line 1128, in _find_and_load_unlocked
File "", line 241, in _call_with_frames_removed
File "", line 1206, in _gcd_import
File "", line 1178, in _find_and_load
File "", line 1149, in _find_and_load_unlocked
File "", line 690, in _load_unlocked
File "", line 940, in exec_module
File "", line 241, in _call_with_frames_removed
File "/tmp/pip-build-env-01iu6tgy/overlay/lib/python3.11/site-packages/setuptools/init.py", line 10, in
import distutils.core
File "/opt/python-wasm-sdk/devices/emsdk/usr/lib/python3.11/distutils/core.py", line 16, in
from distutils.dist import Distribution
File "/opt/python-wasm-sdk/devices/emsdk/usr/lib/python3.11/distutils/dist.py", line 19, in
from distutils.util import check_environ, strtobool, rfc822_escape
File "/opt/python-wasm-sdk/devices/emsdk/usr/lib/python3.11/distutils/util.py", line 15, in
from distutils.spawn import spawn
File "/opt/python-wasm-sdk/devices/emsdk/usr/lib/python3.11/distutils/spawn.py", line 11, in
import subprocess
File "/opt/python-wasm-sdk/devices/emsdk/usr/lib/python3.11/subprocess.py", line 104, in
from _posixsubprocess import fork_exec as _fork_exec
ModuleNotFoundError: No module named '_posixsubprocess'

Traceback (most recent call last):
File "/opt/python-wasm-sdk/devices/x86_64/usr/lib/python3.11/site-packages/setuptools/installer.py", line 82, in fetch_build_egg
subprocess.check_call(cmd)
File "/opt/python-wasm-sdk/devices/emsdk/usr/lib/python3.11/subprocess.py", line 413, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/opt/python-wasm-sdk/devices/x86_64/usr/bin/python3.11', '-m', 'pip', '--disable-pip-version-check', 'wheel', '--no-deps', '-w', '/tmp/tmptm0lmmy4', '--quiet', 'numpy']' returned non-zero exit status 2.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/home/steph/Dev/sources/ta-lib-python/setup.py", line 146, in
setup(
File "/opt/python-wasm-sdk/devices/x86_64/usr/lib/python3.11/site-packages/setuptools/init.py", line 86, in setup
_install_setup_requires(attrs)
File "/opt/python-wasm-sdk/devices/x86_64/usr/lib/python3.11/site-packages/setuptools/init.py", line 80, in _install_setup_requires
dist.fetch_build_eggs(dist.setup_requires)
File "/opt/python-wasm-sdk/devices/x86_64/usr/lib/python3.11/site-packages/setuptools/dist.py", line 875, in fetch_build_eggs
resolved_dists = pkg_resources.working_set.resolve(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/python-wasm-sdk/devices/x86_64/usr/lib/python3.11/site-packages/pkg_resources/init.py", line 789, in resolve
dist = best[req.key] = env.best_match(
^^^^^^^^^^^^^^^
File "/opt/python-wasm-sdk/devices/x86_64/usr/lib/python3.11/site-packages/pkg_resources/init.py", line 1075, in best_match
return self.obtain(req, installer)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/python-wasm-sdk/devices/x86_64/usr/lib/python3.11/site-packages/pkg_resources/init.py", line 1087, in obtain
return installer(requirement)
^^^^^^^^^^^^^^^^^^^^^^
File "/opt/python-wasm-sdk/devices/x86_64/usr/lib/python3.11/site-packages/setuptools/dist.py", line 945, in fetch_build_egg
return fetch_build_egg(self, req)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/python-wasm-sdk/devices/x86_64/usr/lib/python3.11/site-packages/setuptools/installer.py", line 84, in fetch_build_egg
raise DistutilsError(str(e)) from e
distutils.errors.DistutilsError: Command '['/opt/python-wasm-sdk/devices/x86_64/usr/bin/python3.11', '-m', 'pip', '--disable-pip-version-check', 'wheel', '--no-deps', '-w', '/tmp/tmptm0lmmy4', '--quiet', 'numpy']' returned non-zero exit status 2.
make: *** [Makefile:5: build] Error 1
`

[svax974]

Even if it didn't seem related, I installed the numpy 3.11 package from https://github.com/pygame-web/archives/tree/main/repo/pkg because numpy is a talib dependency and now get this :

`
[PyDK:wasm] /home/steph/Dev/sources/ta-lib-python $ make
/opt/python-wasm-sdk/python3-wasm setup.py build_ext --inplace
/home/steph/Dev/sources/ta-lib-python/setup.py:77: UserWarning: Cannot find ta-lib library, installation may fail.
warnings.warn('Cannot find ta-lib library, installation may fail.')
Traceback (most recent call last):
File "/opt/python-wasm-sdk/devices/emsdk/usr/lib/python3.11/site-packages/numpy/core/init.py", line 23, in
from . import multiarray
File "/opt/python-wasm-sdk/devices/emsdk/usr/lib/python3.11/site-packages/numpy/core/multiarray.py", line 10, in
from . import overrides
File "/opt/python-wasm-sdk/devices/emsdk/usr/lib/python3.11/site-packages/numpy/core/overrides.py", line 6, in
from numpy.core._multiarray_umath import (
ModuleNotFoundError: No module named 'numpy.core._multiarray_umath'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/steph/Dev/sources/ta-lib-python/setup.py", line 146, in
setup(
File "/opt/python-wasm-sdk/devices/x86_64/usr/lib/python3.11/site-packages/setuptools/init.py", line 87, in setup
return distutils.core.setup(**attrs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/python-wasm-sdk/devices/x86_64/usr/lib/python3.11/site-packages/setuptools/_distutils/core.py", line 172, in setup
ok = dist.parse_command_line()
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/python-wasm-sdk/devices/x86_64/usr/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 474, in parse_command_line
args = self._parse_command_opts(parser, args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/python-wasm-sdk/devices/x86_64/usr/lib/python3.11/site-packages/setuptools/dist.py", line 1107, in _parse_command_opts
nargs = _Distribution._parse_command_opts(self, parser, args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/python-wasm-sdk/devices/x86_64/usr/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 533, in _parse_command_opts
cmd_class = self.get_command_class(command)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/python-wasm-sdk/devices/x86_64/usr/lib/python3.11/site-packages/setuptools/dist.py", line 950, in get_command_class
return self.cmdclass[command]
~~~~~~~~~~~~~^^^^^^^^^
File "/home/steph/Dev/sources/ta-lib-python/setup.py", line 100, in getitem
import numpy
File "/opt/python-wasm-sdk/devices/emsdk/usr/lib/python3.11/site-packages/numpy/init.py", line 144, in
from . import core
File "/opt/python-wasm-sdk/devices/emsdk/usr/lib/python3.11/site-packages/numpy/core/init.py", line 49, in
raise ImportError(msg)
ImportError:

IMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE!

Importing the numpy C-extensions failed. This error can happen for
many reasons, often due to issues with your setup or how NumPy was
installed.

We have compiled some common reasons and troubleshooting tips at:

https://numpy.org/devdocs/user/troubleshooting-importerror.html

Please note and check the following:

• The Python version is: Python3.11 from "/opt/python-wasm-sdk/devices/x86_64/usr/bin/python3.11"
• The NumPy version is: "1.22.4"

and make sure that they are the versions you expect.
Please carefully study the documentation linked above for further help.

Original error was: No module named 'numpy.core._multiarray_umath'

make: *** [Makefile:5: build] Error 1
`

[svax974]

In fact just importing numpy produces that output even with that simple script :
import os print("PYTHONPATH:", os.environ.get('PYTHONPATH')) print("PATH:", os.environ.get('PATH')) import numpy inReal = numpy.array([1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0, 9.0, 10.0])

Any idea ? This might be a environnement/setup problem but I do not see what I could do.
Thanks

[svax974]

No good news but a little update : I compiled numpy 1.24.2 with python3-wasm and get the very same result.

[svax974]

@pmp-p Can you tell me how should I install a module I got from your wasm compiled repo ?
I tried python3-wasm -m pip install numpy.whl and python3-wasm -m installer numpy.whl but maybe I should user another command... ?

[pmp-p]

from a very quick look of it, it seems the python build rely upon a needed distribution of ta-lib that cython code will link to (probably statically so most likelty no need to bundle ta-lib with the python wheel )
So first of all you should build (and install to $PREFIX) ta-lib from the wasm shell ( use standard emscripten tools for that emconfigure/emmake or emcmake ).

[svax974]

I guessed it would be a necessary step so I did that, only static libs though, the wasm compiler doesn't support -soname option. Could shared libs be needed ?

But anyway, this a no relation with the import numpy problem that I'm having even with a simple test file import numpy.
I can't find a way to solve it and I can't get further for now.

[svax974]

Could the numpy import error be caused by the fact that numpy is compiled in wasm with emcc and the python3-wasm interpreter is x86_64 ?
If this is the case, numpy might work when imported from a script running with a wasm compiled version of python but how would it then be possible to compile a wasm module depending on numpy ?

I'm sorry, I'm still trying to understand the logic behind all this. Any light on this is welcome :)

[svax974]

Sorry to get back with this but has anyone successfully used the numpy wasm package with python3-wasm ?