Update scipy package version #648

pcanas · 2022-07-13T15:33:53Z

The current supported scipy version is 1.4.1 which was released +2.5 years ago.
As many changes have been developed since, and the library is one of the most popular ones in the Python ecosystem, I'd like to request an update of this library to a newer version.

Thank you!

The text was updated successfully, but these errors were encountered:

mhsmith · 2022-07-13T23:12:10Z

Why specifically do you need the newer version? Please provide some example code if possible.

pcanas · 2022-07-14T07:16:02Z

We rely on the function scipy.signal.find_peaks (ref) which I believe has changed implementation and parameters since this version.

mhsmith · 2022-07-14T13:55:24Z

I can't find any mention of find_peaks in the release notes since version 1.4.1 (see this search). If you've seen a change in the behavior of this function, please post an example.

pcanas · 2022-07-14T15:55:33Z

I can't find any mention to it either; we found an inconsistency between version back in November but I was not able to retrieve back the exact issue. If I find it I will report back here.
In any case, I was wondering if it makes sense to have more frequent updates of these libraries that are the core of Python's language.

mhsmith · 2022-07-15T21:27:32Z

We generally update all packages when we move to a new version of Python, with additional updates if users have a specific need for a newer version and the build is not too difficult.

We're currently planning to move to Python 3.10 later this year, so we'll update SciPy then. But if you can remember the details of your issue, please post them here and we'll consider updating it earlier.

If anyone else needs a new version of this package, please click the thumbs up button above, and post a comment explaining why you need it.

pcanas · 2022-07-21T13:19:52Z

Hi Malcolm.

We have just found the problematic piece of code: https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.median_abs_deviation.html

This function was implemented in version 1.5.0, and we use it in several parts of our codebase.

Should I expect this to be updated later this year, as you suggested in your previous message? To be fair, this one is pretty easy to implement without scipy..

mhsmith · 2022-07-22T08:29:16Z

Should I expect this to be updated later this year, as you suggested in your previous message?

Yes, that's correct.

mhsmith · 2023-04-14T20:37:40Z

Chaquopy now supports Python 3.10, but all SciPy builds are currently blocked because we don't have a working process for setting up an Android Fortran compiler. We did have such a process in 2020 when we moved to Python 3.8, but it hasn't been updated for the fact that build-wheel no longer uses a standalone NDK toolchain.

See target/build-fortran.sh for more details. It may also be worth looking at what Kivy's python-for-android does, since they have Fortran support now.

pasdeloup · 2023-08-14T11:42:26Z

Hi, following Kivy's method I could build a Python 3.9 version of Scipy.

Basically I downloaded https://github.com/mzakharo/android-gfortran/releases/download/r21e/gcc-arm64-linux-x86_64.tar.bz2 and used the provided Fortran compiler.

However I had issues with pybind11 missing header and fixed it by patching server/pypi/env/lib/python/sitecustomize.py
line 36

-valid_dirs = [abspath(path) for path in [src_dir, f"{src_dir}/../requirements"]]
+valid_dirs = [abspath(path) for path in [src_dir, f"{src_dir}/../requirements", f"{src_dir}/../env"]]

did I miss something in the build process?
Currently it seems to be working but better use the proper way to do it...

mhsmith · 2023-08-14T12:46:49Z

Thanks, this could be useful. Which ABIs have you built? And what testing have you done? There are some simple tests in the repository which you can run as described in "Testing a package" in the README.

Please create a PR with your changes, and also include:

Instructions for how to set up the Fortran compiler
Details of the "issues with pybind11 missing header"

pasdeloup · 2023-08-16T07:31:21Z

That was only preliminary work to assess feasibility, I only built for arm64-v8a and succeed to load it and read a simple Python constant from the package on Android, I still have to run more tests.

Fortran compiler setup was also manual: I downloaded the above package and override my own gfortran using update-alternatives

Still have additional tests to run and additional packages to check and then I'll try to create a PR.

pasdeloup · 2023-08-18T10:18:23Z

Additional info: I'm still stuck with 1.4.1 version but I could build it for Python 3.9 and it passes tests (except some deprecation warning due to numpy 1.20)
I will create a PR, but will need some help to build more recent versions as I don't master the art of patching scipy source code: I don't know why we have to get rid of all numpy references and how to deal with additional ones.

pasdeloup · 2023-08-28T10:47:57Z

Since 1.9 SciPy build process has been rewrote to use Meson, and that's supposed to help cross-compilation:
https://docs.scipy.org/doc/scipy-1.11.2/building/cross_compilation.html

mhsmith · 2023-08-29T19:27:08Z

Thanks to the Fortran instructions from @pasdeloup, I've now built and released SciPy 1.4.1 for Python 3.9. Unfortunately this version is too old to build for Python 3.10.

The patches to remove the NumPy references are because the Android native libraries in NumPy can't be imported on Linux. So we need to either reduce it to just importing the pure-Python modules, or remove the imports entirely and replace them with something else.

Based on your link, it looks like newer versions of SciPy have better support for cross-compilation, which may allow many of our patches to be removed. However, to support Meson or any other build system other than Setuptools, we'll have to remove the hack in build-wheel which removes the pyproject.toml file. Since this file is increasingly being used for other critical things, we need to fix this anyway even for some Setuptools-based packages (e.g. #866 and #812), so I'll look at that very soon.

mhsmith · 2023-08-31T13:44:57Z

However, to support Meson or any other build system other than Setuptools, we'll have to remove the hack in build-wheel which removes the pyproject.toml file. Since this file is increasingly being used for other critical things, we need to fix this anyway even for some Setuptools-based packages (e.g. #866 and #812), so I'll look at that very soon.

This has now been done. As a result, any requirements listed in a pyproject.toml's build-system section will now be installed into the build environment. This may cause some of the existing package builds to change or break.

For example, the version of setuptools in NumPy's pyproject.toml file failed to detect some math functions, so I patched it to use the same version as in #935. I then verified that NumPy 1.23.3 and SciPy 1.4.1 can still build and pass the tests, with Python 3.9 and arm64-v8a.

mhsmith · 2023-09-10T17:27:38Z

Thanks to @pasdeloup in #949, SciPy 1.6.3 is now in the public repository for Python 3.9. Unfortunately this version isn't compatbile with Python 3.10 or later.

As for Python 3.8, SciPy 1.6.3 should also be compatible with NumPy 1.19.5, which is probably the last version we'll release in the main repository for Python 3.8 because of #570. But that combination fails to build, and it's not immediately obvious how to fix it:

Error message

building extension "scipy.integrate.vode" sources
creating build/src.linux_armv7l-3.8/scipy/integrate
f2py options: []
f2py: scipy/integrate/vode.pyf
Traceback (most recent call last):
  File "/home/smith/git/chaquo/chaquopy/server/pypi/packages/scipy/build/1.6.3/cp38-cp38-android_21_armeabi_v7a/requirements/numpy/core/__init__.py", line 22, in <module>
    from . import multiarray
  File "/home/smith/git/chaquo/chaquopy/server/pypi/packages/scipy/build/1.6.3/cp38-cp38-android_21_armeabi_v7a/requirements/numpy/core/multiarray.py", line 12, in <module>
    from . import overrides
  File "/home/smith/git/chaquo/chaquopy/server/pypi/packages/scipy/build/1.6.3/cp38-cp38-android_21_armeabi_v7a/requirements/numpy/core/overrides.py", line 7, in <module>
    from numpy.core._multiarray_umath import (
ImportError: /home/smith/git/chaquo/chaquopy/server/pypi/packages/scipy/build/1.6.3/cp38-cp38-android_21_armeabi_v7a/requirements/numpy/core/_multiarray_umath.so: wrong ELF class: ELFCLASS32

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/smith/.venv/pypi/lib/python3.8/site-packages/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
    main()
  File "/home/smith/.venv/pypi/lib/python3.8/site-packages/pyproject_hooks/_in_process/_in_process.py", line 335, in main
    json_out['return_val'] = hook(**hook_input['kwargs'])
  File "/home/smith/.venv/pypi/lib/python3.8/site-packages/pyproject_hooks/_in_process/_in_process.py", line 251, in build_wheel
    return _build_backend().build_wheel(wheel_directory, config_settings,
  File "/home/smith/git/chaquo/chaquopy/server/pypi/packages/scipy/build/1.6.3/cp38-cp38-android_21_armeabi_v7a/env/lib/python3.8/site-packages/setuptools/build_meta.py", line 216, in build_wheel
    return self._build_with_temp_dir(['bdist_wheel'], '.whl',
  File "/home/smith/git/chaquo/chaquopy/server/pypi/packages/scipy/build/1.6.3/cp38-cp38-android_21_armeabi_v7a/env/lib/python3.8/site-packages/setuptools/build_meta.py", line 202, in _build_with_temp_dir
    self.run_setup()
  File "/home/smith/git/chaquo/chaquopy/server/pypi/packages/scipy/build/1.6.3/cp38-cp38-android_21_armeabi_v7a/env/lib/python3.8/site-packages/setuptools/build_meta.py", line 253, in run_setup
    super(_BuildMetaLegacyBackend,
  File "/home/smith/git/chaquo/chaquopy/server/pypi/packages/scipy/build/1.6.3/cp38-cp38-android_21_armeabi_v7a/env/lib/python3.8/site-packages/setuptools/build_meta.py", line 145, in run_setup
    exec(compile(code, __file__, 'exec'), locals())
  File "setup.py", line 612, in <module>
    setup_package()
  File "setup.py", line 608, in setup_package
    setup(**metadata)
  File "/home/smith/git/chaquo/chaquopy/server/pypi/packages/scipy/build/1.6.3/cp38-cp38-android_21_armeabi_v7a/requirements/numpy/distutils/core.py", line 169, in setup
    return old_setup(**new_attr)
  File "/home/smith/git/chaquo/chaquopy/server/pypi/packages/scipy/build/1.6.3/cp38-cp38-android_21_armeabi_v7a/env/lib/python3.8/site-packages/setuptools/__init__.py", line 153, in setup
    return distutils.core.setup(**attrs)
  File "/home/smith/.pyenv/versions/3.8.12/lib/python3.8/distutils/core.py", line 148, in setup
    dist.run_commands()
  File "/home/smith/.pyenv/versions/3.8.12/lib/python3.8/distutils/dist.py", line 966, in run_commands
    self.run_command(cmd)
  File "/home/smith/.pyenv/versions/3.8.12/lib/python3.8/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/home/smith/git/chaquo/chaquopy/server/pypi/packages/scipy/build/1.6.3/cp38-cp38-android_21_armeabi_v7a/env/lib/python3.8/site-packages/wheel/bdist_wheel.py", line 299, in run
    self.run_command('build')
  File "/home/smith/.pyenv/versions/3.8.12/lib/python3.8/distutils/cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "/home/smith/.pyenv/versions/3.8.12/lib/python3.8/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/home/smith/git/chaquo/chaquopy/server/pypi/packages/scipy/build/1.6.3/cp38-cp38-android_21_armeabi_v7a/requirements/numpy/distutils/command/build.py", line 40, in run
    old_build.run(self)
  File "/home/smith/.pyenv/versions/3.8.12/lib/python3.8/distutils/command/build.py", line 135, in run
    self.run_command(cmd_name)
  File "/home/smith/.pyenv/versions/3.8.12/lib/python3.8/distutils/cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "/home/smith/.pyenv/versions/3.8.12/lib/python3.8/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
File "/home/smith/git/chaquo/chaquopy/server/pypi/packages/scipy/build/1.6.3/cp38-cp38-android_21_armeabi_v7a/requirements/numpy/distutils/command/build_src.py", line 144, in run
    self.build_sources()
  File "/home/smith/git/chaquo/chaquopy/server/pypi/packages/scipy/build/1.6.3/cp38-cp38-android_21_armeabi_v7a/requirements/numpy/distutils/command/build_src.py", line 161, in build_sources
    self.build_extension_sources(ext)
  File "/home/smith/git/chaquo/chaquopy/server/pypi/packages/scipy/build/1.6.3/cp38-cp38-android_21_armeabi_v7a/requirements/numpy/distutils/command/build_src.py", line 321, in build_extension_sources
    sources = self.f2py_sources(sources, ext)
  File "/home/smith/git/chaquo/chaquopy/server/pypi/packages/scipy/build/1.6.3/cp38-cp38-android_21_armeabi_v7a/requirements/numpy/distutils/command/build_src.py", line 542, in f2py_sources
    import numpy.f2py
  File "/home/smith/git/chaquo/chaquopy/server/pypi/packages/scipy/build/1.6.3/cp38-cp38-android_21_armeabi_v7a/requirements/numpy/f2py/__init__.py", line 12, in <module>
    from . import f2py_testing
  File "/home/smith/git/chaquo/chaquopy/server/pypi/packages/scipy/build/1.6.3/cp38-cp38-android_21_armeabi_v7a/requirements/numpy/f2py/f2py_testing.py", line 4, in <module>
    from numpy.testing import jiffies, memusage
  File "/home/smith/git/chaquo/chaquopy/server/pypi/packages/scipy/build/1.6.3/cp38-cp38-android_21_armeabi_v7a/requirements/numpy/testing/__init__.py", line 10, in <module>
    from ._private.utils import *
  File "/home/smith/git/chaquo/chaquopy/server/pypi/packages/scipy/build/1.6.3/cp38-cp38-android_21_armeabi_v7a/requirements/numpy/testing/_private/utils.py", line 20, in <module>
    from numpy.core import(
  File "/home/smith/git/chaquo/chaquopy/server/pypi/packages/scipy/build/1.6.3/cp38-cp38-android_21_armeabi_v7a/requirements/numpy/core/__init__.py", line 48, in <module>
    raise ImportError(msg)
ImportError:

IMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE!

Importing the numpy C-extensions failed. This error can happen for
...

mhsmith · 2023-09-10T19:10:38Z

I was able to build SciPy 1.6.3 against Numpy 1.20.3 for Python 3.8, and when used together, they passed all the tests on both ABIs. But if you try to use the SciPy build with NumPy 1.19.5, you get this error:

RuntimeError: module compiled against API version 0xe but this version of numpy is 0xd

So if you want to use SciPy 1.6.3 on Python 3.8, you'll have to add this line to the pip block in your build.gradle file:

options "--extra-index-url", "https://chaquo.com/pypi-extra/"

mhsmith · 2023-09-10T19:23:05Z

The newest SciPy version that was requested above was 1.5.0, so I'm going to close this issue now. If you need a newer SciPy version than 1.6.3, please create a separate issue and explain exactly why you need it.

mhsmith added packages update labels Jul 13, 2022

mhsmith mentioned this issue Apr 14, 2023

Update Packages #853

Closed

mhsmith mentioned this issue May 31, 2023

Package Request : River #875

Open

mhsmith mentioned this issue Jun 24, 2023

Is there any plan to implement pyts #891

Closed

mhsmith mentioned this issue Jul 5, 2023

sisl in chaquopy #901

Open

mhsmith mentioned this issue Aug 8, 2023

Mediapipe package #479

Open

pasdeloup mentioned this issue Aug 18, 2023

Fixes to build Scipy #937

Merged

pasdeloup mentioned this issue Aug 31, 2023

Update SciPy version #949

Merged

mhsmith closed this as completed Sep 10, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update scipy package version #648

Update scipy package version #648

pcanas commented Jul 13, 2022 •

edited by mhsmith

Loading

mhsmith commented Jul 13, 2022

pcanas commented Jul 14, 2022

mhsmith commented Jul 14, 2022 •

edited

Loading

pcanas commented Jul 14, 2022

mhsmith commented Jul 15, 2022 •

edited

Loading

pcanas commented Jul 21, 2022 •

edited

Loading

mhsmith commented Jul 22, 2022

mhsmith commented Apr 14, 2023 •

edited

Loading

pasdeloup commented Aug 14, 2023

mhsmith commented Aug 14, 2023

pasdeloup commented Aug 16, 2023

pasdeloup commented Aug 18, 2023

pasdeloup commented Aug 28, 2023

mhsmith commented Aug 29, 2023 •

edited

Loading

mhsmith commented Aug 31, 2023 •

edited

Loading

mhsmith commented Sep 10, 2023 •

edited

Loading

mhsmith commented Sep 10, 2023

mhsmith commented Sep 10, 2023

Update scipy package version #648

Update scipy package version #648

Comments

pcanas commented Jul 13, 2022 • edited by mhsmith Loading

mhsmith commented Jul 13, 2022

pcanas commented Jul 14, 2022

mhsmith commented Jul 14, 2022 • edited Loading

pcanas commented Jul 14, 2022

mhsmith commented Jul 15, 2022 • edited Loading

pcanas commented Jul 21, 2022 • edited Loading

mhsmith commented Jul 22, 2022

mhsmith commented Apr 14, 2023 • edited Loading

pasdeloup commented Aug 14, 2023

mhsmith commented Aug 14, 2023

pasdeloup commented Aug 16, 2023

pasdeloup commented Aug 18, 2023

pasdeloup commented Aug 28, 2023

mhsmith commented Aug 29, 2023 • edited Loading

mhsmith commented Aug 31, 2023 • edited Loading

mhsmith commented Sep 10, 2023 • edited Loading

mhsmith commented Sep 10, 2023

mhsmith commented Sep 10, 2023

pcanas commented Jul 13, 2022 •

edited by mhsmith

Loading

mhsmith commented Jul 14, 2022 •

edited

Loading

mhsmith commented Jul 15, 2022 •

edited

Loading

pcanas commented Jul 21, 2022 •

edited

Loading

mhsmith commented Apr 14, 2023 •

edited

Loading

mhsmith commented Aug 29, 2023 •

edited

Loading

mhsmith commented Aug 31, 2023 •

edited

Loading

mhsmith commented Sep 10, 2023 •

edited

Loading