Add numpy2 support #2449
Add numpy2 support #2449
Conversation
- added test_weights_2d - interpolate marked as failing
Removing vaex.vaexfast.Error in favor of using std errors
Adressing:
/opt/hostedtoolcache/Python/3.9.21/x64/lib/python3.9/site-packages/vaex/strings.py:10: in <module>
from .superstrings import *
E ImportError: libpcrecpp.so.0: cannot open shared object file: No such file or directory
…merge supposed to do here
|
expression: '((255 - (g - -128)) + -128)' result = sub(255, array([127],int8))
True -> np.True_ and False -> np.False_
For now, yes, should still run with v1, and so far it does considering ci and 3.9 (uses numpy==1.26.4) |
…find_edges percentile_approx_test fails with nan in the result array, caused by vaex.vaexfast.grid_find_edges returning edges_floor / edges_ceil with [from, to] where from == to
|
All outstanding issues related to numpy v2 (and some not) are resolved - only catboost dependent functionality is conditionally disabled. All pipelines run fine - even some previously skipped tests. |
|
awesome work! ❤️ @maartenbreddels if we get lucky, the pybind submodule update in this PR will also solve #2442 |
| except ValueError as e: | ||
| import re | ||
| if re.match(r'numpy.dtype size changed', str(e)): | ||
| pytestmark = pytest.mark.skip(f"all tests skipped due to: numpy2 incompatibility -- {e}") |
There was a problem hiding this comment.
if catboost currently pins numpy v1, and our CI installs vaex-ml[all], then CI should be running with numpy v1. then why is this change necessary?
There was a problem hiding this comment.
catboost does not support v2, yet they do not pin to v1 - it can be installed along with numpy v2
There was a problem hiding this comment.
ah right. they do have an upper cap as of 1.2.6 (catboost/catboost@4b1b8ae), but we don't have a lower pin on catboost in vaex-ml, so CI is now happily installing catboost 1.2.5 alongside numpy v2.
this should resolve itself once catboost releases numpy v2 support. can you add the following inline comment?
# this try-except can be removed once catboost releases numpy v2 support
# ref https://github.com/catboost/catboost/issues/2671
# ref https://github.com/vaexio/vaex/pull/2449
There was a problem hiding this comment.
to be precise the latest catboost version 1.2.7 comes with install_requires: 'numpy (>=1.16.0, <2.0)', while catboost 1.2.5 (installed in ci) comes with install_requires: 'numpy (>=1.16.0)'
.github/workflows/pythonpackage.yml
Outdated
| - name: Install libpcre3-dev | ||
| if: matrix.os == 'ubuntu-latest' | ||
| run: | | ||
| sudo apt update & sudo apt install -y libpcre3-dev |
There was a problem hiding this comment.
we have bin/install_pcre.sh already just above:
vaex/.github/workflows/pythonpackage.yml
Lines 52 to 54 in 4c710c1
is it broken?
There was a problem hiding this comment.
see https://github.com/vaexio/vaex/actions/runs/12819691425/job/35747856845
E ImportError: libpcrecpp.so.0: cannot open shared object file: No such file or directory
There was a problem hiding this comment.
I see. The wheels depend on that script:
vaex/.github/workflows/wheel.yml
Line 101 in 4c710c1
So maybe the wheels are broken and this smoke test doesn't catch it?
vaex/.github/workflows/wheel.yml
Line 107 in 4c710c1
What do you think about adding the apt-get to the install script? Maybe similar to this one, preferring the precompiled one over the source install? https://github.com/ahupp/python-magic/blob/65fb61c9c9aa6348bb95d1dd71b685720c2a8a23/add_libmagic.sh#L69
There was a problem hiding this comment.
I added it to the script.
The source-built libraries are in /usr/local/lib. So LD_LIBRARY_PATH or runtime search path for the extension could be set otherwise.
This pull request intends to enable the use of numpy2 (#2446 #2425) with vaex.
In addition it applies some other fixes.
An (incomplete) summary:
Note, that I encountered some broken test(s) in cmodule.py test_interpolate and those related to Webserver:
Some format changes from tab to space indent have been applied.
Please let me know if (further) changes are required - only validated on Linux system right now.
I ran tests in vaex-core/vaex/test - any more instructions on how to run you full test-suite (apart from ci)?
Thx.
@maartenbreddels @ddelange