numpy v1.26.0 with meson

[h-vetinari]

Let's see if we can build with meson (esp. on windows...)

[conda-forge-webservices]

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.

[h-vetinari]

I based this off of the scipy recipe, and this went pretty far on unix:

Successfully built numpy-1.26.0b1-cp310-cp310-linux_x86_64.whl
+ pip install dist/numpy-1.26.0b1-cp310-cp310-linux_x86_64.whl
Processing ./dist/numpy-1.26.0b1-cp310-cp310-linux_x86_64.whl
Installing collected packages: numpy
Successfully installed numpy-1.26.0b1

However we then have:

INFO:conda_build.build:Packaging numpy
Packaging numpy
Packaging numpy-1.26.0b1-py310ha4c1d20_0
INFO:conda_build.build:Packaging numpy-1.26.0b1-py310ha4c1d20_0
No files or script found for output numpy
WARNING:conda_build.build:No files or script found for output numpy
number of files: 0

I think this might be because the default meson install sets a local --destdir:

[...]
[505/505] Linking target numpy/random/_generator.cpython-310-x86_64-linux-gnu.so
+ $PREFIX/bin/python $SRC_DIR/vendored-meson/meson/meson.py install --quiet --no-rebuild --destdir $SRC_DIR/.mesonpy-8wjmup75/install
[0/196] numpy/core/lib/libnpymath.a
[1/196] numpy/core/include/numpy/__multiarray_api.c
[...]

On windows, the same invocation as on unix fails:

(%BUILD_PREFIX%) %SRC_DIR%>%PREFIX%\python.exe -m build -w -n -x     -Cbuilddir=builddir     -Csetup-args=-Dblas=blas     -Csetup-args=-Dlapack=lapack     -Csetup-args=${MESON_ARGS// / -Csetup-args=} 
ERROR Source / does not appear to be a Python project: no pyproject.toml or setup.py

This is not a very comprehensive take yet, just a first shot. But it's getting late here, so this'll have to suffice for tonight.

[rgommers]

I think this might be because the default meson install sets a local --destdir:

I would not expect that to be relevant, it's an implementation detail that's contained inside what meson-python is doing.

If the produced intermediate wheel is empty, pip would not report successfully installing it. If it's not empty, then I'm unclear why conda-build doesn't find any files. Perhaps easier to figure out from a local build than a CI log?

[h-vetinari]

If it's not empty, then I'm unclear why conda-build doesn't find any files.

My suspicion is that it gets installed somewhere other than $PREFIX resp. SP_DIR.

Perhaps easier to figure out from a local build than a CI log?

Sure. As I said I haven't had much time for this yesterday, just started with opening the PR.

[h-vetinari]

So the devdocs about building are not up-to-date, but that's being tracked, and at I also found numpy/numpy#24279. Not sure what's up with pkgconfig, our netlib blas/lapack builds do have the respective .pc files.

[h-vetinari]

So for cross-compiling osx-arm64, we get

Checking for size of "long double" : 8 

../numpy/core/meson.build:461:2: ERROR: Problem encountered: Unknown long double format of size: 8

which seems to imply that the patterns used for detecting the long double flavour are perhaps not exhaustive.

@matoro, I see you authored numpy/numpy@f1cddd3, could it be that osx-arm64 has a different in-memory representation than what you've coded up for size 8?

[matoro]

So for cross-compiling osx-arm64, we get

Checking for size of "long double" : 8 

../numpy/core/meson.build:461:2: ERROR: Problem encountered: Unknown long double format of size: 8

which seems to imply that the patterns used for detecting the long double flavour are perhaps not exhaustive.

@matoro, I see you authored numpy/numpy@f1cddd3, could it be that osx-arm64 has a different in-memory representation than what you've coded up for size 8?

Hey, I just tried to copy the python code 1-for-1. Does it somehow work on the setuptools implementation? Could you run this snippet and share the output?

#include <stdio.h>

const struct {
    const char before[16];
    const long double x;
    const char after[8];
} foo = {{'\0'}, -123456789.0, {'\0'}};

int main(void)
{
    for (size_t i = 0; i < sizeof(long double); i++) {
        fprintf(stdout, "%04o ", ((const unsigned char*)&foo.x)[i]);
    }
    fprintf(stdout, "\n");
    return 0;
}

[rgommers]

The size of long double is 8 bytes on macOS arm64, and the format is IEEE_DOUBLE_LE in a native build. Since you're trying to cross-compile here, probably something is going wrong. The check does:

meson.get_compiler('c').run(...)

It should run under QEMU I think, and it likely doesn't here, or that is not working (can't tell from the CI log). You can backport numpy/numpy#24414 to bypass this check for now.

[h-vetinari]

Thanks for the response @matoro and the reference @rgommers. This is looking good now!

Is the successful build with the beta enough from your POV (I'm guessing with a passing test suite we should be good normally?), or do you want to publish it to an rc-label to be able to test it against other packages?

[rgommers]

Is the successful build with the beta enough from your POV (I'm guessing with a passing test suite we should be good normally?), or do you want to publish it to an rc-label to be able to test it against other packages?

This is fine I think for now; quite a few downstream packages are testing in CI themselves, because they already need it for Python 3.12 support. So I don't think we'll learn much more from a testing exercise in conda-forge now.

Thanks for verifying this beta though, great to see it working.

[matoro]

This is a legit complaint though and it's my fault for breaking direct compatibility with cross-compiling. I don't think there is any way to make meson .run() work out of the box with cross-compiling. The original python code simply inspected the compiled object file rather than running it, so it worked. This needs to work natively without binfmt_misc set up, doing otherwise is a regression.

[mattip]

I see the remaining CI failure is PyPy + windows, where meson --version is failing with an AttributeError: 'NoneType' object has no attribute 'encoding'.

[mattip]

This seems to be some kind of pytest ineteraction? Using this works with no error:

conda create -n pypy3.9 pypy python=3.9
conda activate pypy3.9
mamba install meson
pypy -c "import subprocess; subprocess.check_call(["meson", "--version"])

[h-vetinari]

This seems to be some kind of pytest interaction?

And/or meson interaction

  File "%PREFIX%\lib\site-packages\mesonbuild\mesonmain.py", line 230, in ensure_stdout_accepts_unicode
    if sys.stdout.encoding and not sys.stdout.encoding.upper().startswith('UTF-'):
AttributeError: 'NoneType' object has no attribute 'encoding'

[h-vetinari]

Anyway, it looks like this'll have to get fixed on the meson-side. I don't think we should hold up 1.26 of win+pypy for now, it's already in high demand due to the impending 3.12.