REGR: cross-compilation fails for osx-arm64 with meson 1.5.0

[h-vetinari]

Describe the bug
A previously (with meson 1.4.2) working build for osx-arm64 broke with the release of meson 1.5.0, apparently trying to run sanitycheckc_cross.exe with a wrong architecture

The Meson build system
Version: 1.5.0
Source dir: /Users/runner/miniforge3/conda-bld/matplotlib-suite_1720666814590/work
Build dir: /Users/runner/miniforge3/conda-bld/matplotlib-suite_1720666814590/work/builddir
Build type: cross build
Program python3 found: YES (/Users/runner/miniforge3/conda-bld/matplotlib-suite_1720666814590/_build_env/bin/python3)
Project name: matplotlib
Project version: 3.9.1

meson.build:1:0: ERROR: Could not invoke sanity test executable: [Errno 86] Bad CPU type in executable: '/Users/runner/miniforge3/conda-bld/matplotlib-suite_1720666814590/work/builddir/meson-private/sanitycheckc_cross.exe'.

Note that we're cross-compiling three different architectures (the other two being linux-aarch64 and linux-ppc64le, both compiled from linux-64), and osx-arm64 is the only one that broke.

More specifically, the cleaned-up build invocation looks as follows

mkdir builddir
$PREFIX/bin/python -m mesonbuild.mesonmain setup builddir \
    --buildtype release --prefix=$PREFIX -Dlibdir=lib \
    --cross-file $BUILD_PREFIX/meson_cross_file.txt \
    --pkg-config-path=$PREFIX/lib/pkgconfig \
    -Dsystem-freetype=true -Dsystem-qhull=true

and the contents of meson_cross_file.txt are:

[host_machine]
system = 'darwin'
cpu = 'arm64'
cpu_family = 'aarch64'
endian = 'little'
[binaries]
cmake = '$BUILD_PREFIX/bin/cmake'
pkg-config = '$BUILD_PREFIX/bin/pkg-config'

To Reproduce

Build the recipe in https://github.com/conda-forge/matplotlib-feedstock

Reverting back to meson <1.5 causes the build to succeed.

Expected behavior
Build succeeds

system parameters

• Is this a cross build or just a plain native build (for the same computer)?
  • cross build
• what operating system (e.g. MacOS Catalina, Windows 10, CentOS 8.0, Ubuntu 18.04, etc.)
  • build environment is macOS-12 image (x64) from azure, target is macOS>=11.0 (arm64)
• what Python version are you using e.g. 3.8.0
  • 3.9-3.12
• what meson --version
  • 1.5.0
• what ninja --version if it's a Ninja build
  • 1.12.1

[eli-schwartz]

Would you be able to bisect this by installing different git commits of meson into that feedstock? Nothing obviously jumps out at me as related to this, except maybe c107624 (although that commit does look suspiciously related ;)).

[h-vetinari]

although that commit does look suspiciously related ;)

Your suspicion was spot on. I tried this in conda-forge/matplotlib-feedstock#398:

• 6b56952 (parent of c107624) ✅ (CI)
• c107624 ❌ (CI)

[eli-schwartz]

Thanks for verifying that. @xclaesse as the author of that commit, can you take a look at this?

(It is bedtime now so that's all the debugging I will be doing for now.)

[h-vetinari]

Originally I worried that this would have a huge blast radius for conda-forge (well, for those packages being built by meson at least), as that's one of our main architectures, and we only cross-compile it (due to lack of native agents in azure).

I restarted conda-forge/scipy-feedstock#269 ¹, and to my surprise, this doesn't run into the same issue:

+ meson setup $SRC_DIR $SRC_DIR/builddir -Dbuildtype=release \
    -Db_ndebug=if-release -Db_vscrt=md -Dblas=blas -Dlapack=lapack -Duse-g77-abi=true \
    --prefix=$PREFIX -Dlibdir=lib --cross-file $BUILD_PREFIX/meson_cross_file.txt \
    --native-file=$SRC_DIR/builddir/meson-python-native-file.ini
The Meson build system
Version: 1.5.0
Source dir: $SRC_DIR
Build dir: $SRC_DIR/builddir
Build type: cross build
Project name: scipy
Project version: 1.14.0
C compiler for the host machine: arm64-apple-darwin20.0.0-clang (clang 16.0.6 "clang version 16.0.6")
C linker for the host machine: arm64-apple-darwin20.0.0-clang ld64 711

One difference that stands out is --native-file, though I don't understand under what conditions this is created; it's not part of the build script, and it doesn't seem to be created in scipy itself either (though there's a cross-compiled setting in __config__.py.in). CC @rgommers

conda-forge/scikit-learn-feedstock#270 similarly passes (and sets --native-file). So it doesn't seem to affect everything, but I don't understand the conditions under which failure happens or not.

Footnotes

1. originally restricted to windows as it was meant only for debugging add initial support for llvm-flang #13323 ↩

[rgommers]

One difference that stands out is --native-file, though I don't understand under what conditions this is created;

--native-file is the setup argument coming from meson-python, to ensure that the native Python interpreter Meson picks is actually the right one (i.e., the one in the environment for which meson-python itself is installed). It's always passed when you build an sdist or wheel with meson-python.

[rgommers]

@h-vetinari this line is wrong: https://github.com/conda-forge/matplotlib-feedstock/blob/414fc00e6cde964337d84b5234673a647cc22c00/recipe/build_base.sh (both because meson-python may redo the configure stage, and because invoking python -m mesonbuild.mesonmain isn't supported by Meson AFAIK). If you adapt it to what's used in scipy-feedstock, that may help.

[h-vetinari]

Thanks a lot @rgommers; I fixed the double-execution in conda-forge/matplotlib-feedstock#398 (now just going through meson-python as it should), but the issue persists with 1.5.0, and in fact the same result for the specific commits as above (links to CI logs have been updated below):

• 6b56952 ✅ (CI)
• c107624 ❌ (CI)

[lazka]

Here is a potential fix: #13411

[h-vetinari]

Thanks a lot @lazka! 🙏

I've tested your PR in conda-forge/matplotlib-feedstock#398 and it works again! 🥳

[h-vetinari]

Is there any approximate timeline for a 1.5.1 release?

[eli-schwartz]

There are two more bugs open on the 1.5.1 milestone, one of them is going to be investigated and hopefully fixed tomorrow. I'd like to get either or both of them fixed in the next couple of days, then release 1.5.1.

[lazka]

@h-vetinari 1.5.1 is out

[h-vetinari]

I saw, thanks a lot for everyone's effort!