"symbol not found" in MacOS arm64 wheels for C-extension dependent on NumPy

[flexatone]

Description

Greetings, and thanks for such an excellent tool.

I am using cibuildwheel in a GitHub Action (https://github.com/static-frame/arraykit/blob/149/py312/.github/workflows/ci.yml) to build wheels of a relatively straightforward Python C-extension which depends on NumPy C headers (https://github.com/static-frame/arraykit/blob/149/py312/src/_arraykit.c). There are no other special dependencies. Everything generally works fine, except for on MacOS arm64. While the wheels are built without error, when the module is imported on an Apple Silicon machine I get the following error:

ImportError: dlopen(/Users/ariza/.env-test-311/lib/python3.11/site-packages/arraykit/_arraykit.cpython-311-darwin.so, 0x0002): symbol not found in flat namespace '_npy_double_to_half'

When I build the wheels on an Apple Silicon machine, the wheel functions as expected. I suspect that the problem would be solved if cibuildwheel ran on an Apple Silicon arm64 architecture, however I do not have a GitHub Team account at this time to get access to the macos-13-large runner.

Is there any other alternative? Is there a cibuildwheel configuration I am missing to properly link the NumPy dependency?

Thanks in advance.

Build log

No response

CI config

https://github.com/static-frame/arraykit/blob/149/py312/.github/workflows/ci.yml

[joerick]

Cross-compiling against other python dependencies at a C API level is particularly tricky, because when installing dependencies, pip installs the x86_64 versions, so you're building the arm64 version against the headers/libraries of the x86_64 version. I suspect that's the cause of the errors here.

There might be a way around the issue using a build process built with crossenv, however, even that's not foolproof, and we never managed to get that integrated into cibuildwheel.

I suspect an easier route for you would be to find a way to build on an Apple Silicon machine. CirrusCI offers free access to Apple Silicon runners, though it is now limited to 50 credits per month, you might get by with that.

[flexatone]

Many thanks for your response. I was afraid building on Apple Silicon was the only sure approach! I will explore the options you mentioned.