python3.9 rejects the macosx_10_9_x86_64 platform tag

[mattip]

I am a NumPy developer and do not use brew. We have been getting reports across the scientific python ecosystem (matplotlib, numpy, pandas, scipy, and more) that python3.9 -m pip install numpy is failing to properly install NumPy. It is building from source and linking to the buggy Apple Accelerate library (instead of the OpenBLAS library we ship with our binary wheels. NumPy has a smoke test at import that Accelerate fails, so end-users are trying to install numpy, which is building from source, which does not work.

I think we tracked it down to a build problem where python3.9 from HomeBrew is not accepting the macosx_10_9_x86_64 platform tag. Since the NumPy wheels use that tag, python3.9 -m pip install numpy will scan available binary packages, not find a compatible one, and trigger a build from source, leading to the situation described above.

We tried hard to detect Accelerate in our build system but failed. Maybe you all have some good idea how to avoid using it?

xref numpy/numpy#17784.

[fxcoudert]

Hi @mattip,

• Regarding “python3.9 from HomeBrew is not accepting the macosx_10_9_x86_64 platform tag”, it is not something we are doing on purpose. We install the latest versions of setuptools, pip, and wheel. Could it be because Big Sur is 11.0 (and not 10.something)?
• Regarding “We tried hard to detect Accelerate in our build system but failed”: I wondered indeed why you could not fail earlier in the build process?

[fxcoudert]

@mattip I think your issue is this one pypa/pip#9138

[mattip]

Could you build the python3.9 version on a 10.X system (like python3.8, python3.7) to unblock users?

[fxcoudert]

All Homebrew python versions are now built on 11.0
For users the simplest solution is probably to install our numpy with brew install numpy

[mattip]

OK, thanks for the clarifications.

[fxcoudert]

Regarding:

We tried hard to detect Accelerate in our build system but failed

It seems the support for Accelerate was removed in this commit: numpy/numpy#15759
So I'm not sure why it's still linking against it. Do you have an analysis somewhere?

---

Building 1.19.4 from source right now, on my mac, I get:

    Cythonizing sources
    blas_opt_info:
    blas_mkl_info:

[…]

    accelerate_info:
      libraries accelerate not found in ['/usr/local/Cellar/python@3.9/3.9.0_1/Frameworks/Python.framework/Versions/3.9/lib', '/usr/local/lib', '/usr/lib']
    Library accelerate was not found. Ignoring
      libraries veclib not found in ['/usr/local/Cellar/python@3.9/3.9.0_1/Frameworks/Python.framework/Versions/3.9/lib', '/usr/local/lib', '/usr/lib']
    Library veclib was not found. Ignoring
      FOUND:
        extra_compile_args = ['-msse3', '-I/System/Library/Frameworks/vecLib.framework/Headers']
        extra_link_args = ['-Wl,-framework', '-Wl,Accelerate']
        define_macros = [('NO_ATLAS_INFO', 3), ('HAVE_CBLAS', None)]

      FOUND:
        extra_compile_args = ['-msse3', '-I/System/Library/Frameworks/vecLib.framework/Headers']
        extra_link_args = ['-Wl,-framework', '-Wl,Accelerate']
        define_macros = [('NO_ATLAS_INFO', 3), ('HAVE_CBLAS', None)]

and

    atlas_info:
      libraries lapack_atlas not found in /usr/local/Cellar/python@3.9/3.9.0_1/Frameworks/Python.framework/Versions/3.9/lib
      libraries f77blas,cblas,atlas not found in /usr/local/Cellar/python@3.9/3.9.0_1/Frameworks/Python.framework/Versions/3.9/lib
      libraries lapack_atlas not found in /usr/local/lib
      libraries f77blas,cblas,atlas not found in /usr/local/lib
      libraries lapack_atlas not found in /usr/lib
      libraries f77blas,cblas,atlas not found in /usr/lib
    <class 'numpy.distutils.system_info.atlas_info'>
      NOT AVAILABLE

      FOUND:
        extra_compile_args = ['-msse3', '-I/System/Library/Frameworks/vecLib.framework/Headers']
        extra_link_args = ['-Wl,-framework', '-Wl,Accelerate']
        define_macros = [('NO_ATLAS_INFO', 3), ('HAVE_CBLAS', None)]

So something is still clearly searching for Accelerate. I don't know your codebase so I don't know where that's coming from, but I suspect the behaviour you want is a hard failure at this point: that will let users know that they should install a working BLAS implementation.

[fxcoudert]

The code that is finding and linking Accelerate is here: https://github.com/numpy/numpy/blob/master/numpy/distutils/system_info.py#L2450
It's present on both the 1.19 branch and master.

@mattip I'd be happy to help fix this, because it's a bug report we also get often. But I need a bit more context, for example as to why that code wasn't removed.

[mattip]

After numpy/numpy#15759 we no longer search for Accelerate . Whille we could not totally remove the code you point to since numpy/distutils is used in downstream projects, in that PR we tried hard to disallow specifying Accelerate as a valid linalg backend. I think the issue is that we find /usr/lib/libblas.so which is an alias to Accelerate, and cannot differentiate it from Accelerate in any clean way before compilation. Note that even if we could, that would only kick the compile-in-place can down the road to performance issues: NumPy would work with an internal lapack-lite implementation with horrible performance.

[fxcoudert]

@mattip I think the good news is: in more recent macOS versions there is no /usr/lib/libblas.so anymore, so that might help.

Closing this bug, as the issue probably with pypa/pip#9138