pip install fails on arm64 (M1 Mac)

[janosh]

pip install crystal-toolkit fails with:

OSError: Could not find library geos_c or load any of its variants ['/Library/Frameworks/GEOS.framework/Versions/Current/GEOS', '/opt/local/lib/libgeos_c.dylib', '/usr/local/lib/libgeos_c.dylib', '/opt/homebrew/lib/libgeos_c.dylib']

Full pip install logs

  Preparing metadata (setup.py) ... error
  error: subprocess-exited-with-error
  
  × python setup.py egg_info did not run successfully.
  │ exit code: 1
  ╰─> [14 lines of output]
      Failed `CDLL(/Library/Frameworks/GEOS.framework/Versions/Current/GEOS)`
      Failed `CDLL(/opt/local/lib/libgeos_c.dylib)`
      Failed `CDLL(/usr/local/lib/libgeos_c.dylib)`
      Failed `CDLL(/opt/homebrew/lib/libgeos_c.dylib)`
      Traceback (most recent call last):
        File "<string>", line 2, in <module>
        File "<pip-setuptools-caller>", line 34, in <module>
        File "/private/var/folders/5m/sb0wx83j0m36xk_zswkfnffh0000gn/T/pip-install-stxi_jkw/shapely_52a19b9f150646e4bcd48901f3c4c7e7/setup.py", line 74, in <module>
          from shapely._buildcfg import geos_version_string, geos_version, \
        File "/private/var/folders/5m/sb0wx83j0m36xk_zswkfnffh0000gn/T/pip-install-stxi_jkw/shapely_52a19b9f150646e4bcd48901f3c4c7e7/shapely/_buildcfg.py", line 189, in <module>
          lgeos = load_dll('geos_c', fallbacks=alt_paths)
        File "/private/var/folders/5m/sb0wx83j0m36xk_zswkfnffh0000gn/T/pip-install-stxi_jkw/shapely_52a19b9f150646e4bcd48901f3c4c7e7/shapely/_buildcfg.py", line 159, in load_dll
          raise OSError(
      OSError: Could not find library geos_c or load any of its variants ['/Library/Frameworks/GEOS.framework/Versions/Current/GEOS', '/opt/local/lib/libgeos_c.dylib', '/usr/local/lib/libgeos_c.dylib', '/opt/homebrew/lib/libgeos_c.dylib']
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.

Worked fine installing on a Linux cluster so seems M1 specific.

[janosh]

After brew install geos, the install gets much further but eventually still fails with

ERROR: Could not build wheels for h5py, which is required to install pyproject.toml-based projects

pip logs

Stored in directory: /Users/janosh/Library/Caches/pip/wheels/d7/21/31/f4fabd900c06f400a314f66c9d075ce8d264a78e36636953c3
  Building wheel for h5py (pyproject.toml) ... error
  error: subprocess-exited-with-error

  × Building wheel for h5py (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [7 lines of output]
      running bdist_wheel
      running build
      running build_py
      running build_ext
      Loading library to get build settings and version: libhdf5.dylib
      error: Unable to load dependency HDF5, make sure HDF5 is installed properly
      error: dlopen(libhdf5.dylib, 0x0006): tried: '/opt/homebrew/lib/libhdf5.dylib' (no such file), 'libhdf5.dylib' (no such file), '/usr/local/lib/libhdf5.dylib' (no such file), '/usr/lib/libhdf5.dylib' (no such file), '/private/var/folders/5m/sb0wx83j0m36xk_zswkfnffh0000gn/T/pip-install-t1hjiokd/h5py_acdd7db360994a2da2351357200f4e30/libhdf5.dylib' (no such file), '/usr/local/lib/libhdf5.dylib' (no such file), '/usr/lib/libhdf5.dylib' (no such file)
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for h5py
  Building wheel for llvmlite (setup.py) ... error
  error: subprocess-exited-with-error

  × python setup.py bdist_wheel did not run successfully.
  │ exit code: 1
  ╰─> [11 lines of output]
      running bdist_wheel
      /Users/janosh/.venv/py310/bin/python3.10 /private/var/folders/5m/sb0wx83j0m36xk_zswkfnffh0000gn/T/pip-install-t1hjiokd/llvmlite_d7a0d1ca79904004b9b3251c685f7130/ffi/build.py
      LLVM version... Traceback (most recent call last):
        File "/private/var/folders/5m/sb0wx83j0m36xk_zswkfnffh0000gn/T/pip-install-t1hjiokd/llvmlite_d7a0d1ca79904004b9b3251c685f7130/ffi/build.py", line 220, in <module>
          main()
        File "/private/var/folders/5m/sb0wx83j0m36xk_zswkfnffh0000gn/T/pip-install-t1hjiokd/llvmlite_d7a0d1ca79904004b9b3251c685f7130/ffi/build.py", line 214, in main
          main_posix('osx', '.dylib')
        File "/private/var/folders/5m/sb0wx83j0m36xk_zswkfnffh0000gn/T/pip-install-t1hjiokd/llvmlite_d7a0d1ca79904004b9b3251c685f7130/ffi/build.py", line 134, in main_posix
          raise RuntimeError(msg) from None
      RuntimeError: Could not find a `llvm-config` binary. There are a number of reasons this could occur, please see: https://llvmlite.readthedocs.io/en/latest/admin-guide/install.html#using-pip for help.
      error: command '/Users/janosh/.venv/py310/bin/python3.10' failed with exit code 1
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for llvmlite
  Running setup.py clean for llvmlite
  Building wheel for json2html (setup.py) ... done
  Created wheel for json2html: filename=json2html-1.3.0-py3-none-any.whl size=7609 sha256=7c7f96011016e5e1e38514f5118d5042e507aa39de392d9fddec03f53ec17aad
  Stored in directory: /Users/janosh/Library/Caches/pip/wheels/e0/d8/b3/6f83a04ab0ec00e691de794d108286bb0f8bcdf4ade19afb57
  Building wheel for fido (setup.py) ... done
  Created wheel for fido: filename=fido-4.2.2-py2.py3-none-any.whl size=7405 sha256=877cecce564f9afc71c1eefd09e46c861a1d8ca8cefd40dd70de3973b6c1bad0
  Stored in directory: /Users/janosh/Library/Caches/pip/wheels/e4/52/c4/b282ce656dc1d8d9f48ae2e808515163450ac91f8417768e33
  Building wheel for zope.interface (setup.py) ... done
  Created wheel for zope.interface: filename=zope.interface-5.4.0-cp310-cp310-macosx_12_0_arm64.whl size=208391 sha256=07eeb661aa240c704bb40447215fed63809bfe632615d223af607ea21c2e238d
  Stored in directory: /Users/janosh/Library/Caches/pip/wheels/21/a9/8b/0bfc5594d8e109d5b25d6b69e0cff14d09d93e3522dcb16d2b
  Building wheel for pubchempy (setup.py) ... done
  Created wheel for pubchempy: filename=PubChemPy-1.0.4-py3-none-any.whl size=13834 sha256=0b955dcbb9b1a535c974b640aa09538a937354082a817b5b1738d22df430eea1
  Stored in directory: /Users/janosh/Library/Caches/pip/wheels/90/7c/45/18a0671e3c3316966ef7ed9ad2b3f3300a7e41d3421a44e799
  Building wheel for wrapt (setup.py) ... done
  Created wheel for wrapt: filename=wrapt-1.13.3-cp310-cp310-macosx_12_0_arm64.whl size=33891 sha256=b6272002b4d746442600fdf072e6c4e934f91f4257f94bb697aacb93adc2930a
  Stored in directory: /Users/janosh/Library/Caches/pip/wheels/e2/0a/6b/521a4b208be1ff910c1bb6e21dd1a79ac84e54eaf0163a258c
  Building wheel for strict-rfc3339 (setup.py) ... done
  Created wheel for strict-rfc3339: filename=strict_rfc3339-0.7-py3-none-any.whl size=18149 sha256=1afa3044b6a2e6bf05347deab6a7bc2e126239584901429d2f3e583049de7354
  Stored in directory: /Users/janosh/Library/Caches/pip/wheels/d4/c2/43/78ecbcc49dd7594ca41c72be6ddb7844a486bae4b306fb3d14
Successfully built numba shapely json2html fido zope.interface pubchempy wrapt strict-rfc3339
Failed to build h5py llvmlite
ERROR: Could not build wheels for h5py, which is required to install pyproject.toml-based projects

[janosh]

Ah, looks like there's a well known issue with llvmlite on M1 and numba is working on fixing it.

[janosh]

FWIW, this finally succeeded:

brew install llvm@11
LLVM_CONFIG="/opt/homebrew/Cellar/llvm@11/11.1.0_4/bin/llvm-config" pip install llvmlite

brew install hdf5
export HDF5_DIR=/opt/homebrew/Cellar/hdf5/1.12.1
pip install --no-binary=h5py h5py

pip install crystal-toolkit

[mkhorton]

Thanks for the report and update on this issue.

It's possible this dependency can be relaxed in future versions too, but hopefully with arm64 support getting gradually better throughout the ecosystem this won't be necessary.

[janosh]

The numba dependence still causes installation problems (at least on M1 Macs).

Quoting @mkhorton: numba is used by dscribe, which is used by both matminer (an emmet-core dep) and also the local_env feature in crystal toolkit. it could certainly be made optional. It's a headache because numba doesn't support the latest numpy, and pymatgen is often tracking a later numpy version (because it requires binary compatibility with the compiled extensions).

Decided to go with optional numba dependence. Will just need a corresponding change here to check for an ImportError:

crystaltoolkit/crystal_toolkit/components/localenv.py

Lines 11 to 12 in ae03e10

	from dscribe.descriptors import SOAP
	from dscribe.kernels import REMatchKernel