import pyg_lib fails due to OSError: dlopen ... (no such file, not in dyld cache) on macOS

[semihcanturk]

🐛 Describe the bug

I'm unable to get pyg-lib to import on my Mac. It seems to affect both Intel and M1 Macs (I'm on an Intel, but an M1 user reported the same issue: #211 (comment)). The original issue in the thread may not be relevant, thus I'm opening a new issue.

Minimal example is simply importing pyg_lib:

import pyg_lib

In my case, I was importing torch_geometric.graphgym.config, resulting in the following stack trace:

  File "/Users/semo/PycharmProjects/pretrained-pe/graphgym/act/example.py", line 3, in <module>
    from torch_geometric.graphgym.config import cfg
  File "/Users/semo/PycharmProjects/pretrained-pe/venv/lib/python3.10/site-packages/torch_geometric/__init__.py", line 4, in <module>
    import torch_geometric.data
  File "/Users/semo/PycharmProjects/pretrained-pe/venv/lib/python3.10/site-packages/torch_geometric/data/__init__.py", line 1, in <module>
    from .data import Data
  File "/Users/semo/PycharmProjects/pretrained-pe/venv/lib/python3.10/site-packages/torch_geometric/data/data.py", line 20, in <module>
    from torch_sparse import SparseTensor
  File "/Users/semo/PycharmProjects/pretrained-pe/venv/lib/python3.10/site-packages/torch_sparse/__init__.py", line 39, in <module>
    from .storage import SparseStorage  # noqa
  File "/Users/semo/PycharmProjects/pretrained-pe/venv/lib/python3.10/site-packages/torch_sparse/storage.py", line 7, in <module>
    from torch_sparse.utils import Final, index_sort
  File "/Users/semo/PycharmProjects/pretrained-pe/venv/lib/python3.10/site-packages/torch_sparse/utils.py", line 5, in <module>
    import torch_sparse.typing
  File "/Users/semo/PycharmProjects/pretrained-pe/venv/lib/python3.10/site-packages/torch_sparse/typing.py", line 2, in <module>
    import pyg_lib  # noqa
  File "/Users/semo/PycharmProjects/pretrained-pe/venv/lib/python3.10/site-packages/pyg_lib/__init__.py", line 38, in <module>
    load_library('libpyg')
  File "/Users/semo/PycharmProjects/pretrained-pe/venv/lib/python3.10/site-packages/pyg_lib/__init__.py", line 35, in load_library
    torch.ops.load_library(spec.origin)
  File "/Users/semo/PycharmProjects/pretrained-pe/venv/lib/python3.10/site-packages/torch/_ops.py", line 573, in load_library
    ctypes.CDLL(path)
  File "/Users/semo/.pyenv/versions/3.10.9/lib/python3.10/ctypes/__init__.py", line 374, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: dlopen(/Users/semo/PycharmProjects/pretrained-pe/venv/lib/python3.10/site-packages/libpyg.so, 0x0006): Library not loaded: /Users/runner/hostedtoolcache/Python/3.10.8/x64/lib/libpython3.10.dylib
  Referenced from: <16F9F1F8-8305-3FAA-9EB1-C519E67F7DAC> /Users/semo/PycharmProjects/pretrained-pe/venv/lib/python3.10/site-packages/libpyg.so
  Reason: tried: '/Users/runner/hostedtoolcache/Python/3.10.8/x64/lib/libpython3.10.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/Users/runner/hostedtoolcache/Python/3.10.8/x64/lib/libpython3.10.dylib' (no such file), '/Users/runner/hostedtoolcache/Python/3.10.8/x64/lib/libpython3.10.dylib' (no such file), '/usr/local/lib/libpython3.10.dylib' (no such file), '/usr/lib/libpython3.10.dylib' (no such file, not in dyld cache)

Environment

• pyg-lib version: 0.1.0/latest from source(0.1.0+pt113)
• PyTorch version: 1.13.1/2.0.0
• OS: macOS 13.2.1
• Python version: 3.10.8
• CUDA/cuDNN version: None/CPU (Intel Core i9)
• How you installed PyTorch and pyg-lib (conda, pip, source): Both, see below
• Any other relevant information:

Things I've tried:

• Install torch & torch_geometric packages via conda and pyg_lib via pip in a clean env
• Install torch, torch_geometric packages and pyg_lib via pip only
• Test with PyTorch 2.0.0 and 1.13.1 (I'm positive the same setup worked with 1.13.1 and latest torch_geometric/pyg_lib last week)

[semihcanturk]

Turns out a (partial) solution is available in the pytorch_geometric repo, which is simply uninstalling pyg-lib: pyg-team/pytorch_geometric#6530 (comment). Nevertheless keeping this issue open as this is likely to be undesirable behavior.

[rusty1s]

Please see my comment in #211 (comment).

[semihcanturk]

That didn't work unfortunately, but I managed to resolve the issues by reinstalling all packages from source. I think that way it updates the DYLD_LIBRARY_PATH correctly during setup.

pip install git+https://github.com/pyg-team/pyg-lib.git
pip install git+https://github.com/rusty1s/pytorch_sparse.git
pip install git+https://github.com/rusty1s/pytorch_scatter.git
pip install git+https://github.com/rusty1s/pytorch_cluster.git

[rusty1s]

I hit the same issue today :( I tried many different things, but in all scenarios the final binary was searching for a Python folder in /Library/Frameworks/Python.framework/Versions. Installing it from the universal install at https://www.python.org/downloads/macos/ fixed it for me.

[Martins6]

It seems that even installing from Python official solves it, but the error message changes somewhat @rusty1s.

Traceback (most recent call last):
  File "/Users/adrielmartins/Documents/gcaptain/testingsomething.py", line 1, in <module>
    import pyg_lib
  File "/Users/adrielmartins/Documents/gcaptain/.venv/lib/python3.11/site-packages/pyg_lib/__init__.py", line 39, in <module>
    load_library('libpyg')
  File "/Users/adrielmartins/Documents/gcaptain/.venv/lib/python3.11/site-packages/pyg_lib/__init__.py", line 36, in load_library
    torch.ops.load_library(spec.origin)
  File "/Users/adrielmartins/Documents/gcaptain/.venv/lib/python3.11/site-packages/torch/_ops.py", line 1295, in load_library
    ctypes.CDLL(path)
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/ctypes/__init__.py", line 376, in __init__
    self._handle = _dlopen(self._name, mode)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^
OSError: dlopen(/Users/adrielmartins/Documents/gcaptain/.venv/lib/python3.11/site-packages/libpyg.so, 0x0006): Symbol not found: __ZN3c1010Dispatcher17runRecordFunctionERN2at14RecordFunctionENSt3__117reference_wrapperIKNS_14FunctionSchemaEEENS_11DispatchKeyE
  Referenced from: <75FFC412-93B5-322B-8E6D-268DA3498CF4> /Users/adrielmartins/Documents/gcaptain/.venv/lib/python3.11/site-packages/libpyg.so
  Expected in:     <9C467D80-5BAC-3156-B480-F0FF7E45DEF2> /Users/adrielmartins/Documents/gcaptain/.venv/lib/python3.11/site-packages/torch/lib/libtorch_cpu.dylib

It seems that is still expecting packages elsewhere. I'm running this on M3 Pro Mac, using Python3.11.8, torch 2.3.0, torch_geometric >=2.5.3.

Do you have any other proposal, please?