[ci] [python-package] add arm64 macOS wheels

[jameslamb]

Contributes to #5328
Contributes to #4229
Might help with #5764

• adds arm64 macOS wheels for the Python package (for the M1/M2/M3 Macs)
• patches OpenMP linking details in lib_lightgbm.dylib so that a pip-installed lightgbm won't segfault when used in a conda environment

Notes for Reviewers

Proposed approach for building arm64 wheels

It seems that Azure DevOps does not support arm64 macOS runners... only GitHub Actions: actions/runner-images#8971.

So I'm proposing using GitHub Actions new macos-14 runners, which are arm64 (#5328 (comment)).

Proposed approach for fixing loading errors

The problem "using a pip-installed lightgbm inside a conda environment segfaults" is not specific to arm64. It's been around in lightgbm's macOS support for a few years. I decided to try to fix it here to be able to test the new arm64 wheels.

I'm proposing the following approach:

• continue dynamically linking to OpenMP
• do not vendor a copy of libomp
• patch the search paths in lib_lightgbm.dylib to be more forgiving about where libomp.dylib is found

In short:

• the hard-coded path like /opt/homebrew/opt/libomp/lib/libomp.dylib currently stored in lib_lightgb.dylib forces the macOS loader to load a library from that exact path, even if there is already a libomp.dylib loaded from some other source (like the llvm-openmp shipped y conda)
• this PR replaces that with @rpath/libomp.dylib, which means that at runtime of lightgbm, the loader will do the following (in order)
  1. if there is a libomp.dylib already loaded, just use that
  2. if not, first search for one in /opt/homebrew/opt/libomp/lib/
  3. if there isn't one there, search wherever one was found at build time (which might also be /opt/homebrew/opt/libomp/lib/)
  4. if one still hasn't been found, search standard paths like /usr/local/lib and /usr/lib

See "References" section for the MANY resources I consulted to get to this point. Particularly:

• "Dynamic Library Usage Guidelines" (Apple developer docs)
• "Loading Dynamic Libraries on Mac" (Clark Kromenaker's blog)

How I tested this

downloaded the wheel built in CI (click me)

Got a run ID from CI, looking at the URL.

e.g. for this build: https://github.com/microsoft/LightGBM/actions/runs/9476495757/job/26109485388?pr=6391

the run id is 9476495757.

Then, following https://docs.github.com/en/actions/managing-workflow-runs/downloading-workflow-artifacts?tool=cli, used the GitHub CLI to download the wheels. A fine-grained access token with just read-only permission on Actions should be enough.

From the root of the repo, on my M2 Mac laptop:

rm -rf ./whl-tmp
mkdir -p ./whl-tmp

RUN_ID=9476495757
gh run download \
    --repo microsoft/LightGBM \
    "${RUN_ID}" \
    -n macosx-arm64-wheel \
    --dir ./whl-tmp

created a new conda environment (click me)

Created a conda environment.

conda create \
  --name delete-me-lgb-dev \
  --yes \
  --file ./.ci/conda-envs/ci-core.txt \
  python=3.10 \
  pydistcheck

source activate delete-me-lgb-dev

Inspected the wheel with `pydistcheck` (click me)

pydistcheck \
    --inspect ./whl-tmp/lightgbm-4.3.0.99-py3-none-macosx_14_0_arm64.whl

==================== running pydistcheck ====================                                                                                                  
                      
checking './whl-tmp/lightgbm-4.3.0.99-py3-none-macosx_14_0_arm64.whl'
----- package inspection summary -----
file size
  * compressed size: 1.5M
  * uncompressed size: 6.5M
  * compression space saving: 76.7%
contents
  * directories: 0
  * files: 16 (1 compiled)
size by extension
  * .dylib - 6.0M (93.2%)
  * .py - 0.4M (6.4%)
  * no-extension - 21.7K (0.3%)
  * .txt - 9.0B (0.0%)
  * .typed - 0.0B (0.0%)
largest files
  * (6.0M) lightgbm/lib/lib_lightgbm.dylib
  * (0.2M) lightgbm/basic.py
  * (65.4K) lightgbm/dask.py
  * (63.6K) lightgbm/sklearn.py
  * (35.7K) lightgbm/engine.py
------------ check results -----------
errors found while checking: 0

Installed from the wheel (on my M2 mac, using macOS 14.4.1).

pip install \
    --no-deps \
    ./whl-tmp/lightgbm-4.3.0.99-py3-none-macosx_14_0_arm64.whl

Ran the tests

pytest tests/python_package_test

Those all passed! Removed the brew-installed libomp on my system and ran that again.

brew uninstall libomp
pytest tests/python_package_test

They passed again 🎉

Can lightgbm really safely link to any libomp.dylib?

This project's macOS wheels have already been doing that for years, by shipping with an absolute path that evaluates to "whatever brew install libomp installs", and no limits comparing the version linked against at build time to the one used at runtime.

The changes in this PR are actually safer than the existing behavior of lightgbm's macOS wheels because they avoid a well-known source of runtime issue (loading a libomp.dylib that conflicts with the one loaded by scikit-learn).

Why not just vendor libomp.dylib like other libraries do?

Other projects like lightgbm bundle a copy of LLVM OpenMP (libomp.dylib) in their wheels.

• numpy and scipy: https://pypackaging-native.github.io/key-issues/native-dependencies/blas_openmp/
• xgboost: [CI] Build Python wheels for MacOS (x86_64 and arm64) dmlc/xgboost#7621
• scikit-learn: https://github.com/scikit-learn/scikit-learn/blob/7e8ad632ff7aec09e99f0c4b8e493e13e19aef14/build_tools/wheels/build_wheels.sh#L20-L22

That comes with a significantly reduced risk of "cannot find libomp.dylib" types of runtime issues, but in exchange for:

• risk of conflicts caused by multiple copies of the same library being loaded
• larger wheels (and therefore environments)
• lack of coordination across libraries (e.g. one's omp_set_num_threads() does not affect all the loaded OpenMP libraries, and so projects' parallelism can fight each other and lead to oversubscription)
• greater risk of ABI incompatibilities (e.g. a version loaded at runtime having a different interface than the one that was built against)

Since lightgbm has already been using dynamic linking + not vendoring for several years and since "you have to brew install libomp" has generally not been a problem for macOS users, I'm proposing continuing those practices in this release.

Will these CMake changes break the lightgbm conda package?

They shouldn't.

conda-build rewrites paths in binaries it produces to correctly link between its packages. See "Making packages relocatable" (conda docs)

References

• https://discuss.python.org/t/conflicting-binary-extensions-in-different-packages/25332/2
• https://stackoverflow.com/questions/3146274/is-it-ok-to-use-dyld-library-path-on-mac-os-x-and-whats-the-dynamic-library-s
• http://clarkkromenaker.com/post/library-dynamic-loading-mac/
• https://cmake.org/pipermail/cmake/2019-September/069996.html
• https://gitlab.kitware.com/cmake/community/-/wikis/doc/cmake/RPATH-handling
• https://stackoverflow.com/questions/73263834/cmake-how-to-set-rpath-in-a-shared-library-with-only-target-link-directories-w
• https://stackoverflow.com/questions/43330165/how-to-link-a-shared-library-with-cmake-with-relative-path/43333118#43333118
• https://stackoverflow.com/a/51504440/3986677
• https://stackoverflow.com/a/9263735/3986677
• [CI] Build Python wheels for MacOS (x86_64 and arm64) dmlc/xgboost#7621
• BLD Fixes osx build by downgrading to 11.X [cd build] scikit-learn/scikit-learn#21227
• SEGFAULT on MacOS with scikit-learn==1.0.0 dmlc/xgboost#7355
• https://developer.apple.com/library/archive/documentation/DeveloperTools/Conceptual/DynamicLibraries/100-Articles/DynamicLibraryUsageGuidelines.html
• https://developer.apple.com/library/archive/documentation/DeveloperTools/Conceptual/DynamicLibraries/100-Articles/RunpathDependentLibraries.html
• https://stackoverflow.com/a/57290032/3986677
• Python package looks for library in wrong path #5106 (comment)

[jameslamb]

tagging some others who I think might be interested in this approach / topic:

@hcho3 @trivialfis @thomasjpfan

[guolinke]

Thank you, LGTM!

[hcho3]

Thanks for pinging me. I would like to adopt a similar approach for XGBoost so that we don't have to vendor libomp.

[jameslamb]

Looks like this needs a bit more work on the R side.

[100%] Linking CXX shared library /private/var/folders/vy/h7r6h43j203gstfh6fj9_tyh0000gn/T/RtmpuVEgw6/Rbuild34b875aa9c05/lightgbm/src/lightgbm.so
Replacing hard-coded OpenMP install_name with '@rpath/libgomp.dylib'...
error: /Applications/Xcode_14.3.1.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/install_name_tool: can't open file: /private/var/folders/vy/h7r6h43j203gstfh6fj9_tyh0000gn/T/RtmpuVEgw6/Rbuild34b875aa9c05/lightgbm/src/lib_lightgbm.dylib (No such file or directory)
make[3]: *** [/private/var/folders/vy/h7r6h43j203gstfh6fj9_tyh0000gn/T/RtmpuVEgw6/Rbuild34b875aa9c05/lightgbm/src/lightgbm.so] Error 1
make[3]: *** Deleting file `/private/var/folders/vy/h7r6h43j203gstfh6fj9_tyh0000gn/T/RtmpuVEgw6/Rbuild34b875aa9c05/lightgbm/src/lightgbm.so'

https://github.com/microsoft/LightGBM/actions/runs/9499965097/job/26181969335?pr=6391

I'll look later, but I suspect the issue is my unconditional use of .dylib here:

LightGBM/CMakeLists.txt

Line 747 in f240bb7

"${PROJECT_SOURCE_DIR}/lib_lightgbm.dylib"

It should probably be templating through CMAKE_SHARED_LIBRARY_SUFFIX, to account for this:

LightGBM/R-package/src/install.libs.R

Line 187 in 4e74403

shlib_ext_arg <- sprintf("-DCMAKE_SHARED_LIBRARY_SUFFIX_CXX='%s'", SHLIB_EXT)

I really like that those tests caught the use of gcc + libgomp on macOS! Thankful every day that @StrikerRUS prioritized compiler coverage so much when setting up all the testing in this repo.

[jameslamb]

I had commented this out in #6377 (comment), to keep merging things in preparation for the release.

Looks like the issues with filelock have been resolved by a new release of that project, and this now runs successfully again 🎉

[jameslamb]

Given @guolinke's approval + @jmoralez 's 🚀 , I'm going to merge this once CI passes.

Then we'll just be onto the final release tasks over in #6439, like checking the docs and adding verisonadded markers (I'll do those).

Getting close!