Compiling YDF for MacOSX 13 or 14 - ARM64

[lusis-ai]

Hi,

Wheel for MacOSX 14 ARM64 are proposed here.

First note that when installing ydf (0.2.0) for python 3.10 on MacOS 14 ARM64 (M1), if I import pandas AND ydf I get a segmentation fault and the python interpreter crash instantly after the two imports. Importing only one of these two packages works.

Now my true question : How do you compile YDF python port to get the wheel for MacOSX 13 or 14 ARM64 as these instructions failed.

PYTHON_BIN=python3.9  
./tools/test_pydf.sh  
./tools/build_pydf.sh $PYTHON_BIN

When trying to compile it, GCC9 is required. So I installed it with brew (gcc@9 formula), but it's only supported on Intel Mac, so ydf cannot be compiled.

How are you doing it on your side ?

Many Thanks for your help

[rstz]

Hi,
sorry to hear this. I've been meaning to update the build instructions. Can you try this from a fresh virtualenv?

RUN_TESTS=0 COMPILERS="clang" ./tools/test_pydf.sh
./tools/build_pydf.sh python

[lusis-ai]

Hi, thanks you.

It still failed.

Here is my setup and procedure.

MacOSX 13.5 (ventura) - M2 Max
gcc --version
Apple clang version 15.0.0 (clang-1500.1.0.2.5)
Target: arm64-apple-darwin22.6.0

1. I create a new Conda env (miniforge3) : "yggdrasilcompil"
2. I Install python 3.10 (conda install python = 3.10)
3. I install last version of bazel (conda install bazel)

Then I type :

1. RUN_TESTS=0
2. COMPILERS="clang"
3. ./tools/test_pydf.sh (I'm under yggdrasil_decision_forests/port/python)

So it fails after a while with (I removed the stack trace, just keeping errors for readability):

ERROR: An error occurred during the fetch of repository 'local_config_cc':
....
Auto-Configuration Error: Cannot find gcc or CC (gcc-9); either correct your path or set the CC environment variable
ERROR: /DEFAULT.WORKSPACE.SUFFIX:520:13: fetching cc_autoconf rule //external:local_config_cc: Traceback (most recent call last):
....

Thanks for your help

[lusis-ai]

For conda I use (and we use in all our production and research system) not Anaconda but miniforge : https://github.com/conda-forge/miniforge

After a few search, it seems it can be related to Bazel version.
What version do you use ?

[rstz]

We're using 5.3.0, though 6.1.0 should work as well

[rstz]

IIUC, the bazel team recommends using https://github.com/bazelbuild/bazelisk which picks the bazel based on what .bazelversion specifies

[lusis-ai]

Succeded with Bazel = 6.3.1, Python 3.10

The problem was due to this combination of versions + the way environment variable are set.

Setting COMPILERS="clang" is different from export COMPILERS="clang" for instance, as it cannot be retrieved with the env dict in the scripts.

Using export make it work.

real	4m31.638s
user	0m0.169s
sys	0m0.174s
+ [[ 0 = 1 ]]
+ echo 'PYDF build / test complete.'
PYDF build / test complete.

Now I have to find where the .whl is and test.

[rstz]

Nice! You can create the wheel with

PY_BINARY=python3.9  # substitute for the python binary you want to build the wheel for
./tools/build_pydf.sh $PY_BINARY

Note that this script packages the compiled binaries from the bazel-bin directories together with the python files and protobuf wrappers and essentially just zip's them into a wheel with setuptools

[lusis-ai]

Hi, it seems to work now.

Only issue : pandas must be installed from pip and not from conda, else when loading pandas AND ydf, it results in a Segmentation fault in the python interpreter.

I installed the rest including JupyterLab with conda and there is no issue, only pandas from conda fails to work with ydf.

Best regards

[rstz]

Thank you!

[lusis-ai]

In fact after double checking, it's not so simple. Maybe it comes from another package, pandas alone is not sufficient to create the issue (installing with conda or pip works when there is only these two packages while I have a bigger config where it fails). I have an environment having the issue but I would like to isolate the minimal combination. Will check it when I will have time and let you know, maybe not soon.

[lusis-ai]

On more question: If I want to use the CLI, how can I compile it and where can I found it ?

[rstz]

On Linux, doing INSTALL_DEPENDENCIES=1 BUILD=1 PACK=1 ./tools/build_binary_release.sh builds the CLI tools and places them (zipped) in dist/cli_linux.zip. Tried on Ubuntu 20.04

On MacOS, installing dependencies is not necessary, so BUILD=1 PACK=1 ./tools/build_binary_release.sh does the trick (tried on Apple M1). The script still assumes bazelisk is available, which you can download with

curl -L -o bazelisk https://github.com/bazelbuild/bazelisk/releases/download/v1.14.0/bazelisk-darwin-arm64
chmod +x bazelisk

You can peek into the tools/build_binary_release.sh script for configuration options, e.g. changing to

  cp -f WORKSPACE_NO_TF WORKSPACE

  BAZEL="./bazelisk"
  FLAGS="--config=linux_cpp17 --config=linux_avx2 --features=-fully_static_link --repo_env=CC=clang-12"

will remove support for Tensorflow data formates but give you faster compilation and probably smaller binaries. You can also change the BAZEL variable to point to your bazel installation.

I'll attach compiled binaries for v.1.8.0 to the github release later today.

[lusis-ai]

I've found the situation producing a segmentation fault.

It comes from pyarrow.

Setup : MacOSX 13 or 14, Mac M hardware

Prerequisite : Install miniforge3

% conda create --name ydfpandasissue
% conda activate ydfpandasissue
% conda install python=3.10
% conda install pandas
% pip install ydf-0.2.0-cp310-cp310-macosx_13_0_arm64.whl

When running this program (ydf_test.py), it works.

import ydf
import pandas as pd
import numpy as np

dataset = {
    "x1": np.array([0, 0, 0, 1, 1, 1]),
    "x2": np.array([1, 1, 0, 0, 1, 1]),
    "y": np.array([0, 0, 0, 0, 1, 1]),
}

model = ydf.CartLearner(label="y", min_examples=1, task=ydf.Task.CLASSIFICATION).train(dataset)
print(model.describe())

Now install pyarrow from conda or pip the result is the same, only the error message is different: it fails

% conda install pyarrow
% python ydf_test.py
zsh: segmentation fault  python ydf_test.py

% conda uninstall pyarrow
% pip install pyarrow
% python ydf_test.py
libc++abi: terminating due to uncaught exception of type std::__1::system_error: mutex lock failed: Invalid argument
zsh: abort      python ydf_test.py

[lusis-ai]

Hi,

After doing a git pull of the very last release today (1.9.0) the same compilation than above, that was previously working, failed with many messages like this, listing many .h:

ERROR: /XXXXXXXXX/miniforge3/envs/yggdrasilcompil/share/bazel/4aa755d4b0ad00344bfbbd099450b5d7/external/local_config_python/_python3/BUILD:25:8: declared output 'external/local_config_python/_python3/_python3_include/traceback.h' is a dangling symbolic link

[lusis-ai]

Something strange : when trying from another user account on my mac (Perso vs Pro), it works. I'm investigating.

[lusis-ai]

It works now, after recreating my conda compilation environment instead of uprading the python version of the previous one.

[lusis-ai]

Great,

New version (0.3.0) can be directly installed through pip on a Mac ARM and it works perfectly.
No need to compile it anymore.
Great, thanks a lot.

[rstz]

Perfect, thank you for letting me know

[lusis-ai]

Hi,

The 0.4.1 is not available through pip.

So I tried to compile and it fails for several reasons.

OS : MacOSX 14.4.1
Python 3.11.8

My compilation env is the same than previously described above, same Bazel and python. Only the OS version was updated since.

1. an issue with uname --machine : unknown param, you must replace it by uname -m in the script
2. a write access failed to files in tmp_package so I needed to run it as sudo (it's never a good thing)
3. a missing line at the end of the build_pydf.py file - the current is ending by assemble_files ${PYTHON} so it does not compile anything
4. After adding the builing line, the ydf dist is finaly generated but failed to import as it says it was compiled with python 3.12 ... but I'm on 3.11

So can you either fix the compilation procedure and files or push a compiled macosx for ARM to pypi ?

Many thanks for your help

[chatnord]

Hello!

Additional comment on the above:

• the pre-compiled version of the library (0.3.0) does not install on MacOSX 14.5 (ARM64) - even when I download the wheels and manually change the OSs version, I get hit with "Wrong architecture"
• I am currently re-compiling it from scratch, but I am not very confident that it will work (given the above comments)

Do you guys have any plans to release a new pre-compiled version or fix the compilation procedure for MacOS ARM machines? Thank you!

[rstz]

Hi,

yes, we plan to release for MacOS Arm64 very soon (and possibly update the release instructions). AFAICT, once we change the linkopts in yggdrasil_decision_forests/port/python/ydf/cc/BUILD to

pybind_extension(
    name = "ydf",
    srcs = ["ydf.cc"],
    deps = [
        "//ydf/dataset:dataset_cc",
        "//ydf/learner:learner_cc",
        "//ydf/learner:worker_cc",
        "//ydf/metric:metric_cc",
        "//ydf/model:model_cc",
        "//ydf/utils:log_cc",
        "@pybind11_protobuf//pybind11_protobuf:native_proto_caster",
    ],
    linkopts = select({
        "@bazel_tools//src/conditions:darwin": ["-Wl,-exported_symbol,_PyInit_ydf"],
        "//conditions:default": [],
    }),
)

compiling and interplay with pyarrow / tensorflow / tensorflow-decision-forest / protobuf etc. should work.

[rstz]

Update: Support for MacOS has been restored with version 0.7.0

[chatnord]

Thanks a lot!

If anyone encounters the same issue as I did, if you try to install it in a conda environment it might not work, as conda by default created an osx-64 env.

To fix it, create the env as follows:

CONDA_SUBDIR=osx-arm64 conda create -n my_env python