[BSE-4203] Use Pixi for Pip Build Deps #46
Conversation
srilman
force-pushed
the
slade/pixi-pip
branch
from
cc4c442 to
fab1eaa
Compare
pyproject.toml
Outdated
| # Increase pip debugging output | ||
| build-verbosity = 1 | ||
| build-frontend = { name = "pip" } | ||
| #, args = ["--only-binary", "mpi4py", "--extra-index-url", "https://pypi.anaconda.org/mpi4py/simple"] } |
There was a problem hiding this comment.
TODO: Remove when addressing comments
| needs: build_bodo_linux_wheels | ||
| steps: | ||
| - uses: actions/download-artifact@v4 | ||
| id: download-artifact | ||
| with: | ||
| pattern: cibw-wheels-ubuntu-* | ||
| path: . | ||
| - uses: actions/setup-python@v5 | ||
| with: | ||
| python-version: ${{ matrix.python_version }} |
There was a problem hiding this comment.
I did this because I couldn't add set -exo pipefail inside the docker container. Thus, the job kept passing even though the test failed. This way, the job fails is the test fails
| pip install setuptools_scm | ||
| echo "bodo_version=`python -m setuptools_scm`" >> $GITHUB_OUTPUT | ||
| outputs: | ||
| bodo_version: ${{ steps.get_version.outputs.bodo_version }} |
There was a problem hiding this comment.
We ran into a problem where cibuildwheel was using the wrong version during build. This fixes that by fetching the version and fixing it beforehand
| CPPFLAGS="-isystem /project/.pixi/envs/pip-cpp/include" | ||
| CC=/project/.pixi/envs/pip-cpp/bin/x86_64-conda-linux-gnu-gcc | ||
| CXX=/project/.pixi/envs/pip-cpp/bin/x86_64-conda-linux-gnu-g++ | ||
| DISABLE_CCACHE=1 |
There was a problem hiding this comment.
CIBW is annoying and wont combine the envs from CIBW_ENVIRONMENT and CIBW_ENVIRONMENT_LINUX on Linux, so I had to copy them here.
Plus, Pixi shell activation wasn't working inside of the docker container, so I had to manually set envs to find the compiler correctly.
srilman
changed the title
srilman
changed the title
| @@ -151,6 +176,7 @@ jobs: | |||
| pip install bodo bodosql | |||
| - name: Test | |||
| run: | | |||
| set -exo pipefail | |||
There was a problem hiding this comment.
| set -exo pipefail | |
| set -eo pipefail |
Had issues with this printing tokens
There was a problem hiding this comment.
I'm actually fixing that in a separate PR. Its a different issue, so we can keep this
| @@ -53,6 +76,7 @@ jobs: | |||
| pip install bodo | |||
| - name: Test | |||
| run: | | |||
| set -exo pipefail | |||
There was a problem hiding this comment.
| set -exo pipefail | |
| set -eo pipefail |
Had issues with this printing tokens
| @@ -34,8 +34,10 @@ def patch_lib(fpath): | |||
| # @rpath/libarrow.500.dylib (compatibility version 500.0.0, current version 500.0.0) | |||
| # @rpath/libc++.1.dylib (compatibility version 1.0.0, current version 1.0.0) | |||
| # /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1311.0.0) | |||
| print("Loaded Libs for", fpath) | |||
There was a problem hiding this comment.
do we want to keep prints in this file?
There was a problem hiding this comment.
Yes for sure, it was useful for debugging
There was a problem hiding this comment.
And it doesn't cause any issues
Changes included in this PR
Replace the conda-lock usage in our Pip pipelines with Pixi.
We use conda-lock to set up a reproducible environment for our C++ dependencies. However, its not really reproducible because we construct the lockfile during the build process. Furthermore, we install more dependencies than we need for building. We do all this because conda-lock doesn't have the features we need.
Pixi has features like multiple environments that we can use instead. Furthermore, with Pixi, we can only install the C++ dependencies during the Pip build.
Some other changes:
Note that we unfortunately need to still use the C++ compilers from Pixi / Conda. Ideally, we should use the compiler provided by cibuildwheel for best compatibility, but we can't until we find other sources for our C++ dependencies. Most are already provided (libcurl on the system, AWS SDK from PyArrow, Boost through CMake build), but HDF5 is the hard one.
Testing strategy
Running the Pip CIs.
User facing changes
Ideally none.
Checklist
[run CI]in your commit message.