Remove NumPy <2 pin, remove proj pin #1441
Conversation
seberg
added
non-breaking
github-actions
bot
added
conda
|
I merged in the upstream to see if CI is failing in the same way we see here: rapidsai/docker#709 (comment) |
|
Closing and reopening to restart CI now that cuDF is done: rapidsai/cudf#16300 |
|
Seeing some unexpected and unrelated failures that appear to be due to a cuDF change Running CI in a no change PR to see if this reproduces there: #1444 Edit: Seeing the same error over there ( #1444 (comment) ). Following up separately |
|
On the Conda side, it looks like we are still seeing NumPy 1 in CI. Think we are missing another dependency. Will need to take a closer look |
|
Took all of the dependencies on CI (except RAPIDS ones) and tried to create a Conda environment with them locally on Linux ARM (though there should be no difference between x86_64 in this regard). It created the environment using NumPy 2 Didn't see anything obviously missing in this repo The other RAPIDS dependencies are all migrated Not sure what we are missing. May need to dig deeper later |
Strongly suspect this is a result of the changes from rapidsai/cudf#15483. I'll work on adapting |
|
Depends on PR: #1447 |
Contributes to rapidsai/build-planning#33. Proposes the following for `cuspatial` wheels: * add build and runtime dependencies on `libcudf` wheels * stop vendoring copies of `libcudf.so`, `libnvcomp.so`, `libnvcomp_bitcomp.so`, and `libnvcomp_gdeflate.so` - *(load `libcudf.so` dynamically at runtime instead)* And other related changes for development/CI: * combine all `pip install` calls into 1 in wheel-testing scripts - *like rapidsai/cudf#16575 - *to improve the chance that packaging issues are discovered in CI* * `dependencies.yaml` changes: - more use of YAML anchors = less duplication - use dedicated `depends_on_librmm` and `depends_on_libcudf` groups * explicitly pass a package type to `gha-tools` wheel uploading/downloading scripts ## Notes for Reviewers ### Benefits of these changes Unblocks CI in this repo (ref: #1444 (comment), #1441 (comment)). Reduces wheel sizes for `cuspatial` wheels by about 125MB 😁 | wheel | size (before) | size (this PR) | |:-----------:|-------------:|---------------:| | `cuspatial` | 146.0M | 21M | | `cuproj ` | 0.9M | 0.9M | |**TOTAL** | **146.9M** | **21.9M** | *NOTES: size = compressed, "before" = 2024-08-21 nightlies (c60bd4d), CUDA = 12, Python = 3.11* <details><summary>how I calculated those (click me)</summary> ```shell # note: 2024-08-21 because that was the most recent date with # successfully-built cuspatial nightlies # docker run \ --rm \ -v $(pwd):/opt/work:ro \ -w /opt/work \ --network host \ --env RAPIDS_NIGHTLY_DATE=2024-08-21 \ --env RAPIDS_NIGHTLY_SHA=c60bd4d \ --env RAPIDS_PR_NUMBER=1447 \ --env RAPIDS_PY_CUDA_SUFFIX=cu12 \ --env RAPIDS_REPOSITORY=rapidsai/cuspatial \ --env WHEEL_DIR_BEFORE=/tmp/wheels-before \ --env WHEEL_DIR_AFTER=/tmp/wheels-after \ -it rapidsai/ci-wheel:cuda12.5.1-rockylinux8-py3.11 \ bash mkdir -p "${WHEEL_DIR_BEFORE}" mkdir -p "${WHEEL_DIR_AFTER}" py_projects=( cuspatial cuproj ) for project in "${py_projects[@]}"; do # before RAPIDS_BUILD_TYPE=nightly \ RAPIDS_PY_WHEEL_NAME="${project}_${RAPIDS_PY_CUDA_SUFFIX}" \ RAPIDS_REF_NAME="branch-24.10" \ RAPIDS_SHA=${RAPIDS_NIGHTLY_SHA} \ rapids-download-wheels-from-s3 python "${WHEEL_DIR_BEFORE}" # after RAPIDS_BUILD_TYPE=pull-request \ RAPIDS_PY_WHEEL_NAME="${project}_${RAPIDS_PY_CUDA_SUFFIX}" \ RAPIDS_REF_NAME="pull-request/${RAPIDS_PR_NUMBER}" \ rapids-download-wheels-from-s3 python "${WHEEL_DIR_AFTER}" done du -sh ${WHEEL_DIR_BEFORE}/* du -sh ${WHEEL_DIR_BEFORE} du -sh ${WHEEL_DIR_AFTER}/* du -sh ${WHEEL_DIR_AFTER} ``` </details> Reduces the amount of additional work required to start shipping `libcuspatial` wheels. ### Background This is part of ongoing work towards packaging `libcuspatial` as a wheel. relevant prior work: * packaging `libcudf` wheels: rapidsai/cudf#15483 * consolidating `pip install` calls in CI scripts for `cudf`: rapidsai/cudf#16575 * `cudf` dropping its Arrow library dependency: rapidsai/cudf#16640 ### How I tested this Confirmed in local builds and CI logs that `cudf` is being *found*, not *built*, in `cuspatial` builds. ```text -- CPM: Using local package cudf@24.10.0 ``` ([build link](https://github.com/rapidsai/cuspatial/actions/runs/10602971716/job/29386288614?pr=1447#step:9:23472)) Built `cuspatial` wheels locally and ran all the unit tests, without issue. # Authors: - James Lamb (https://github.com/jameslamb) Approvers: - Bradley Dice (https://github.com/bdice) - Vyas Ramasubramani (https://github.com/vyasr) - Matthew Roeschke (https://github.com/mroeschke) URL: #1447
|
#1447 is merged and I've pulled in those changes here |
|
Ok think I understand what is happening with NumPy 1 being installed on CI now, OSMnx 1.9.4 (currently latest) pins to NumPy 1 IIUC OSMnx 2.0 will relax this constraint I don't think there is a lot we can do about that Looking at the code here, it appears the cuSpatial library doesn't actually use OSMnx. Only the notebooks do Perhaps there is a way to move that dependency out of the main CI paths so we can test with NumPy 2 here (ignoring notebooks which will have to stay on NumPy 1) Think if we can do that, we can move forward Edit: Nvm this is already only installed in the notebook job. So this is not an issue |
|
Should add another piece is This can only happen if While |
|
🎉 🎉 🎉 |
jameslamb
added
3 - Ready for Review
Co-authored-by: Bradley Dice <bdice@bradleydice.com>
jameslamb
removed
3 - Ready for Review
|
Ran CI again after removing all testing/debugging changes... still getting I think this can be merged. |
|
/merge |
|
Huzzah! 🥳 Thanks for all of the hard work here. Especially pushing through |
Contributes to rapidsai/build-planning#56 With rapidsai/cuspatial#1441, it should be possible to revert some of the workarounds introduced in #719. ## Notes for Reviewers ### How to test this if this is working, we should see the following in the conda solves: * `fmt >=11.0.2` * `spdlog >=1.14.1` We won't see `numpy >=2` yet, because `cugraph` doesn't support it yet (rapidsai/cugraph#4615). Authors: - James Lamb (https://github.com/jameslamb) Approvers: - Bradley Dice (https://github.com/bdice) URL: #722
This PR removes the NumPy<2 pin which is expected to work for
RAPIDS projects once CuPy 13.3.0 is released (CuPy 13.2.0 had
some issues preventing the use with NumPy 2).
Also contributes to rapidsai/build-planning#56 by removing the pin on
proj(which allows this project to work with newerfmt/spdlog, in sync with conda-forge).