Install test dependencies at the same time as cuml packages. #5781

bdice · 2024-02-22T01:11:26Z

This PR creates a conda environment containing both test dependencies and cuml packages. This is a workaround for some issues seen with conda being unable to downgrade from Arrow 15 (in the initial environment with test dependencies) to Arrow 14 (currently pinned by cudf, which is a dependency of cuml).

This is a partial solution for rapidsai/build-planning#22. (More work is needed for other RAPIDS repos.)

Depends on rapidsai/dependency-file-generator#67.

dantegd

I really like the approach even though it'd be nice in the future if RDFG could handle this automatically

dantegd · 2024-02-22T01:15:58Z

ci/test_cpp.sh

+EOF
+
+# TODO: install conda-merge in the CI images
+pip install conda-merge


Small question: why use pip for installing conda-merge?

It's fastest to do it this way for the prototype. I am installing this into the conda environment in rapidsai/ci-imgs#112.

We may also want to add conda-merge to the base image if we find it useful. Could imagine this pattern showing up in other projects' CI

bdice · 2024-02-22T01:31:01Z

I really like the approach even though it'd be nice in the future if RDFG could handle this automatically

I'm still thinking about ways to improve this. Just wanted to get a prototype up first. What do you think about something like this, with CLI arguments for --extra-channels and --extra-packages? Other ideas welcome!

rapids-logger "Downloading artifacts from previous jobs"
CPP_CHANNEL=$(rapids-download-conda-from-s3 cpp)
PYTHON_CHANNEL=$(rapids-download-conda-from-s3 python)

rapids-dependency-file-generator \
  --output conda \
  --file_key test_python \
  --matrix "cuda=${RAPIDS_CUDA_VERSION%.*};arch=$(arch);py=${RAPIDS_PY_VERSION}" \
  --extra-channels "${CPP_CHANNEL};${PYTHON_CHANNEL}" \
  --extra-packages "libcuml cuml" | tee env.yaml

rapids-mamba-retry env create --force -f env.yaml -n test

bdice · 2024-02-22T01:22:14Z

ci/test_cpp.sh

+  - libcuml-tests
+EOF
+
+# TODO: install conda-merge in the CI images


I created a PR to the CI images for this: rapidsai/ci-imgs#112 but that would be unnecessary if we adopt this alternative approach: #5781 (comment)

bdice · 2024-02-22T01:24:36Z

ci/test_cpp.sh

+EOF
+
+# TODO: install conda-merge in the CI images
+pip install conda-merge


It's fastest to do it this way for the prototype. I am installing this into the conda environment in rapidsai/ci-imgs#112.

bdice · 2024-02-22T01:41:51Z

@dantegd If you prefer the rapids-dependency-file-generator approach proposed above, I will work on that. If this fixes CI, we might just merge it as-is and clean it up after the necessary features are added to rapids-dependency-file-generator.

charlesbluca · 2024-02-22T02:44:23Z

ci/test_cpp.sh

+  - libcuml
+  - libcuml-tests


Assuming that conda-merge will place CPP_CHANNEL at a sufficient priority so that we grab build artifacts, but do we lose anything also constraining the packages to that channel?

Suggested change

- libcuml

- libcuml-tests

- ${CPP_CHANNEL}::libcuml

- ${CPP_CHANNEL}::libcuml-tests

This is a good question. I designed rapidsai/dependency-file-generator#67 in such a way that the CI artifact channels will always be prepended. It's still possible that another channel could be used, but there's not a good way to inject the channel names corresponding to those packages if we go with the route of rapidsai/dependency-file-generator#67.

dantegd · 2024-02-22T03:07:58Z

@bdice I think the rapids-dependency-file-generator is nicer for doing it in general in all RAPIDS repos, but also very happy to merge this as is since it seems to fix CI. Thanks!

bdice · 2024-02-22T03:32:25Z

I put up a PR with a feature --prepend-channels for rapids-dependency-file-generator: rapidsai/dependency-file-generator#67

Some dask test failed, seems like a network issue that a rerun will fix. Then let's merge this to unblock other PRs. I'll work on a follow-up for cuml (and the rest of RAPIDS) after we get rapidsai/dependency-file-generator#67 merged.

charlesbluca

Looks like we end up pulling stable packages instead of from the build channels?

https://github.com/rapidsai/cuml/actions/runs/7997932570/job/21846716467?pr=5781

We might need to tighten the pinnings on the added packages - is it reasonable to pin these to the current nightly version and update the release scripts to bump this version?

charlesbluca · 2024-02-22T13:58:33Z

ci/test_cpp.sh

+
+# TODO: install conda-merge in the CI images
+pip install conda-merge
+conda-merge test.yaml cuml.yaml > env.yaml


Could we cat the output of this file to see the placement of the new channels in the resulting environment file?

Just checked this locally, looks like the order isn't super defined when we don't have a common channel between the merged environments:

channels: - rapidsai - /tmp/cpp_channel - rapidsai-nightly - /tmp/python_channel - dask/label/dev - conda-forge - nvidia

Think that putting rapidsai below the artifact channels in the new environment file should be sufficient to get the channel ordering we desire (at least for this PR, think the approach in rapidsai/dependency-file-generator#67 should improve things)

Ah, thanks for checking that. I agree the approach with prepend-channels will solve this.

Not sure about the CPP test failure, but looks like we're grabbing from the correct channel now 🎉

https://github.com/rapidsai/cuml/actions/runs/8005972527/job/21867897863?pr=5781

ci/test_cpp.sh

ci/test_notebooks.sh

ci/test_python_common.sh

…hannels. Co-authored-by: Charles Blackmon-Luca <20627856+charlesbluca@users.noreply.github.com>

charlesbluca

Thanks for this work @bdice; is there anything else that needs to happen now that rapidsai/shared-workflows#178 is in for these changes to be reflected?

bdice · 2024-02-22T22:56:22Z

Thanks for this work @bdice; is there anything else that needs to happen now that rapidsai/shared-workflows#178 is in for these changes to be reflected?

😢 I made an empty commit earlier to rerun CI but didn't push it. I came back to this PR hoping it would be ready to merge by now. At this point, the CI images with the "proper" solution of rapids-dependency-file-generator are nearly published (status). I will refactor this PR to use the proper long-term solution since it will be only a few more minutes of work relative to the long CI testing time.

bdice · 2024-02-23T00:12:30Z

I updated the implementation of this PR to use --prepend-channels. I'll check the CI results in a bit, but I think this is a lot cleaner than the previous implementation.

vyasr

LGTM, I agree this is much cleaner. Thanks!

jakirkham · 2024-02-26T17:55:15Z

dependencies.yaml

+  test_libcuml:
+    common:
+      - output_types: conda
+        packages:
+          - libcuml==24.4.*
+          - libcuml-tests==24.4.*
+  test_cuml:
+    common:
+      - output_types: conda
+        packages:
+          - libcuml==24.4.*
+          - cuml==24.4.*


Would we want this for wheels too?

No, it's not needed. This dependency list is used exclusively for conda CI, to generate the environment with the PR artifacts and test dependencies in one solve. Wheels go through a different installation path and I'm not sure if it's possible to consolidate that environment generation into one step or not. Even if it is possible, it's unlikely that listing cuml-cu12 in the requirements file would be sufficient, since we need to install a specific filename for the wheel that was fetched from CI artifacts.

bdice · 2024-02-26T18:38:04Z

/merge

jakirkham · 2024-02-26T18:42:14Z

/merge

This is a port of rapidsai/cuml#5781 and part of rapidsai/build-planning#22 Authors: - Mike Sarahan (https://github.com/msarahan) - Paul Taylor (https://github.com/trxcllnt) Approvers: - Paul Taylor (https://github.com/trxcllnt) - https://github.com/jakirkham - Ray Douglass (https://github.com/raydouglass) URL: #1387

Contributes to rapidsai/build-planning#106 Proposes specifying the RAPIDS version in `conda install` calls that install CI artifacts, to reduce the risk of CI jobs picking up artifacts from other releases. ## Notes for Reviewers This only changes the docs build, because other packages already solve in a single `conda install` with version constraints, thanks to #5781. Authors: - James Lamb (https://github.com/jameslamb) Approvers: - Bradley Dice (https://github.com/bdice) - Dante Gama Dessavre (https://github.com/dantegd) URL: #6103

Install test dependencies at the same time as cuml packages.

1770700

github-actions bot added the ci label Feb 22, 2024

bdice added improvement Improvement / enhancement to an existing function non-breaking Non-breaking change ci and removed ci labels Feb 22, 2024

bdice self-assigned this Feb 22, 2024

bdice mentioned this pull request Feb 22, 2024

Pin pyarrow #5780

Closed

dantegd reviewed Feb 22, 2024

View reviewed changes

bdice mentioned this pull request Feb 22, 2024

Add conda-merge to ci-conda images. rapidsai/ci-imgs#112

Closed

bdice commented Feb 22, 2024

View reviewed changes

charlesbluca reviewed Feb 22, 2024

View reviewed changes

bdice mentioned this pull request Feb 22, 2024

feat: Add support for prepending conda channels via the CLI. rapidsai/dependency-file-generator#67

Merged

bdice marked this pull request as ready for review February 22, 2024 03:32

bdice requested a review from a team as a code owner February 22, 2024 03:32

charlesbluca reviewed Feb 22, 2024

View reviewed changes

charlesbluca suggested changes Feb 22, 2024

View reviewed changes

ci/test_cpp.sh Outdated Show resolved Hide resolved

ci/test_notebooks.sh Outdated Show resolved Hide resolved

ci/test_python_common.sh Outdated Show resolved Hide resolved

Add rapidsai channel to force DAG sort to prioritize local artifact c…

44dcfc6

…hannels. Co-authored-by: Charles Blackmon-Luca <20627856+charlesbluca@users.noreply.github.com>

charlesbluca approved these changes Feb 22, 2024

View reviewed changes

galipremsagar approved these changes Feb 22, 2024

View reviewed changes

Use rapids-dependency-file-generator with --prepend-channels.

3a3e1a8

vyasr approved these changes Feb 23, 2024

View reviewed changes

Add test_cuml to test_notebooks.

38ac4a1

vyasr mentioned this pull request Feb 26, 2024

Consider consolidating conda solves in CI tests rapidsai/build-planning#22

Open

18 tasks

jakirkham reviewed Feb 26, 2024

View reviewed changes

AyodeAwe approved these changes Feb 26, 2024

View reviewed changes

jakirkham approved these changes Feb 26, 2024

View reviewed changes

dantegd approved these changes Feb 26, 2024

View reviewed changes

rapids-bot bot merged commit 5661d02 into rapidsai:branch-24.04 Feb 26, 2024
55 checks passed

vyasr mentioned this pull request Mar 1, 2024

Test CI changes rapidsai/cudf#15201

Closed

3 tasks

msarahan mentioned this pull request May 22, 2024

create conda ci test env in one step rapidsai/cuspatial#1387

Merged

3 tasks

jameslamb mentioned this pull request Oct 8, 2024

make conda installs in CI stricter #6103

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Install test dependencies at the same time as cuml packages. #5781

Install test dependencies at the same time as cuml packages. #5781

bdice commented Feb 22, 2024 •

edited

Loading

dantegd left a comment

dantegd Feb 22, 2024

bdice Feb 22, 2024

jakirkham Feb 22, 2024

bdice commented Feb 22, 2024

bdice Feb 22, 2024

bdice Feb 22, 2024

bdice commented Feb 22, 2024 •

edited

Loading

charlesbluca Feb 22, 2024

bdice Feb 22, 2024

dantegd commented Feb 22, 2024

bdice commented Feb 22, 2024

charlesbluca left a comment

charlesbluca Feb 22, 2024

charlesbluca Feb 22, 2024

bdice Feb 22, 2024

charlesbluca Feb 22, 2024

charlesbluca left a comment

bdice commented Feb 22, 2024 •

edited

Loading

bdice commented Feb 23, 2024

vyasr left a comment

jakirkham Feb 26, 2024

bdice Feb 26, 2024 •

edited

Loading

bdice commented Feb 26, 2024

jakirkham commented Feb 26, 2024

-  - libcuml
-  - libcuml-tests
+  - ${CPP_CHANNEL}::libcuml
+  - ${CPP_CHANNEL}::libcuml-tests

Install test dependencies at the same time as cuml packages. #5781

Install test dependencies at the same time as cuml packages. #5781

Conversation

bdice commented Feb 22, 2024 • edited Loading

dantegd left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bdice commented Feb 22, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bdice commented Feb 22, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dantegd commented Feb 22, 2024

bdice commented Feb 22, 2024

charlesbluca left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

charlesbluca left a comment

Choose a reason for hiding this comment

bdice commented Feb 22, 2024 • edited Loading

bdice commented Feb 23, 2024

vyasr left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bdice Feb 26, 2024 • edited Loading

Choose a reason for hiding this comment

bdice commented Feb 26, 2024

jakirkham commented Feb 26, 2024

bdice commented Feb 22, 2024 •

edited

Loading

bdice commented Feb 22, 2024 •

edited

Loading

bdice commented Feb 22, 2024 •

edited

Loading

bdice Feb 26, 2024 •

edited

Loading