Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

remove streamz git dependency, standardize build dependency names, consolidate some dependency lists #16611

Merged
merged 3 commits into from
Aug 20, 2024

Conversation

jameslamb
Copy link
Member

@jameslamb jameslamb commented Aug 20, 2024

Description

Proposes some additional cleanup in dependencies.yaml, for things I noticed while working through #15483.

  • standardizes the naming of keys in the files: section for build dependencies
  • removes the dependency on building streamz from latest source on GitHub
    • custreamz conda packages and wheels depend on packages for those, not this git dependency
    • if this is really needed, I don't think it belongs in the build_python_cudf set
    • the last commit to streamz was 2 years ago (https://github.com/python-streamz/streamz), this doesn't seem like a rapids-dask-dependency, try-to-always-test-against-latest, situation to me
    • I'm guessing this is left over from a time before streamz was regularly publishing wheels... it's been in dependencies.yaml since that file was first introduced here in November 2022 (Bifurcate Dependency Lists [skip-gpuci] #11674)
    • the last release, v0.6.4, was made on July 27, 2022. There have been around 20 commits to master since then (history link) ... but if custreamz really needed those, I'd expect custreamz to depend on the version built from GitHub sources. I strongly suspect that that isn't the case.
  • removes build_python_cudf and build_python_libcudf lists in dependencies.yaml, in favor of re-using the depends_on_rmm and depends_on_pylibcudf lists

Checklist

  • I am familiar with the Contributing Guidelines.
  • New or existing tests cover these changes.
  • The documentation is up to date with these changes.

@jameslamb jameslamb added 2 - In Progress Currently a work in progress improvement Improvement / enhancement to an existing function non-breaking Non-breaking change labels Aug 20, 2024
@jameslamb jameslamb changed the title WIP: remove streamz git dependency, standardize build dependency names remove streamz git dependency, standardize build dependency names Aug 20, 2024
@jameslamb jameslamb added 3 - Ready for Review Ready for review by team and removed 2 - In Progress Currently a work in progress labels Aug 20, 2024
@jameslamb jameslamb marked this pull request as ready for review August 20, 2024 15:07
@jameslamb jameslamb requested a review from a team as a code owner August 20, 2024 15:07
@jameslamb jameslamb requested a review from bdice August 20, 2024 15:07
Copy link
Contributor

@bdice bdice left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving in the current state but I have a proposal for further improvement.

- output_types: requirements
packages:
# pip recognizes the index as a global option for the requirements.txt file
# This index is needed for rmm-cu{11,12}.
- --extra-index-url=https://pypi.nvidia.com
- --extra-index-url=https://pypi.anaconda.org/rapidsai-wheels-nightly/simple
- git+https://github.com/python-streamz/streamz.git@master
specific:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we can refactor the py_rapids_build_cudf list to include lists named depends_on_rmm and depends_on_pylibcudf rather than specifying those packages here. We already have a depends_on_pylibcudf that we can use, but we should add a depends_on_rmm list. It'd help deduplicate a lot.

That would make build_python_cudf an empty dependency list, which is probably fine.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I totally agree! I'm happy to just push that here as well and use this PR for more dependencies.yaml refactoring. It'd help with #15483.

I'll do that and request another review.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok @bdice I just pushed 8bfbb95, which does the following:

  • adds a depends_on_rmm list
  • removes build_python_pylibcudf (which only held an rmm requirement)
  • removes build_python_cudf (which only held rmm and pylibcudf requirements)
  • corresponding changes in files: section (including some reordering so dependency lists are alphabetical)

@jameslamb jameslamb changed the title remove streamz git dependency, standardize build dependency names remove streamz git dependency, standardize build dependency names, consolidate some dependency lists Aug 20, 2024
@jameslamb jameslamb requested a review from bdice August 20, 2024 16:20
rapids-bot bot pushed a commit that referenced this pull request Aug 20, 2024
Removes unnecessary installation of `cudf` wheels in wheel testing for `cudf_polars`.

`cudf_polars` doesn't depend on `cudf`, and neither do its tests. However, right now it's downloading `cudf` during it's wheel tests. I mistakenly introduced that in #16575.

This introduced a race condition that could lead to CI failures whenever the `cudf` wheels aren't published yet by the time the `cudf_polars` tests. Because the `cudf_polars` wheel tests (rightly) do not wait for `cudf` wheels to be available:

https://github.com/rapidsai/cudf/blob/555734dee7a8fb10f50c8609a8e4fb2c025e6305/.github/workflows/pr.yaml#L154-L155

https://github.com/rapidsai/cudf/blob/555734dee7a8fb10f50c8609a8e4fb2c025e6305/.github/workflows/pr.yaml#L145-L146

Noticed this in #16611

```text
[rapids-download-from-s3] Downloading and decompressing s3://rapids-downloads/ci/cudf/pull-request/16611/a6b7eff/cudf_wheel_python_cudf_cu12_py310_x86_64.tar.gz into ./dist
download failed: s3://rapids-downloads/ci/cudf/pull-request/16611/a6b7eff/cudf_wheel_python_cudf_cu12_py310_x86_64.tar.gz to - An error occurred (404) when calling the HeadObject operation: Not Found
```

([build link](https://github.com/rapidsai/cudf/actions/runs/10472939821/job/29004728278?pr=16611))

Authors:
  - James Lamb (https://github.com/jameslamb)

Approvers:
  - Bradley Dice (https://github.com/bdice)

URL: #16612
@jameslamb
Copy link
Member Author

Thanks for the help @bdice ! I'm happy to see this get simplified 😊

@jameslamb
Copy link
Member Author

/merge

@rapids-bot rapids-bot bot merged commit e450baf into rapidsai:branch-24.10 Aug 20, 2024
83 checks passed
@jameslamb jameslamb deleted the more-dependencies-yaml branch August 20, 2024 19:01
@jameslamb jameslamb mentioned this pull request Aug 21, 2024
3 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
3 - Ready for Review Ready for review by team improvement Improvement / enhancement to an existing function non-breaking Non-breaking change
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants