Drop support for Python 3.9 #88

bdice · 2024-07-31T16:57:30Z

This issue tracks dropping Python 3.9, following from the discussion on #40:

When should we drop Python 3.9?

Typically RAPIDS has kept the matrix of supported Python minor versions to 2 or 3 versions at a time. When adding Python 3.12, we should probably drop Python 3.9 as well.

SPEC 0 recommended dropping support for Python 3.9 in 2023Q4. Meanwhile, NEP 29 recommended dropping support for Python 3.9 as of Apr 05, 2024. Both of these deadlines have passed and several large Python libraries are now moving towards dropping Python 3.9, so it is probably reasonable to drop Python 3.9 around the same time that we add Python 3.12.

We are beyond the SPEC 0 date for dropping Python 3.9 now, and expect that RAPIDS dependencies will gradually drop support. We are exploring dropping Python 3.9 in RAPIDS 24.10.

Tasks:

Things to be careful about:

Python version embedded in Jupyter notebook metadata (like this)
avoid changing benchmarking results and the code to reproduce them
pynvjitlink is used by more than just RAPIDS (like https://github.com/NVIDIA/nvmath-python), and uses its own CI matrix to support a broader range of versions than RAPIDS (link)... we may sometimes drop a Python version across RAPIDS but still support it in pynvjitlink
generic tools like rapids-reviser, rapids-build-backend, rapids-metadata, etc. shouldn't have their floors bumped unless there's a compelling reason to do so... that'd make it harder to fix bugs in those tools that affect older releases

before making changes in individual repos

Give feedback

Announce deprecation: Dropping Support for Python v3.9 and adding Python 3.12 Support in Release v24.10 docs#528
Drop Python 3.9 from shared-workflows: Drop Python 3.9 from workflows. shared-workflows#235
Options

after all libraries are updated

Give feedback

pypi-wheel-scripts (https://github.com/rapidsai/pypi-wheel-scripts/pull/16)
ci-imgs (Drop Python 3.9, clean up CentOS references, update latest.yaml ci-imgs#173)
dask-build-environment (Drop python 3.9 builds dask-build-environment#97)
update install docs? (https://github.com/rapidsai/docs/blob/01dbbcb67c199893b839651f5dca01356c825068/install/install.md?plain=1#L101) (install selector: Drop python 3.9 for nightlies, default to CUDA 12.5 for stable and nightly docs#534)
Update release selector: need to disable Python 3.9 on nightlies https://github.com/rapidsai/docs/blob/01dbbcb67c199893b839651f5dca01356c825068/_includes/selector.html (install selector: Drop python 3.9 for nightlies, default to CUDA 12.5 for stable and nightly docs#534)
Options

The text was updated successfully, but these errors were encountered:

jakirkham · 2024-07-31T23:23:35Z

Some quick data points:

NumPy 2.1.0 drops Python 3.9 (note 2.0.0 has 3.9 support)
- ENH: Drop support for Python 3.9? numpy/numpy#26242
Pandas next release drops Python 3.9 (cites SPEC 0)
- DEPS: Drop Python 3.9 pandas-dev/pandas#58238
CuPy 14 plans to drop Python 3.9 (cites SPEC 0)
- CuPy v14 Release Plan cupy/cupy#8269
Numba still supports Python 3.9
scikit-learn still supports Python 3.9
scikit-image still supports Python 3.9
NetworkX 3.3 dropped Python 3.9 (came out Oct 2023)
- Drop Python 3.9 support networkx/networkx#7028
Bokeh 3.5 dropped Python 3.9
- Drop support for Python 3.9 and modernize the codebase bokeh/bokeh#13634
Dask is discussing dropping Python 3.9
- End of Python3.9 support dask/community#373
XGBoost next release drops Python 3.9
- [breaking] Bump Python requirement to 3.10. dmlc/xgboost#10434

jameslamb · 2024-08-02T21:39:28Z

I'm +1 on doing this in this timeframe.

DLFW builds are using Python 3.10 (DLFW support matrix), as that's the stable python3 that ships with Ubuntu 22.04 (Python version across Linux distros), so this change shouldn't be disruptive for those builds.

And nemo-curator is on python >= 3.10 (code link), so they shouldn't be affected either.

rjzamora · 2024-08-12T19:59:29Z

We are exploring dropping Python 3.9 in RAPIDS 24.10.

Just curious to know what the ETA is for dropping Python 3.9?

I believe we will need to remain pinned to dask 2024.7.1 until then. In the meantime, we cannot catch breaking changes in dask/distributed.

jameslamb · 2024-08-12T23:16:39Z

The plan is to do this "within 24.10", but we can't be more precise than that.

I believe we will need to remain pinned to dask 2024.7.1 until then. In the meantime, we cannot catch breaking changes in dask/distributed.

For Python 3.9, yes, but I think rapids-dask-dependency could take on a dependency on newer dask / distributed for Python 3.10+

Like this:

# TODO: drop the python_version specifiers and just use latest versions built from GitHub once
#              RAPIDS drops Python 3.9 support
dependencies = [
    "dask @ git+https://github.com/dask/dask.git@main ; python_version >= '3.10'",
    "distributed @ git+https://github.com/dask/distributed.git@main ; python_version >= '3.10'",
    "dask-expr @ git+https://github.com/dask/dask-expr.git@main ; python_version >= '3.10'",
    "dask==2024.7.1 ; python_version < '3.10'",
    "distributed==2024.7.1 ; python_version < '3.10'",
    "dask-expr ; python_version < '3.10'",
]

That'd allow projects to start testing against the latest development version of Dask throughout the 24.10 release cycle.

rjzamora · 2024-08-12T23:27:44Z

That'd allow projects to start testing against the latest development version of Dask throughout the 24.10 release cycle.

Okay, thanks for the response @jameslamb! This may be our best option -- I'll look into it.

jakirkham · 2024-08-12T23:32:16Z

For rapids-dask-dependency, likely Conda will work ok as-is since the solver will pick packages that are Python 3.9 (or 3.10+) compatible. Though please let us know if not

bdice · 2024-08-12T23:49:32Z

Dropping 3.9 builds should now be safe, and would also unblock the work of updating our dask pinning.

This PR does that: rapidsai/shared-workflows#235

This change in shared-workflows is also safe to merge before we do the cleanup in each repo to remove dependencies, configs, etc. referencing Python 3.9. We're just building/testing fewer packages.

This PR removes hardcoded Python versions from CI workflows. It is a prerequisite for dropping Python 3.9. See rapidsai/build-planning#88. Authors: - Bradley Dice (https://github.com/bdice) Approvers: - James Lamb (https://github.com/jameslamb) URL: #16540

Unpins dask and distributed for 24.10 development. ~**Blocked by rapidsai/build-planning#88 (Dask no longer supports Python 3.9. Therefore, unpinning too early is a "breaking" change).~ Other Blockers: - [x] rapidsai/cudf#16535 - [x] dask/dask-expr#1122 Authors: - Richard (Rick) Zamora (https://github.com/rjzamora) Approvers: - GALI PREM SAGAR (https://github.com/galipremsagar) URL: #58

jakirkham · 2024-08-13T21:52:19Z

Closing the loop on Dask, we dropped Python 3.9 from CI, so no special casing was needed (or used) there

Just a normal unpinning of Dask as part of RAPIDS post-release cleanup

Contributes to rapidsai/build-planning#88 Finishes the work of dropping Python 3.9 support. This project stopped building / testing against Python 3.9 as of rapidsai/shared-workflows#235. This PR updates configuration and docs to reflect that. ## Notes for Reviewers ### How I tested this Checked that there were no remaining uses like this: ```shell git grep -E '3\.9' git grep '39' git grep 'py39' ``` And similar for variations on Python 3.8 (to catch things that were missed the last time this was done). Authors: - James Lamb (https://github.com/jameslamb) Approvers: - https://github.com/jakirkham - Bradley Dice (https://github.com/bdice) URL: #272

Contributes to rapidsai/build-planning#88 Finishes the work of dropping Python 3.9 support. This project stopped building / testing against Python 3.9 as of rapidsai/shared-workflows#235. This PR updates configuration and docs to reflect that. ## Notes for Reviewers ### How I tested this Checked that there were no remaining uses like this: ```shell git grep -E '3\.9' git grep '39' git grep 'py39' ``` And similar for variations on Python 3.8 (to catch things that were missed the last time this was done). Authors: - James Lamb (https://github.com/jameslamb) Approvers: - https://github.com/jakirkham URL: #1659

Contributes to rapidsai/build-planning#88 Finishes the work of dropping Python 3.9 support. This project stopped building / testing against Python 3.9 as of rapidsai/shared-workflows#235. This PR updates configuration and docs to reflect that. ## Notes for Reviewers ### How I tested this Checked that there were no remaining uses like this: ```shell git grep -E '3\.9' git grep '39' git grep 'py39' ``` And similar for variations on Python 3.8 (to catch things that were missed the last time this was done). Authors: - James Lamb (https://github.com/jameslamb) Approvers: - Bradley Dice (https://github.com/bdice) URL: #684

Contributes to rapidsai/build-planning#88 Finishes the work of dropping Python 3.9 support. This project stopped building / testing against Python 3.9 as of rapidsai/shared-workflows#235. This PR updates configuration and docs to reflect that. ## Notes for Reviewers ### How I tested this Checked that there were no remaining uses like this: ```shell git grep -E '3\.9' git grep '39' git grep 'py39' ``` And similar for variations on Python 3.8 (to catch things that were missed the last time this was done). Authors: - James Lamb (https://github.com/jameslamb) Approvers: - Bradley Dice (https://github.com/bdice) URL: #623

Contributes to rapidsai/build-planning#88 Finishes the work of dropping Python 3.9 support. This project stopped building / testing against Python 3.9 as of rapidsai/shared-workflows#235. This PR updates configuration and docs to reflect that. ## Notes for Reviewers ### How I tested this Checked that there were no remaining uses like this: ```shell git grep -E '3\.9' git grep '39' git grep 'py39' ``` And similar for variations on Python 3.8 (to catch things that were missed the last time this was done). Authors: - James Lamb (https://github.com/jameslamb) Approvers: - https://github.com/jakirkham URL: #1377

Contributes to rapidsai/build-planning#88 Finishes the work of dropping Python 3.9 support. This project stopped building / testing against Python 3.9 as of rapidsai/shared-workflows#235. This PR updates configuration and docs to reflect that. ## Notes for Reviewers ### How I tested this Checked that there were no remaining uses like this: ```shell git grep -E '3\.9' git grep '39' git grep 'py39' ``` And similar for variations on Python 3.8 (to catch things that were missed the last time this was done). Authors: - James Lamb (https://github.com/jameslamb) Approvers: - https://github.com/jakirkham URL: #443

Contributes to rapidsai/build-planning#88 Finishes the work of dropping Python 3.9 support. This project stopped building / testing against Python 3.9 as of rapidsai/shared-workflows#235. This PR updates configuration and docs to reflect that. ## Notes for Reviewers ### How I tested this Checked that there were no remaining uses like this: ```shell git grep -E '3\.9' git grep '39' git grep 'py39' ``` And similar for variations on Python 3.8 (to catch things that were missed the last time this was done). Authors: - James Lamb (https://github.com/jameslamb) Approvers: - https://github.com/jakirkham URL: #1065

Contributes to rapidsai/build-planning#88 Finishes the work of dropping Python 3.9 support. This project stopped building / testing against Python 3.9 as of rapidsai/shared-workflows#235. This PR updates configuration and docs to reflect that. ## Notes for Reviewers ### How I tested this Checked that there were no remaining uses like this: ```shell git grep -E '3\.9' git grep '39' git grep 'py39' ``` And similar for variations on Python 3.8 (to catch things that were missed the last time this was done). Authors: - James Lamb (https://github.com/jameslamb) Approvers: - https://github.com/jakirkham URL: #301

Contributes to rapidsai/build-planning#88 Finishes the work of dropping Python 3.9 support. This project stopped building / testing against Python 3.9 as of rapidsai/shared-workflows#235. This PR updates configuration and docs to reflect that. ## Notes for Reviewers ### How I tested this Checked that there were no remaining uses like this: ```shell git grep -E '3\.9' git grep '39' git grep 'py39' ``` And similar for variations on Python 3.8 (to catch things that were missed the last time this was done). Authors: - James Lamb (https://github.com/jameslamb) Approvers: - Mark Harris (https://github.com/harrism) - https://github.com/jakirkham URL: #1443

jakirkham · 2024-08-23T02:52:13Z

Thanks James! 🙏

Have shepherded a good chunk of these through

Of the remainder:

RAFT - Had a (flaky?) test failure. Trying rerunning to see if it clears. Fixed on restart. Now merged!
cuGraph-GNN & Wholegraph - Have active discussions (though seems like they are close to resolution)
Draft PRs - A few are marked as draft. Maybe we can take them out of draft?
Others - Did not get to some or saw others were already looking

Please let me know if there are others you need help following up on

Contributes to rapidsai/build-planning#88 Finishes the work of dropping Python 3.9 support. This project stopped building / testing against Python 3.9 as of rapidsai/shared-workflows#235. This PR updates configuration and docs to reflect that. ## Notes for Reviewers ### How I tested this Checked that there were no remaining uses like this: ```shell git grep -E '3\.9' git grep '39' git grep 'py39' ``` And similar for variations on Python 3.8 (to catch things that were missed the last time this was done). Authors: - James Lamb (https://github.com/jameslamb) Approvers: - https://github.com/jakirkham URL: #2417

Contributes to rapidsai/build-planning#88 Finishes the work of dropping Python 3.9 support. This project stopped building / testing against Python 3.9 as of rapidsai/shared-workflows#235. This PR updates configuration and docs to reflect that. ## Notes for Reviewers ### How I tested this Checked that there were no remaining uses like this: ```shell git grep -E '3\.9' git grep '39' git grep 'py39' ``` And similar for variations on Python 3.8 (to catch things that were missed the last time this was done). Authors: - James Lamb (https://github.com/jameslamb) Approvers: - Bradley Dice (https://github.com/bdice) URL: #6040

jameslamb · 2024-08-23T13:36:44Z

Thanks John!

A few are marked as draft. Maybe we can take them out of draft?

Yep, my approach has been to open these PRs in draft first, to not bother people with notifications for review until I see CI (or at least, most CI) passing.

Happy to see all of those that were still in draft as of yesterday are looking ok, so I've opened them all up for review.

Contributes to rapidsai/build-planning#88 Finishes the work of dropping Python 3.9 support. This project stopped building / testing against Python 3.9 as of rapidsai/shared-workflows#235. This PR updates configuration and docs to reflect that. ## Notes for Reviewers ### How I tested this Checked that there were no remaining uses like this: ```shell git grep -E '3\.9' git grep '39' git grep 'py39' ``` And similar for variations on Python 3.8 (to catch things that were missed the last time this was done). Authors: - James Lamb (https://github.com/jameslamb) Approvers: - Bradley Dice (https://github.com/bdice) URL: #209

@leofang

Contributes to rapidsai/build-planning#88. Drops Python 3.9 support. ## Notes for Reviewers ### Is this safe for `nvmath-python`? I think yes. In an offline conversation, I was told that `nvmath-python` follows NEP 29 (https://numpy.org/neps/nep-0029-deprecation_policy.html) and so would want to drop Python 3.9 in its next release anyway. That's documented in its docs as well: https://github.com/NVIDIA/nvmath-python/blob/7c485842d0f3300e03ec780056936503913910fe/docs/sphinx/overview.rst?plain=1#L274 cc @leofang ### How I tested this Checked that there were no remaining uses like this: ```shell git grep -E '3\.9' git grep '39' git grep 'py39' ``` And similar for variations on Python 3.8 (to catch things that were missed the last time this was done). ### What are these other changes? `ruff` raised two new errors here: ```text .ycm_extra_conf.py:24:5: N802 Function name `Settings` should be lowercase pynvjitlink/patch.py:226:20: UP031 Use format specifiers instead of percent format Found 2 errors. No fixes available (1 hidden fix can be enabled with the `--unsafe-fixes` option). ``` Left `Settings` untouched... looks like `ycm` expects that to be exactly `Settings` ([ycm-core docs](https://github.com/ycm-core/YouCompleteMe/blob/0d855962a60bd993980d287a3ac97f57ecd02ea6/README.md?plain=1#L1431)). The `verify-copyright` hook also complained about one file missing a copyright header ```text In file .ycm_extra_conf.py:1:1: import os warning: no copyright notice found ``` So I added one. Authors: - James Lamb (https://github.com/jameslamb) Approvers: - Bradley Dice (https://github.com/bdice) - Leo Fang (https://github.com/leofang) - https://github.com/jakirkham URL: #102

Contributes to rapidsai/build-planning#88 Finishes the work of dropping Python 3.9 support. This project stopped building / testing against Python 3.9 as of rapidsai/shared-workflows#235. This PR updates configuration and docs to reflect that. ## Notes for Reviewers ### How I tested this Checked that there were no remaining uses like this: ```shell git grep -E '3\.9' git grep '39' git grep 'py39' ``` And similar for variations on Python 3.8 (to catch things that were missed the last time this was done). Authors: - James Lamb (https://github.com/jameslamb) Approvers: - Kyle Edwards (https://github.com/KyleFromNVIDIA) - Bradley Dice (https://github.com/bdice) - Brad Rees (https://github.com/BradReesWork) URL: #4625

Also proposes requiring Python 3.10, since rapidsai/build-planning#88 is being rolled out. Fixes #48

Contributes to rapidsai/build-planning#88 Finishes the work of dropping Python 3.9 support. This project stopped building / testing against Python 3.9 as of rapidsai/shared-workflows#235. This PR updates configuration and docs to reflect that. ## Notes for Reviewers ### How I tested this Checked that there were no remaining uses like this: ```shell git grep -E '3\.9' git grep '39' git grep 'py39' ``` And similar for variations on Python 3.8 (to catch things that were missed the last time this was done). Authors: - James Lamb (https://github.com/jameslamb) Approvers: - Bradley Dice (https://github.com/bdice) - Lawrence Mitchell (https://github.com/wence-) URL: #16637

sisodia1701 · 2024-08-27T16:34:20Z

Discussed in today's build infra sync, this is almost done. Once complete, the team will start working on Add support for Python 3.12 #40

jameslamb · 2024-08-28T16:00:05Z

Did some more GitHub searches, found a few more things that were missed.

All that's left here is those PRs and dask-build-environment: rapidsai/dask-build-environment#97.

This is close enough to done that we can move forward with Python 3.12 (#40)

jameslamb · 2024-08-30T15:05:51Z

Did another set of GitHub searches, and I think this work is now complete! Thanks for the help everyone 😁

jakirkham · 2024-08-30T19:04:50Z

Thanks James! 🙏

bdice mentioned this issue Jul 31, 2024

Add support for Python 3.12 #40

Closed

5 tasks

jakirkham added the epic label Aug 3, 2024

sisodia1701 assigned jameslamb Aug 7, 2024

charlesbluca mentioned this issue Aug 9, 2024

Drop python 3.9 builds rapidsai/dask-build-environment#97

Merged

rjzamora mentioned this issue Aug 12, 2024

Avoid calling array attribute on cudf.Series dask/dask-expr#1122

Merged

rjzamora mentioned this issue Aug 12, 2024

Unpin dask and distributed for 24.10 development rapidsai/rapids-dask-dependency#58

Merged

2 tasks

bdice mentioned this issue Aug 12, 2024

Drop Python 3.9 from workflows. rapidsai/shared-workflows#235

Merged

bdice mentioned this issue Aug 13, 2024

Remove hardcoded versions from workflows. rapidsai/cudf#16540

Merged

3 tasks

This was referenced Aug 22, 2024

Drop Python 3.9 support rapidsai/cuvs#301

Merged

Drop Python 3.9 support rapidsai/cuxfilter#623

Merged

Drop Python 3.9 support rapidsai/dask-cuda#1377

Merged

Drop Python 3.9 support rapidsai/kvikio#443

Merged

This was referenced Aug 23, 2024

Drop Python 3.9 support rapidsai/pynvjitlink#102

Merged

Drop Python 3.9 support, add Python 3.12 rapidsai/cugraph-pg#25

Merged

KyleFromNVIDIA mentioned this issue Aug 26, 2024

Display notice when copyrighted file is copied or renamed rapidsai/pre-commit-hooks#52

Merged

mmccarty added this to the v24.10 milestone Aug 27, 2024

KyleFromNVIDIA added a commit to rapidsai/pre-commit-hooks that referenced this issue Aug 27, 2024

Display notice when copyrighted file is copied or renamed (#52)

80e3e1b

Also proposes requiring Python 3.10, since rapidsai/build-planning#88 is being rolled out. Fixes #48

jameslamb closed this as completed Aug 30, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Drop support for Python 3.9 #88

Drop support for Python 3.9 #88

bdice commented Jul 31, 2024 •

edited by jameslamb

Loading

When should we drop Python 3.9?

before making changes in individual repos

Updates for libraries

after all libraries are updated

jakirkham commented Jul 31, 2024

jameslamb commented Aug 2, 2024 •

edited

Loading

rjzamora commented Aug 12, 2024

jameslamb commented Aug 12, 2024 •

edited

Loading

rjzamora commented Aug 12, 2024

jakirkham commented Aug 12, 2024

bdice commented Aug 12, 2024

jakirkham commented Aug 13, 2024

jakirkham commented Aug 23, 2024 •

edited

Loading

jameslamb commented Aug 23, 2024

sisodia1701 commented Aug 27, 2024

jameslamb commented Aug 28, 2024 •

edited

Loading

jameslamb commented Aug 30, 2024

jakirkham commented Aug 30, 2024

Drop support for Python 3.9 #88

Drop support for Python 3.9 #88

Comments

bdice commented Jul 31, 2024 • edited by jameslamb Loading

When should we drop Python 3.9?

before making changes in individual repos

Updates for libraries

after all libraries are updated

jakirkham commented Jul 31, 2024

jameslamb commented Aug 2, 2024 • edited Loading

rjzamora commented Aug 12, 2024

jameslamb commented Aug 12, 2024 • edited Loading

rjzamora commented Aug 12, 2024

jakirkham commented Aug 12, 2024

bdice commented Aug 12, 2024

jakirkham commented Aug 13, 2024

jakirkham commented Aug 23, 2024 • edited Loading

jameslamb commented Aug 23, 2024

sisodia1701 commented Aug 27, 2024

jameslamb commented Aug 28, 2024 • edited Loading

jameslamb commented Aug 30, 2024

jakirkham commented Aug 30, 2024

bdice commented Jul 31, 2024 •

edited by jameslamb

Loading

jameslamb commented Aug 2, 2024 •

edited

Loading

jameslamb commented Aug 12, 2024 •

edited

Loading

jakirkham commented Aug 23, 2024 •

edited

Loading

jameslamb commented Aug 28, 2024 •

edited

Loading