Add support for Python 3.12 #40

bdice · 2024-04-18T19:14:14Z

Overview

Python 3.12 was released in October 2023. This issue tracks the work to add support for Python 3.12 to RAPIDS.

In #3, RAPIDS added support for Python 3.11, which was released in RAPIDS 24.04. The work to add Python 3.11 was heavily automated, and that could be done again for Python 3.12 to reduce the load on RAPIDS maintainers.

When should we drop Python 3.9?

Typically RAPIDS has kept the matrix of supported Python minor versions to 2 or 3 versions at a time. When adding Python 3.12, we should probably drop Python 3.9 as well.

SPEC 0 recommended dropping support for Python 3.9 in 2023Q4. Meanwhile, NEP 29 recommended dropping support for Python 3.9 as of Apr 05, 2024. Both of these deadlines have passed and several large Python libraries are now moving towards dropping Python 3.9, so it is probably reasonable to drop Python 3.9 around the same time that we add Python 3.12.

Tasks

Each section should be fully completed before moving to the next section.

CI images

Add Python 3.12 builds of miniforge-cuda. Drop CUDA 11.2 and 11.5, add Python 3.11 and 3.12. miniforge-cuda#55
Add Python 3.12 CI images (ci-conda, ci-wheel, citestwheel) Add Python 3.12 ci-imgs#137

CI workflows

Branch Strategy:

Create a branch on shared-workflows called python-3.12
Add Python 3.12 to the build matrix on the python-3.12 branch
Add Python 3.12 to the test matrix on the python-3.12 branch
- When adjusting the test matrix, be aware of total GPU resource consumption. Build jobs are CPU only but test jobs require GPUs. We want to keep our GPU consumption roughly the same (don't double the test matrix size), even if it gets a bit sparser in its coverage. We have some rough guidelines for how to decide on the matrix entries to include.

The above tasks are handled by: rapidsai/shared-workflows#213

RAPIDS repositories

This list is intentionally in RAPIDS dependency order... they'll need to be completed in roughly that order.

For each repo,

Update .github/workflows/*.yaml to point to the python-3.12 branch of shared-workflows
Update dependencies.yaml to add support for Python 3.12.
Review any pyproject.toml files for necessary changes (classifiers, etc.)
Update docs (README, etc) that reference a single Python version to point to the latest (3.12).
Once CI passes, merge the PR.

Most of this is easy to automate with rapids-reviser, and we can copy from this previous migrator for Python 3.11: https://github.com/rapidsai/rapids-reviser/pull/11. We still need to manually review the PRs for missing pieces.

Once all repos are migrated to the python-3.12 branch, the migration is complete. We merge python-3.12 into the development branch on shared-workflows and then open follow-up PRs to each repo to reset the branches to that development branch. This "reset" is simple and should be automated with rapids-reviser.

Notes:

The :latest image from ci-imgs is frequently used by CI jobs for building docs and testing notebooks. Be aware that issues may arise in those jobs.

The text was updated successfully, but these errors were encountered:

bdice · 2024-05-03T22:19:24Z

Strategically, it may make sense to wait until we publish RAPIDS C++ wheels (#33). That will reduce the build time impact of adding another Python version -- and we already plan to keep the total amount of testing constant. I think it may be reasonable to target early RAPIDS 24.08 for adding Python 3.12 support.

vyasr · 2024-05-06T18:44:27Z

I think that's a good idea, may as well minimize the impact. Although I should note that we were limiting our Python versions to the same degree even pre-wheels (i.e. when we only had conda builds, so half as many total artifacts to output) so I doubt that the C++ wheels effort materially changes the calculus on how many Python versions we will support at a time.

bdice · 2024-05-31T16:34:47Z

Here is a proposal for the CI matrix I would recommend for Python 3.12: rapidsai/shared-workflows#213

bdice · 2024-07-31T16:58:30Z

I have opened a separate issue to track dropping Python 3.9: #88

jakirkham · 2024-07-31T23:30:10Z

According to PEP 719, Python 3.13.0 is due out October 1st

This comes out before the RAPIDS 24.10 release, which is currently due October 10th

So it would be ideal to have Python 3.12 supported in RAPIDS by then

jameslamb · 2024-08-28T21:36:04Z

I'm working on a rapids-reviser PR to roll out most of the changes. I'm planning to modify the existing example at https://github.com/rapidsai/rapids-reviser/tree/main/examples/add-python-3.11.

I've also added task lists to the issue description with what I think is a comprehensive list of repos to be updated. Will double-check that with some GitHub searches.

Contributes to rapidsai/build-planning#40 This PR adds support for Python 3.12. ## Notes for Reviewers This is part of ongoing work to add Python 3.12 support across RAPIDS. It temporarily introduces a build/test matrix including Python 3.12, from rapidsai/shared-workflows#213. A follow-up PR will revert back to pointing at the `branch-24.10` branch of `shared-workflows` once all RAPIDS repos have added Python 3.12 support. ### This will fail until all dependencies have been updates to Python 3.12 CI here is expected to fail until all of this project's upstream dependencies support Python 3.12. This can be merged whenever all CI jobs are passing. Authors: - James Lamb (https://github.com/jameslamb) Approvers: - Bradley Dice (https://github.com/bdice) URL: #1666

Contributes to rapidsai/build-planning#40 This PR adds support for Python 3.12. ## Notes for Reviewers This is part of ongoing work to add Python 3.12 support across RAPIDS. It temporarily introduces a build/test matrix including Python 3.12, from rapidsai/shared-workflows#213. A follow-up PR will revert back to pointing at the `branch-24.10` branch of `shared-workflows` once all RAPIDS repos have added Python 3.12 support. ### This will fail until all dependencies have been updates to Python 3.12 CI here is expected to fail until all of this project's upstream dependencies support Python 3.12. This can be merged whenever all CI jobs are passing. Authors: - James Lamb (https://github.com/jameslamb) Approvers: - Bradley Dice (https://github.com/bdice) URL: #688

Contributes to rapidsai/build-planning#40 This PR adds support for Python 3.12. ## Notes for Reviewers This is part of ongoing work to add Python 3.12 support across RAPIDS. It temporarily introduces a build/test matrix including Python 3.12, from rapidsai/shared-workflows#213. A follow-up PR will revert back to pointing at the `branch-24.10` branch of `shared-workflows` once all RAPIDS repos have added Python 3.12 support. ### This will fail until all dependencies have been updates to Python 3.12 CI here is expected to fail until all of this project's upstream dependencies support Python 3.12. This can be merged whenever all CI jobs are passing. Authors: - James Lamb (https://github.com/jameslamb) Approvers: - Bradley Dice (https://github.com/bdice) URL: #214

Contributes to rapidsai/build-planning#40 This PR adds support for Python 3.12. ## Notes for Reviewers This is part of ongoing work to add Python 3.12 support across RAPIDS. It temporarily introduces a build/test matrix including Python 3.12, from rapidsai/shared-workflows#213. A follow-up PR will revert back to pointing at the `branch-24.10` branch of `shared-workflows` once all RAPIDS repos have added Python 3.12 support. ### This will fail until all dependencies have been updates to Python 3.12 CI here is expected to fail until all of this project's upstream dependencies support Python 3.12. This can be merged whenever all CI jobs are passing. Authors: - James Lamb (https://github.com/jameslamb) Approvers: - Bradley Dice (https://github.com/bdice) URL: #773

Contributes to rapidsai/build-planning#40 This PR adds support for Python 3.12. ## Notes for Reviewers This is part of ongoing work to add Python 3.12 support across RAPIDS. It temporarily introduces a build/test matrix including Python 3.12, from rapidsai/shared-workflows#213. A follow-up PR will revert back to pointing at the `branch-24.10` branch of `shared-workflows` once all RAPIDS repos have added Python 3.12 support. ### This will fail until all dependencies have been updates to Python 3.12 CI here is expected to fail until all of this project's upstream dependencies support Python 3.12. This can be merged whenever all CI jobs are passing. Authors: - James Lamb (https://github.com/jameslamb) Approvers: - Bradley Dice (https://github.com/bdice) URL: #1380

Contributes to rapidsai/build-planning#40 This PR adds support for Python 3.12. Other changes required to add that support: * updating `librdkafka` / `python-confluent-kafka`, `1.9.* -> 2.5.*` ([link to thread](#16745 (comment))) * removing use of `ast.Num` in syntax tree parsing, in favor of checking the `.value` of an `ast.Constant` against a hard-coded list of builtin types ([link to thread](https://github.com/rapidsai/cudf/pull/16745/files#r1745876846)) * ignoring deprecation warnings about `datetime.datetime.utcnow()` ([link to thread](https://github.com/rapidsai/cudf/pull/16745/files#r1746075083)) * skipping doctests that end up running `repr()` on an `OrderedDict` ([link to thread](https://github.com/rapidsai/cudf/pull/16745/files#r1746079415)) ## Notes for Reviewers This is part of ongoing work to add Python 3.12 support across RAPIDS. It temporarily introduces a build/test matrix including Python 3.12, from rapidsai/shared-workflows#213. A follow-up PR will revert back to pointing at the `branch-24.10` branch of `shared-workflows` once all RAPIDS repos have added Python 3.12 support. ### This will fail until all dependencies have been updates to Python 3.12 CI here is expected to fail until all of this project's upstream dependencies support Python 3.12. This can be merged whenever all CI jobs are passing. Authors: - James Lamb (https://github.com/jameslamb) Approvers: - Bradley Dice (https://github.com/bdice) URL: #16745

Contributes to rapidsai/build-planning#40 This PR adds support for Python 3.12. ## Notes for Reviewers This is part of ongoing work to add Python 3.12 support across RAPIDS. It temporarily introduces a build/test matrix including Python 3.12, from rapidsai/shared-workflows#213. A follow-up PR will revert back to pointing at the `branch-24.10` branch of `shared-workflows` once all RAPIDS repos have added Python 3.12 support. ### This will fail until all dependencies have been updates to Python 3.12 CI here is expected to fail until all of this project's upstream dependencies support Python 3.12. This can be merged whenever all CI jobs are passing. Authors: - James Lamb (https://github.com/jameslamb) Approvers: - Bradley Dice (https://github.com/bdice) URL: #1067

Contributes to rapidsai/build-planning#40 This PR adds support for Python 3.12. ## Notes for Reviewers This is part of ongoing work to add Python 3.12 support across RAPIDS. It temporarily introduces a build/test matrix including Python 3.12, from rapidsai/shared-workflows#213. A follow-up PR will revert back to pointing at the `branch-24.10` branch of `shared-workflows` once all RAPIDS repos have added Python 3.12 support. ### This will fail until all dependencies have been updates to Python 3.12 CI here is expected to fail until all of this project's upstream dependencies support Python 3.12. This can be merged whenever all CI jobs are passing. Authors: - James Lamb (https://github.com/jameslamb) - Peter Andreas Entschev (https://github.com/pentschev) Approvers: - Bradley Dice (https://github.com/bdice) - Peter Andreas Entschev (https://github.com/pentschev) URL: #276

…as>=1.0 (#1453) Contributes to rapidsai/build-planning#40 This PR adds support for Python 3.12. ## Notes for Reviewers This is part of ongoing work to add Python 3.12 support across RAPIDS. It temporarily introduces a build/test matrix including Python 3.12, from rapidsai/shared-workflows#213. A follow-up PR will revert back to pointing at the `branch-24.10` branch of `shared-workflows` once all RAPIDS repos have added Python 3.12 support. Other changes required to add that support: * pinning `proj` back to 9.3.x, reverting #1449 (#1453 (comment)) * requiring `geopandas >= 1.0` (#1453 (comment)) ### This will fail until all dependencies have been updates to Python 3.12 CI here is expected to fail until all of this project's upstream dependencies support Python 3.12. This can be merged whenever all CI jobs are passing. Authors: - James Lamb (https://github.com/jameslamb) - Bradley Dice (https://github.com/bdice) Approvers: - Bradley Dice (https://github.com/bdice) URL: #1453

Contributes to rapidsai/build-planning#40 This PR adds support for Python 3.12. ## Notes for Reviewers This is part of ongoing work to add Python 3.12 support across RAPIDS. It temporarily introduces a build/test matrix including Python 3.12, from rapidsai/shared-workflows#213. A follow-up PR will revert back to pointing at the `branch-24.10` branch of `shared-workflows` once all RAPIDS repos have added Python 3.12 support. ### This will fail until all dependencies have been updates to Python 3.12 CI here is expected to fail until all of this project's upstream dependencies support Python 3.12. This can be merged whenever all CI jobs are passing. Authors: - James Lamb (https://github.com/jameslamb) Approvers: - Bradley Dice (https://github.com/bdice) URL: #2428

Contributes to rapidsai/build-planning#40 This PR adds support for Python 3.12. ## Notes for Reviewers This is part of ongoing work to add Python 3.12 support across RAPIDS. It temporarily introduces a build/test matrix including Python 3.12, from rapidsai/shared-workflows#213. A follow-up PR will revert back to pointing at the `branch-24.10` branch of `shared-workflows` once all RAPIDS repos have added Python 3.12 support. ### This will fail until all dependencies have been updates to Python 3.12 CI here is expected to fail until all of this project's upstream dependencies support Python 3.12. This can be merged whenever all CI jobs are passing. Authors: - James Lamb (https://github.com/jameslamb) Approvers: - Bradley Dice (https://github.com/bdice) URL: #312

Contributes to rapidsai/build-planning#40 This PR adds support for Python 3.12. ## Notes for Reviewers This is part of ongoing work to add Python 3.12 support across RAPIDS. It temporarily introduces a build/test matrix including Python 3.12, from rapidsai/shared-workflows#213. A follow-up PR will revert back to pointing at the `branch-24.10` branch of `shared-workflows` once all RAPIDS repos have added Python 3.12 support. Additional changes needed to add this support: * dropping dependency on `pyppeteer` (https://github.com/rapidsai/cuxfilter/pull/625/files#r1750741079) ### This will fail until all dependencies have been updates to Python 3.12 CI here is expected to fail until all of this project's upstream dependencies support Python 3.12. This can be merged whenever all CI jobs are passing. Authors: - James Lamb (https://github.com/jameslamb) - Bradley Dice (https://github.com/bdice) Approvers: - Bradley Dice (https://github.com/bdice) URL: #625

Contributes to rapidsai/build-planning#40 This PR adds support for Python 3.12. ## Notes for Reviewers This is part of ongoing work to add Python 3.12 support across RAPIDS. It temporarily introduces a build/test matrix including Python 3.12, from rapidsai/shared-workflows#213. A follow-up PR will revert back to pointing at the `branch-24.10` branch of `shared-workflows` once all RAPIDS repos have added Python 3.12 support. ### This will fail until all dependencies have been updates to Python 3.12 CI here is expected to fail until all of this project's upstream dependencies support Python 3.12. This can be merged whenever all CI jobs are passing. Authors: - James Lamb (https://github.com/jameslamb) Approvers: - Bradley Dice (https://github.com/bdice) - Rick Ratzel (https://github.com/rlratzel) URL: #4647

Contributes to rapidsai/build-planning#40 This PR adds support for Python 3.12. Other changes required for this: * updating `umap-learn`, `0.5.3 -> 0.5.6` (https://github.com/rapidsai/cuml/pull/6060/files#r1745915933) ## Notes for Reviewers This is part of ongoing work to add Python 3.12 support across RAPIDS. It temporarily introduces a build/test matrix including Python 3.12, from rapidsai/shared-workflows#213. A follow-up PR will revert back to pointing at the `branch-24.10` branch of `shared-workflows` once all RAPIDS repos have added Python 3.12 support. ### This will fail until all dependencies have been updates to Python 3.12 CI here is expected to fail until all of this project's upstream dependencies support Python 3.12. This can be merged whenever all CI jobs are passing. Authors: - James Lamb (https://github.com/jameslamb) - Bradley Dice (https://github.com/bdice) Approvers: - Bradley Dice (https://github.com/bdice) URL: #6060

Contributes to rapidsai/build-planning#40 #388 pinned to Python 3.11 for building the mambaforge images here, to prevent issues in the situation where some but not all RAPIDS repos had added Python 3.12 support. This proposes moving that pin to Python 3.12. Keeping this hard-coded will prevent similar issues the next time we go to add a Python version (Python 3.13) across RAPIDS.

…711) Contributes to rapidsai/build-planning#40. * adds Python 3.12 images * defaults to latest Python (3.12) and CUDA (12.5[.1]) in docs and comments ## Notes for Reviewers Builds here will fail until all RAPIDS libraries are supporting Python 3.12, but figured we don't need to wait on that to come to an agreement about the building and testing matrices. Blocked by: * [x] rapidsai/cuml#6060 * [x] rapidsai/cugraph#4647 * [x] rapidsai/integration#719 Authors: - James Lamb (https://github.com/jameslamb) Approvers: - Bradley Dice (https://github.com/bdice) URL: #711

jameslamb · 2024-09-19T15:07:28Z

Ok folks, that's it!

The last thing is to remove Python 3.9 and add Python 3.12 in the "stable" section of the install selector at https://docs.rapids.ai/install. But that can't be done until the 24.10 release goes out. I've put up a task on the release checklist to track it, so this issue can be closed.

Thanks for the help everyone!!

vyasr mentioned this issue Apr 30, 2024

[FEA] python 3.12 support rapidsai/cuml#5840

Open

jakirkham mentioned this issue May 15, 2024

Updating Python versions rapidsai/dask-build-environment#93

Closed

bdice mentioned this issue May 31, 2024

Add Python 3.12. rapidsai/shared-workflows#213

Merged

bdice mentioned this issue Jul 31, 2024

Drop support for Python 3.9 #88

Closed

11 tasks

jakirkham added the epic label Aug 3, 2024

sisodia1701 assigned jameslamb Aug 7, 2024

sisodia1701 added this to the RAPIDS v24.10 milestone Aug 14, 2024

sisodia1701 modified the milestones: RAPIDS v24.10, v24.10 Aug 22, 2024

This was referenced Sep 5, 2024

Add support for Python 3.12 rapidsai/ucxx#276

Merged

Add support for Python 3.12. rapidsai/cugraph-gnn#41

Merged

Drop Python 3.9 support, add Python 3.12 rapidsai/cugraph-pg#25

Merged

jameslamb mentioned this issue Sep 5, 2024

Pin to Python 3.11. rapidsai/devcontainers#388

Merged

jameslamb mentioned this issue Sep 6, 2024

[BUG] test_read_region_cuda_memleak failing (GPUtil does not support Python 3.12) rapidsai/cucim#775

Closed

jameslamb mentioned this issue Sep 9, 2024

Add support for Python 3.12 rapidsai/integration#719

Merged

2 tasks

jameslamb mentioned this issue Sep 9, 2024

update 'latest' tags to Python 3.12 images rapidsai/ci-imgs#188

Merged

jameslamb mentioned this issue Sep 16, 2024

pin mambaforge images to Python 3.12 rapidsai/devcontainers#391

Merged

bdice mentioned this issue Sep 17, 2024

Update flake8 to fix false positive with Python 3.12 #101

Closed

14 tasks

jameslamb closed this as completed Sep 19, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for Python 3.12 #40

Add support for Python 3.12 #40

bdice commented Apr 18, 2024 •

edited by jameslamb

Loading

Updates for libraries

Post-migration

bdice commented May 3, 2024

vyasr commented May 6, 2024

bdice commented May 31, 2024

bdice commented Jul 31, 2024

jakirkham commented Jul 31, 2024

jameslamb commented Aug 28, 2024

jameslamb commented Sep 19, 2024

Add support for Python 3.12 #40

Add support for Python 3.12 #40

Comments

bdice commented Apr 18, 2024 • edited by jameslamb Loading

Overview

When should we drop Python 3.9?

Tasks

CI images

CI workflows

RAPIDS repositories

Updates for libraries

Post-migration

bdice commented May 3, 2024

vyasr commented May 6, 2024

bdice commented May 31, 2024

bdice commented Jul 31, 2024

jakirkham commented Jul 31, 2024

jameslamb commented Aug 28, 2024

jameslamb commented Sep 19, 2024

bdice commented Apr 18, 2024 •

edited by jameslamb

Loading