Add Python 3.12.

[bdice]

Contributes to rapidsai/build-planning#40.

This PR adds Python 3.12 entries to conda/wheel build and test jobs.

To keep the total number of GPU test jobs fixed, I modified the Python versions used by some existing jobs.

See previous migration to Python 3.11: #176

[bdice]

I recommend we start rolling out Python 3.12 to RAPIDS repositories after these items:

• rapids-build-backend: Update RAPIDS Python packages to use rapids-build-backend build-planning#31
• C++ wheels, at least for most of RAPIDS: Support dynamic linking between RAPIDS wheels build-planning#33

[vyasr]

It's worth noting that while we have historically talked about "we can't claim to support anything we don't test", we already violate that rule by only testing a subset, even in nightlies. Therefore I'd argue that the fact that this PR brings us to an even smaller subset isn't categorically bad.

[bdice]

@jameslamb For awareness -- I updated this PR following #236. It should be ready for you to pick up when you're ready, perhaps after a bit more work on C++ wheels.

[bdice]

@seberg @vyasr @jameslamb Do we need CI coverage for "oldest dependencies + latest Python version"? I think it might be good to cover that somewhere in our CI to ensure that our pinnings are compatible with all the Python versions we support, but I'm not sure whether others think that is necessary.

[seberg]

I would lean towards not worrying too much about it and not do it if that is the only purpose of the job. I doubt it is a very common setup, and while I think having some tests with the oldest versions is important, I doubt that covering it for all Python versions is.

[jameslamb]

I could go either way on this... right now I'm weakly in favor of adding an "oldest dependencies + latest Python version" job right for nightlies.

To help myself think through this, I wrote out some specific examples below.

Problems I can envision such a job catching that none of the other CI jobs catch (for a fictional dependency somedep):

1. somedep==1.2.0 does not have wheels for python={newest}
   • which we only catch in CI because compilation of an sdist fails, or there isn't an sdist available at all
2. somedep==1.2.0 uses some parts of the Python standard library that were changed in some breaking way by python={newest}
   • most likely when somedep publishes py3-none wheels (pure Python like dask or not linking against Python like xgboost)
3. somedep==1.2.0 has a restrictive pin on Python-requires
   • e.g. requires-python = ">=3.10,<3.12" ... this is rare, but not impossible
   • some discussion on this from 2021: https://discuss.python.org/t/requires-python-upper-limits/12663/7

A CI job that exposes these types of things could inform more detailed pins for wheels, like this:

dependencies = [
  "somedep>=1.2.0 ; python_version < '3.10'",
  "somedep>=1.4.1 ; python_version >= '3.10'"
]

That richer detail is useful for folks using uv's support for solving to the oldest compatible versions (#228 (comment)) or for anyone in an environment where library versions are locked (e.g. with pip freeze) and who want to upgrade as rarely and as slowly as possible.

I've done exactly that before, actually... I worked on a team managing a medium-sized Apache Airflow deployment where a couple dozen Python codebases had to all share a single Python version... and share it with the service itself. When we went to upgrade the Python version in that system, we tried to upgrade as few libraries as possible, and to update them no further than absolutely necessary to support the next version of Python. In that exercise, I was happy to find out about issues like these via failed pip solves... and sad to find out about them at runtime 😅

[jameslamb]

Talked about this with @bdice offline.

I still support adding something like it.... but not for this release. We're getting close to burndown for the 24.10 release and trying to get a large amount of packaging changes in.

I think the matrix proposed here should be left as-is and we could come back to testing "latest Python + oldest dependencies" at some time in the future.

[jameslamb]

I've moved this out of draft, and started the process of opening up PRs to test. See rapidsai/build-planning#40 for a list of those.

I recommend we start rolling out Python 3.12 to RAPIDS repositories after these items:
...
C++ wheels, at least for most of RAPIDS: rapidsai/build-planning#33

To be clear... most of rapidsai/build-planning#33 is NOT done yet, but we've decided to try to get Python 3.12 out in this upcoming release anyway, given that the first Python 3.13 is only a month away: https://devguide.python.org/versions/.

[jameslamb]

I won't delete the python-3.12 branch here until we've updated all the repos currently referencing it.

[jameslamb]

There are 0 remaining references to this branch in the rapidsai org: https://github.com/search?q=org%3Arapidsai%20%22%40python-3.12%22&type=code, so I'm deleting this branch.

If we end up needing it for any reason, it'd be easy to manually recreate from the small diff on this PR.