Add support for Python 3.11, require NumPy 1.23+

[jameslamb]

Description

Contributes to rapidsai/build-planning#3

This PR adds support for Python 3.11.

It also bumps uses of NumPy to numpy>=1.23, see rapidsai/build-planning#3 (comment).

Notes for Reviewers

This is part of ongoing work to add Python 3.11 support across RAPIDS.

The Python 3.11 CI workflows introduced in rapidsai/shared-workflows#176 are optional... they are not yet required to run successfully for PRs to be merged.

This PR can be merged once all jobs are running successfully (including the non-required jobs for Python 3.11). The CI logs should be verified that the jobs are building and testing with Python 3.11.

See rapidsai/shared-workflows#176 for more details.

[jakirkham]

cc @thewtex (for awareness)

[jakirkham]

It looks like the imports were reordered in a bunch of places. Is this from isort or something?

Also are there any related updates needed to .pre-commit-config.yaml?

[jameslamb]

It looks like the imports were reordered in a bunch of places. Is this from isort or something?

All of these import-related changes were made by ruff when I ran pre-commit run --all-files.

I see the exact same changes made if I run pre-commit the way this project does in CI.

cucim/ci/check_style.sh

Line 18 in 47e0f74

pre-commit run --hook-stage manual --all-files --show-diff-on-failure

So I'm not sure how these changes weren't previously caught in CI. All of the checks in https://github.com/rapidsai/cucim/blob/branch-24.04/.pre-commit-config.yaml are pinned to specific versions of things, so not sure what changed between #694 and now 🤔

[KyleFromNVIDIA]

https://github.com/rapidsai/cucim/actions/runs/8070438319/job/22048142083?pr=704 is failing because we're building against numpy 1.21.3, which doesn't support Python 3.11.

[jameslamb]

because we're building against numpy 1.21.3, which doesn't support Python 3.11

Thanks for finding that!

I just did a bit of research, sharing some notes that might help inform what we do about this.

From https://numpy.org/news/#numpy-1240-released, it looks like numpy==1.24.* (December 2022) was the first one to fully support Python 3.11.

It looks like numpy==1.23.2 (August 2022) was the first to have CPython 3.11 wheels (https://pypi.org/project/numpy/1.23.2/#files).

cudf is allowing numpy to float, numpy>=1.21 (cudf/pyproject.toml).

cucim is not pinning to a specific numpy in its wheel runtime dependencies

cucim/python/cucim/pyproject.toml

Line 29 in 47e0f74

"numpy",

But is pinning to numpy==1.21.* in its conda runtime dependencies

cucim/conda/recipes/cucim/meta.yaml

Line 66 in 47e0f74

- numpy 1.21

[KyleFromNVIDIA]

It was pinned specifically to 1.21 in #480, because that's the oldest version of numpy we support.

[bdice]

But [cucim] is pinning to numpy==1.21.* in its conda runtime dependencies

cucim/conda/recipes/cucim/meta.yaml

Line 66 in 47e0f74

- numpy 1.21

⚠️ Note the dependency on numpy is in host, so it's a build dependency here, not a runtime dependency! We try to always build against the oldest supported numpy version, because numpy requires run_version >= (build or host)_version.

conda-forge is using numpy 1.23 with python 3.11: https://github.com/conda-forge/conda-forge-pinning-feedstock/blob/bd0cc0a05ae8fc09f687ca52fcdc0cf4acbb5306/recipe/conda_build_config.yaml#L694-L699

We should be able to update this pin to 1.23 wherever numpy is used as a host dependency. This also applies to some other packages like RAFT.

[jakirkham]

Yeah in Python 3.11 NumPy was bumped to 1.23

The simplest solution would be to bump to NumPy 1.23 here

If for some reason that does not work, we can use NumPy 1.21 for older Python versions and NumPy 1.23 for Python 3.11

Edit: It looks like Bradley posted this at the same time

[jameslamb]

⚠️ Note the dependency on numpy is in host, so it's a build dependency here, not a runtime dependency

Ah yeah sorry about that, I could have chosen a better link. It's build dependency AND a runtime dependency, via this

cucim/conda/recipes/cucim/meta.yaml

Line 75 in 47e0f74

- {{ pin_compatible('numpy') }}

right?

[jakirkham]

/merge

[jakirkham]

Thanks all! 🙏