NumPy does not install on Python 3.12.0b1

[bersbersbers]

Describe the issue:

Related:
pypa/pip#11501 (comment)
pypa/setuptools#3935

https://github.com/numpy/numpy/blob/cc0abd768575d7f9e862de0b4912af27f6e9690d/pyproject.toml#LL7C6-L7C16

Reproduce the code example:

pip install numpy

Error message:

pip install numpy
Collecting numpy
  Using cached numpy-1.24.3.tar.gz (10.9 MB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... error
  error: subprocess-exited-with-error

  × Getting requirements to build wheel did not run successfully.
  │ exit code: 1
  ╰─> [33 lines of output]
      Traceback (most recent call last):
        File "C:\Users\bers\.pyenv-win-venv\envs\project_3.12\Lib\site-packages\pip\_vendor\pyproject_hooks\_in_process\_in_process.py", line 353, in <module>
          main()
        File "C:\Users\bers\.pyenv-win-venv\envs\project_3.12\Lib\site-packages\pip\_vendor\pyproject_hooks\_in_process\_in_process.py", line 335, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "C:\Users\bers\.pyenv-win-venv\envs\project_3.12\Lib\site-packages\pip\_vendor\pyproject_hooks\_in_process\_in_process.py", line 112, in get_requires_for_build_wheel
          backend = _build_backend()
                    ^^^^^^^^^^^^^^^^
        File "C:\Users\bers\.pyenv-win-venv\envs\project_3.12\Lib\site-packages\pip\_vendor\pyproject_hooks\_in_process\_in_process.py", line 77, in _build_backend
          obj = import_module(mod_path)
                ^^^^^^^^^^^^^^^^^^^^^^^
        File "C:\Users\bers\.pyenv\pyenv-win\versions\3.12.0b1\Lib\importlib\__init__.py", line 90, in import_module
          return _bootstrap._gcd_import(name[level:], package, level)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "<frozen importlib._bootstrap>", line 1293, in _gcd_import
        File "<frozen importlib._bootstrap>", line 1266, in _find_and_load
        File "<frozen importlib._bootstrap>", line 1216, in _find_and_load_unlocked
        File "<frozen importlib._bootstrap>", line 400, in _call_with_frames_removed
        File "<frozen importlib._bootstrap>", line 1293, in _gcd_import
        File "<frozen importlib._bootstrap>", line 1266, in _find_and_load
        File "<frozen importlib._bootstrap>", line 1237, in _find_and_load_unlocked
        File "<frozen importlib._bootstrap>", line 841, in _load_unlocked
        File "<frozen importlib._bootstrap_external>", line 994, in exec_module
        File "<frozen importlib._bootstrap>", line 400, in _call_with_frames_removed
        File "C:\Users\bers\AppData\Local\Temp\pip-build-env-8dt1uf4c\overlay\Lib\site-packages\setuptools\__init__.py", line 16, in <module>
          import setuptools.version
        File "C:\Users\bers\AppData\Local\Temp\pip-build-env-8dt1uf4c\overlay\Lib\site-packages\setuptools\version.py", line 1, in <module>
          import pkg_resources
        File "C:\Users\bers\AppData\Local\Temp\pip-build-env-8dt1uf4c\overlay\Lib\site-packages\pkg_resources\__init__.py", line 2172, in <module>
          register_finder(pkgutil.ImpImporter, find_on_path)
                          ^^^^^^^^^^^^^^^^^^^
      AttributeError: module 'pkgutil' has no attribute 'ImpImporter'. Did you mean: 'zipimporter'?
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.

Runtime information:

Cannot install numpy :)
Python 3.12.0b1 on Windows 10 21H2

Context for the issue:

No response

[mattip]

We deliberately pin setuptools to work around other problems with numpy.distutils. The plan for python 3.12 is to move to the meson build system, and to deprecate use of setuptools. You can try out the meson build using python3.12 -m pip install -r build_requirements.txt; python3.12 -m spin build.

[bersbersbers]

Thanks, understood!

I just came across https://numpy.org/devdocs/reference/distutils_status_migration.html, which reads

It may be beneficial to wait with migrating until there are examples from other projects to follow

I agree in principle, but I feel that this approach may deadlock if many other projects depend on numpy :) for example, matplotlib does not install on 3.12.0b1 because it depends on numpy at build time, and then that dependency is carried further to all projects depending on matplotlib.

[mattip]

That advice could use an update, especially before the 1.25 release freezes the documentation. SciPy has migrated to meson-python, and NumPy should migrate soon. Rather than trying to get setuptools to work with python3.12, we should start testing the CPython 3.12b1 with meson.

[charris]

That advice could use an update, especially before the 1.25 release freezes the documentation.

Could you open an issue for that? I am going to release 1.25.0rc1 without it, but there will still be time before the final.

[mattip]

matplotlib does not install on 3.12.0b1 because it depends on numpy at build time,

Maybe we should be uploading pre-release wheels to https://anaconda.org/scientific-python-nightly-wheels/numpy. It seems cibuildwheel has a option for this

[rgommers]

I think now that 1.25.x has branched, we can make the switch to Meson by default in main. That will make pip install . & co work with Python 3.12 beta's (module any crashes due to incompatibilities between CPython, NumPy and Cython of course - and we know we have one of those right now: gh-23766). WDYT?

Re uploading a nightly: let's consider it once NumPy actually passes its test suite with a CPython beta? It seems too early now, we don't even run it in our own CI yet.

[mattip]

We need SIMD testing to work to make the transition.

[rgommers]

We need SIMD testing to work to make the transition.

That can still be done with python setup.py install, as is already done in armv7_simd_test.

[rgommers]

I think now that 1.25.x has branched, we can make the switch to Meson by default in main. That will make pip install . & co work with Python 3.12 beta's (module any crashes due to incompatibilities between CPython, NumPy and Cython of course

This is being done in gh-23838 - which is close, but needs working through some CI failures. Until it is merged, one can either use the branch from which I made that PR, or follow the instructions at https://github.com/numpy/numpy/blob/main/building_with_meson.md#pip-install.

Questions about this continue to come in, so I'll pin this issue.

[EwoutH]

Congratulations on getting #23838 in!

What are the next steps in Python 3.12 support?

[rgommers]

What are the next steps in Python 3.12 support?

Uploading nightlies to https://anaconda.org/scientific-python-nightly-wheels, so other projects can use those. PR for that is up, I'll update here once they're available.

[rgommers]

Wheels for Python 312 x86-64 (macOS, Windows, manylinux and musllinux) are up on https://anaconda.org/scientific-python-nightly-wheels/numpy/files. The rest (arm64/aarch64) should land as well by tomorrow when the next scheduled wheel build on Cirrus CI runs.

[lithomas1]

Is there any recommendation for how downstream(e.g. pandas) should build Python 3.12 wheels?

We can't build with the tarball of numpy 1.25 since that uses an ancient setuptools not compatible with Python 3.12, and I don't like the sound of building wheels using a random numpy nightly.

If numpy 2.0 (or whatever the next version is) comes a couple months before Python 3.12 final, I guess it's no big deal, but if not, maybe it might be good to backport #23838?