[BUG] ModuleNotFoundError: No module named 'distutils.msvccompiler'

[rolweber]

setuptools version

setuptools==74.0.0

Python version

Python 3.11

OS

RedHat UBI 9 on x86

Additional environment information

I'm installing packages with pip into conda environments.

Description

I have to install POT==0.8.2 from PyPI on Linux x86_64 in container image builds. A wheel gets built on the fly at installation time. This used to work earlier this week, but is failing today, after the release of setuptools 74.0.0. Debug output shows this:

    File "/tmp/pip-build-env-0kfp4m0w/overlay/lib/python3.11/site-packages/numpy/distutils/mingw32ccompiler.py", line 28, in <module>
      from distutils.msvccompiler import get_build_version as get_build_msvc_version
  ModuleNotFoundError: No module named 'distutils.msvccompiler'
  error: subprocess-exited-with-error
  
  × Getting requirements to build wheel did not run successfully.
  │ exit code: 1
  ╰─> See above for output.

I can work around the problem by constraining setuptools to an older version:

export PIP_CONSTRAINT="/tmp/pip-constraint.txt"
echo >$PIP_CONSTRAINT "setuptools<74"
pip install POT==0.8.2

Not sure if this is actually a problem in setuptools, or just with old package versions being installed while building the wheel. So feel free to close this issue if things work as designed. However, I'm surprised that setuptools is even looking for a Microsoft compiler on Linux x86, and even more so that it fails while looking, rather than just proceeding as if it hasn't found anything.

Expected behavior

Building of the POT wheel should succeed.

How to Reproduce

pip install -v POT==0.8.2 on Python 3.11, for which there is no wheel on PyPI.

or pip install -v --no-binary POT POT==0.8.2 on Python versions for which there is a wheel on PyPI.

Output

Using pip 23.3.1 from /opt/conda/envs/Python-RT24.1/lib/python3.11/site-packages/pip (python 3.11)
Collecting POT==0.8.2
  Downloading POT-0.8.2.tar.gz (255 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 255.8/255.8 kB 3.3 MB/s eta 0:00:00
  Running command pip subprocess to install build dependencies
  Collecting setuptools
    Downloading setuptools-74.0.0-py3-none-any.whl.metadata (6.7 kB)
  Collecting wheel
    Downloading wheel-0.44.0-py3-none-any.whl.metadata (2.3 kB)
  Collecting oldest-supported-numpy
    Downloading oldest_supported_numpy-2023.12.21-py3-none-any.whl.metadata (9.8 kB)
  Collecting cython>=0.23
    Downloading Cython-3.0.11-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (3.2 kB)
  Collecting numpy==1.23.2 (from oldest-supported-numpy)
    Downloading numpy-1.23.2-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (2.2 kB)
  Downloading setuptools-74.0.0-py3-none-any.whl (1.3 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.3/1.3 MB 17.7 MB/s eta 0:00:00
  Downloading wheel-0.44.0-py3-none-any.whl (67 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 67.1/67.1 kB 3.8 MB/s eta 0:00:00
  Downloading oldest_supported_numpy-2023.12.21-py3-none-any.whl (4.9 kB)
  Downloading numpy-1.23.2-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (17.0 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 17.0/17.0 MB 50.4 MB/s eta 0:00:00
  Downloading Cython-3.0.11-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.6 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.6/3.6 MB 38.6 MB/s eta 0:00:00
  Installing collected packages: wheel, setuptools, numpy, cython, oldest-supported-numpy
  ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
  ml-dtypes 0.2.0 requires numpy>=1.23.3; python_version > "3.10", but you have numpy 1.23.2 which is incompatible.
  onnxruntime 1.16.3 requires numpy>=1.24.3, but you have numpy 1.23.2 which is incompatible.
  tensorflow 2.14.1 requires numpy<2.0.0,>=1.23.5, but you have numpy 1.23.2 which is incompatible.
  autoai-ts-libs 4.0.10 requires numpy==1.26.*, but you have numpy 1.23.2 which is incompatible.
  autoai-libs 2.0.8 requires numpy==1.26.*, but you have numpy 1.23.2 which is incompatible.
  Successfully installed cython-3.0.11 numpy-1.23.2 oldest-supported-numpy-2023.12.21 setuptools-74.0.0 wheel-0.44.0
  WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
  Installing build dependencies ... done
  Running command Getting requirements to build wheel
  Traceback (most recent call last):
    File "/opt/conda/envs/Python-RT24.1/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
      main()
    File "/opt/conda/envs/Python-RT24.1/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
      json_out['return_val'] = hook(**hook_input['kwargs'])
                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/opt/conda/envs/Python-RT24.1/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 118, in get_requires_for_build_wheel
      return hook(config_settings)
             ^^^^^^^^^^^^^^^^^^^^^
    File "/tmp/pip-build-env-5jaggw7h/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 332, in get_requires_for_build_wheel
      return self._get_build_requires(config_settings, requirements=[])
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/tmp/pip-build-env-5jaggw7h/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 302, in _get_build_requires
      self.run_setup()
    File "/tmp/pip-build-env-5jaggw7h/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 318, in run_setup
      exec(code, locals())
    File "<string>", line 35, in <module>
    File "/tmp/pip-install-lar6ht2_/pot_bb8b65f32d9047bc968e1d2cecf70a43/ot/helpers/openmp_helpers.py", line 65, in check_openmp_support
      output, compile_flags = compile_test_program(
                              ^^^^^^^^^^^^^^^^^^^^^
    File "/tmp/pip-install-lar6ht2_/pot_bb8b65f32d9047bc968e1d2cecf70a43/ot/helpers/pre_build_helpers.py", line 62, in compile_test_program
      ccompiler.compile(['test_program.c'], output_dir='objects',
    File "/tmp/pip-build-env-5jaggw7h/overlay/lib/python3.11/site-packages/numpy/distutils/ccompiler.py", line 89, in <lambda>
      m = lambda self, *args, **kw: func(self, *args, **kw)
                                    ^^^^^^^^^^^^^^^^^^^^^^^
    File "/tmp/pip-build-env-5jaggw7h/overlay/lib/python3.11/site-packages/numpy/distutils/ccompiler.py", line 272, in CCompiler_compile
      jobs = get_num_build_jobs()
             ^^^^^^^^^^^^^^^^^^^^
    File "/tmp/pip-build-env-5jaggw7h/overlay/lib/python3.11/site-packages/numpy/distutils/misc_util.py", line 91, in get_num_build_jobs
      from numpy.distutils.core import get_distribution
    File "/tmp/pip-build-env-5jaggw7h/overlay/lib/python3.11/site-packages/numpy/distutils/core.py", line 24, in <module>
      from numpy.distutils.command import config, config_compiler, \
    File "/tmp/pip-build-env-5jaggw7h/overlay/lib/python3.11/site-packages/numpy/distutils/command/config.py", line 19, in <module>
      from numpy.distutils.mingw32ccompiler import generate_manifest
    File "/tmp/pip-build-env-5jaggw7h/overlay/lib/python3.11/site-packages/numpy/distutils/mingw32ccompiler.py", line 28, in <module>
      from distutils.msvccompiler import get_build_version as get_build_msvc_version
  ModuleNotFoundError: No module named 'distutils.msvccompiler'
  error: subprocess-exited-with-error
  
  × Getting requirements to build wheel did not run successfully.
  │ exit code: 1
  ╰─> See above for output.
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  full command: /opt/conda/envs/Python-RT24.1/bin/python /opt/conda/envs/Python-RT24.1/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py get_requires_for_build_wheel /tmp/tmpfpik1qz4
  cwd: /tmp/pip-install-lar6ht2_/pot_bb8b65f32d9047bc968e1d2cecf70a43
  Getting requirements to build wheel ... error
error: subprocess-exited-with-error

× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.

[abravalheri]

Hi @rolweber, please note that the distutils.msvccompiler module was deprecated 2 years ago: pypa/distutils@c802880, and it has been issuing a deprecation warning since then.

After 2 years it has been removed accordingly. So probably it is worth contacting POT about that.

I'm surprised that setuptools is even looking for a Microsoft compiler on Linux x86, and even more so that it fails while looking, rather than just proceeding as if it hasn't found anything.

Setuptools does not look for it. It is probably some other module. By looking at your traces, it seems that numpy is File "/tmp/pip-build-env-0kfp4m0w/overlay/lib/python3.11/site-packages/numpy/distutils/mingw32ccompiler.py", line 28, in <module>, and somehow POT uses numpy.

Might be related to the fact that numpy community is no longer invested in supporting the latest versions of setuptools/distutils (numpy.distutils has been deprecated for a while too).

[rolweber]

Thanks. POT has a dependency on the oldest numpy version that it supports... no updates will change that. 🤷
I'll be fine with the workaround.

[abravalheri]

If POT is committed to some particular versions of numpy, they might have to pin setuptools too to avoid incompatibility problems like this one for the end user.

[rolweber]

POT already moved on to a 0.9.x series, but the package I have to install still requires the older 0.8.x. I'm hoping for the next version of that... :-)

[mxmlnkn]

I see no mention of what I should use instead of these deprecated and removed classes? I also see this issue suddenly in my CI.