[BUG] distutils cannot be imported on Python 3.12 (without distutils) if pip was imported

[vstinner]

setuptools version

65.5.0

Python version

3.12 (dev, without distutils)

OS

Linux

Additional environment information

No response

Description

I just removed the distutils packages from Python 3.12: commit. setuptools works as expected, it provides its copy of distutils. Good.

But if pip is imported first, import setuptools or import distutils fails on import setuptools, since _distutils_hack has a special path when pip is installed which requires to get the stdlib distutils... which no longer exists.

Extract of _distutils_hack/__init__.py:

    def spec_for_pip(self):
        """
        Ensure stdlib distutils when running under pip.
        See pypa/pip#8761 for rationale.
        """

See pypa/pip#8761 for the rationale.

Well, this special case should be revisited for Python >= 3.12.0 alpha 2.

Expected behavior

Importing setuptools on Python 3.12 after pip was imported should just work.

How to Reproduce

$ python3.12
Python 3.12.0a1+ (heads/main:f09da28768, Nov  4 2022, 17:46:22) [GCC 12.2.1 20220819 (Red Hat 12.2.1-2)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pip
>>> import setuptools

Output

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/py312/lib/python3.12/site-packages/setuptools/__init__.py", line 8, in <module>
    import _distutils_hack.override  # noqa: F401
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/py312/lib/python3.12/site-packages/_distutils_hack/override.py", line 1, in <module>
    __import__('_distutils_hack').do_override()
  File "/opt/py312/lib/python3.12/site-packages/_distutils_hack/__init__.py", line 77, in do_override
    ensure_local_distutils()
  File "/opt/py312/lib/python3.12/site-packages/_distutils_hack/__init__.py", line 60, in ensure_local_distutils
    importlib.import_module('distutils')
  File "/opt/py312/lib/python3.12/importlib/__init__.py", line 124, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ModuleNotFoundError: No module named 'distutils'

[AMDmi3]

It doesn't seem to be related to pip, setuptools just do not support python 3.12. When running setuptools' setup.py install with python 3.12.0a3, as it is done when installing py-setuptools package on FreeBSD:

...
byte-compiling /work/usr/ports/devel/py-setuptools/work-py312/stage/usr/local/lib/python3.12/site-packages/_distutils_hack/override.py to override.cpython-312.pyc
writing byte-compilation script '/tmp/tmph5sr5lui.py'
/usr/local/bin/python3.12 /tmp/tmph5sr5lui.py
Traceback (most recent call last):
  File "/tmp/tmph5sr5lui.py", line 1, in <module>
    from distutils.util import byte_compile
ModuleNotFoundError: No module named 'distutils'
error: command '/usr/local/bin/python3.12' failed with exit code 1

[OTheDev]

I get this even without explicitly importing pip (i.e. just importing setuptools). Any further information on this?

[sathyz]

Hi,
I'm trying to take up this PR. Would it be fair to move distutils modules from cpython to setuptools to handle this?

[abravalheri]

Hi @sathyz, thank you very much for working on this. If you need to change anything of distutils please submit a PR to https://github.com/pypa/distutils.

But on a first glance I think we could tackle this issue by modifying https://github.com/pypa/setuptools/blob/main/_distutils_hack/__init__.py... What if the finder is always enabled for Python >= 3.12?

[sathyz]

Hi @abravalheri
I'm able to get this working by installing pypa/distutils and with the following change,

@@ -61,7 +61,8 @@ def ensure_local_distutils():

     # check that submodules load as expected
     core = importlib.import_module('distutils.core')
-    assert '_distutils' in core.__file__, core.__file__
+    if sys.version_info < (3, 12):
+        assert '_distutils' in core.__file__, core.__file__
     assert 'setuptools._distutils.log' not in sys.modules

I tried installing a package that was breaking earlier and is now working fine. One of the problem that I see is, pypa/distutils is not in pypi.org that I couldn't do pip install distutils, until this is available in PyPi repository we wouldn't be able to have it as dependency, am I correct? Any thoughts?

[nikolay]

After installing Python 3.12 today on macOS, I get this:

$ asdf install python 3.12.0
python-build 3.12.0 /Users/[REDACTED]/.asdf/installs/python/3.12.0
python-build: use openssl@3 from homebrew
python-build: use readline from homebrew
Downloading Python-3.12.0.tar.xz...
-> https://www.python.org/ftp/python/3.12.0/Python-3.12.0.tar.xz
Installing Python-3.12.0...
python-build: use readline from homebrew
python-build: use zlib from xcode sdk
Installed Python-3.12.0 to /Users/[REDACTED]/.asdf/installs/python/3.12.0
$ python
Python 3.12.0 (main, Oct  2 2023, 15:38:53) [Clang 15.0.0 (clang-1500.0.40.1)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import distutils
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'distutils'

[abravalheri]

Hi @nikolay , have you installed setuptools?

[nikolay]

@abravalheri So, with 3.12+, do I have to install setuptools separately? The compressed Release Notes don't really say that.

[abravalheri]

So, with 3.12+, do I have to install setuptools separately?

Yes, to use Setuptools' copy of distutils, you need to install setuptools and setuptools is not part of the stdlib.

The compressed Release Notes don't really say that.

Not sure what we can do about that in the context of the setuptools github project. Maybe you should contact the Python project instead?

[vstinner]

The compressed Release Notes don't really say that.

Read What’s New In Python 3.12:

• PEP 632: Remove the distutils package. See the migration guide for advice replacing the APIs it provided. The third-party Setuptools package continues to provide distutils, if you still require it in Python 3.12 and beyond.
• Do not pre-install setuptools in virtual environments created with venv.

[vstinner]

I confirm that the bug is fixed:

vstinner@mona$ /opt/py312/bin/python3
Python 3.12.0 (main, Oct  2 2023, 18:19:14) [GCC 13.2.1 20230728 (Red Hat 13.2.1-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pip
>>> import distutils

vstinner@mona$ /opt/py312/bin/python3 -m pip list
Package         Version
--------------- ------------
pip             23.2.1
setuptools      68.1.2
(...)

Thanks for the fix!

[auxten]

Experiencing same issue on chdb pipeline. Fixed by pip install setuptools