Restore local distutils as the default.

[jaraco]

With the closure of https://github.com/pypa/distutils/milestone/1, Setuptools now should be able to re-enable its local distutils as the default.

Before releasing this change, I'd like for platform reps to test the behavior of Setuptools>=59.2 with SETUPTOOLS_USE_DISTUTILS=local to ensure that the workarounds (reliance on sysconfig schemes, _distutils_system_mod) added in distutils are adequate.

• Debian (@doko42)
• Fedora (@hroncok / @encukou)
• pkgsrc (@0-wiz-0)

I'll give these platforms 30 days (Dec 18) to evaluate the behavior and identify any blockers.

/cc: @hugovk, @mattip, @FFY00

Please feel free to unsubscribe or tag in others.

[hroncok]

I have not seen the status of https://github.com/pypa/distutils/milestone/1 so I will basically just report the status of Fedora:

• Fedora 33 will go EOL this month, so I'll ignore it completely.
• On Fedora 36+ with Python 3.10+, we no longer patch distutils at all. We only patch sysconfig and I assume switching to local distutils should be safe.
• On Pythons 3.11+, the situation is the same on all Fedora versions.
• In Python 3.10 in Fedora 35 and Python 3.9 in Fedora 34 (the "main" Python interpreter in given Fedora releases) we still patch distutils. It is nearly impossible to change this in those already released Fedoras. Fedora 34 goes EOL in ~half a year, Fedora 35 goes EOL in ~1 year.
• In all Fedoras, all older versions of Pythons still patch distutils. We can adapt Python 3.9 to 3.6 on Fedora 36 to match the 3.10+ behavior (it will be arduous but should be possible).

As a side note, we will most likely not be able to change RHEL/EPEL at all. That is a dead-end and I assume we are on our own there. If the setuptools version that switches to local distutils also drops Python 3.6 support, we would be happy.

We could:

1. do the switch in 1 year
2. do the switch conditionally on Python version (as does pip wrt distutils/sysconfig paths)
3. add a hack similar to Integrate Debian patches distutils#4 but instead of guessing the distro, read some stdlib attribute.

I assume that you are not willing to do (1) or (2), so I can definitively look at (3).

Our legacy distutils patch is fedora-python/cpython@42d0bdb#diff-ad882d2747b67da1f6b5a8d3a08f0865ee7bb9fd88c4e8e3ff946d338606e818

Would you take that, guarded by something like if getattr(sysconfig, "_distutils_mangle_rpm_prefix", False)?

[hroncok]

See pypa/distutils#70

[jaraco]

On Fedora 36+ with Python 3.10+, we no longer patch distutils at all. We only patch sysconfig and I assume switching to local distutils should be safe.

One important distinction is that distutils still relies on the headers key of the scheme, which doesn't exist in sysconfig. If the values present in distutils are inadequate for Fedora or Fedora customizes include in sysconfig, I'd double-check that the sysconfig patches include that key.

If the setuptools version that switches to local distutils also drops Python 3.6 support, we would be happy.

That's do-able.

In Python 3.10 in Fedora 35 and Python 3.9 in Fedora 34 (the "main" Python interpreter in given Fedora releases) we still patch distutils. It is nearly impossible to change this in those already released Fedoras. Fedora 34 goes EOL in ~half a year, Fedora 35 goes EOL in ~1 year.

Have you considered in these older environments to recommend/force setting SETUPTOOLS_USE_DISTUTILS=stdlib?

[hroncok]

Have you considered in these older environments to recommend/force setting SETUPTOOLS_USE_DISTUTILS=stdlib?

I wonder how. Setting that environment variable when Python starts?

Also, I assume eventually, this would stop being a possibility, right?

[jaraco]

Have you considered in these older environments to recommend/force setting SETUPTOOLS_USE_DISTUTILS=stdlib?

I wonder how. Setting that environment variable when Python starts?

I don't have any particular opinion. It would need to happen before .pth files are processed. What would happen if it were set system-wide? I guess that would be bad for environments that wish to adopt Setuptools with its default behavior in other Pythons or environments.

Also, I assume eventually, this would stop being a possibility, right?

Eventually, probably around Python 3.9 EOL, but of course we'll strive to provide a smooth transition.

[0-wiz-0]

Hi! Sorry for the perhaps naive question, but what is the best way for me to test this?
Could you please provide a package like you would push to pypi? That seems the easiest way for me to make sure I'll test what will be used later on.

[imba-tjd]

@0-wiz-0 you could use pip install https://github.com/pypa/setuptools/archive/refs/heads/feature/local-distutils.zip

[jaraco]

@0-wiz-0 you could use pip install https://github.com/pypa/setuptools/archive/refs/heads/feature/local-distutils.zip

Or, since all this change does is make local distutils the default, you can simply set SETUPTOOLS_USE_DISTUTILS=local and run with that.

But most important is to use setuptools >= 59.2 as that's the one that's needed to honor platform-specific hooks.

[mkoeppe]

Also need to incorporate mingw/cygwin distutil patches - see pypa/distutils#60 (comment), pypa/distutils#60 (comment);
#2909 adds a CI for Cygwin

[rgommers]

Just checking: does this go into 60.0 and not into 59.x?

[jaraco]

As currently written, the .change will trigger a 0.1 bump. But I agree this should be a .breaking.

[Bo98]

From Homebrew's side we completely override the prefix (base var). This is because under our system, the Python install directory and the install directory that should be used for user-installed packages should be separate (or user packages are lost on upgrades).

We currently do this via distutils.cfg planted in the standard library distutils which will of course not work anymore.

Going forward we're looking at the system that will hopefully be introduced in Python 3.11 to add additional install schemes to sysconfig.py:

• https://gist.github.com/FFY00/625f65681fbcd7fc039dd4d727bb2c2f
• https://bugs.python.org/issue43976

Notably, the idea is schemes should be added rather than replaced. This is why sysconfig.get_preferred_scheme(key) was introduced in Python 3.10. This is also important when dynamic logic may be required. Ideally I reckon distutils should use this when selecting a scheme.

While the above will cover Python 3.10+, there'll still be a question about what we do for <= 3.9. sysconfig._prefix_addition is almost what we want, though I don't know if calculating a relative path and going up levels (..) is a good idea.

[hroncok]

So, it seems that while this is solved wrt our /local/ addition, we have neglected the fact that on Python 3.8 or lesser, we also patch distutils to use lib64: fedora-python/cpython@31397a8

That means, users who run setuptools 60+ on Fedora with Python 3.7 or 3.8 might experience weirdness, as described in for example in pylint-dev/pylint#5704

[hroncok]

I've opened pypa/distutils#110

[0-wiz-0]

I didn't want to open a new issue for this, but I think there's a problem related to the override file.
pkgsrc's python packages install ${PREFIX}/lib/python${PYTHON_VERSION}/site-packages/_distutils_system_mod.py files, they are very small:

# _distutils_system_mod

import distutils.sysconfig


vars(distutils.sysconfig).update(
    _makefile_tmpl='config-{python_ver}{build_flags}',
    _sysconfig_name_tmpl='_sysconfigdata_{platform}',
)

This works in all the normal use cases. However, there seems to be a problem with virtualenv not copying that file:
TritonDataCenter/pkgsrc#323

Is this a bug in virtualenv?

[0-wiz-0]

I've filed pypa/virtualenv#2313 for this.

[jaraco]

The problem here may be that there's no strategic design for how _distutils_system_mod is meant to work. I'd be disinclined to recommend that tools like virtualenv and pip need to start honoring this file specially. On the other hand, since these tools previously would copy patched copies of distutils from the stdlib, maybe it's not so bad that they now also need to copy the monkeypatches needed by distutils as found in setuptools.

The long-term solution here is going to be to remove distutils' own reliance on distutils.sysconfig and instead rely on sysconfig.