[FR] Keep setuptool's distutils fully compatible with Python stdlib's distutils

[tiran]

What's the problem this feature will solve?

setuptools contains a copy of distutils from Python's stdlib. When setuptools is installed globally or in a virtual environment, the _distutils_hack injects setuptool's distutils into the import system. This overrides Python stdlib's distutils with setuptools' copy.

A lot of projects depend on public and internal features of distutils. This means any deviation from Python's distutils is an API incompatibility issue and possibly a breaking change. For example #3505 removed two modules related to MSVC compiler. distutils.msvccompiler is a public, documented API of distutils.

Describe the solution you'd like

setuptools must not remove, change, or deviate from any distutils APIs until it drops support for Python 3.11 (!). Yes, that means you would have to support your copy for distutils until October 2027. In some cases it may even be required to stay bug compatible.

Alternative Solutions

setuptools could consider to use Python stdlib's distutils for Python 3.11 and earlier. The distutils package is available until 3.11. You may have to monkey-patch some of the fixes, though.

Additional context

PR #3505 links a bunch of projects that got broken by removal of distutils.msvccompiler

Code of Conduct

• I agree to follow the PSF Code of Conduct

[maresb]

For package maintainers desperate for a quick workaround: pin setuptools !=65.*.

[random-walks]

bump

[jaraco]

Thanks @tiran for the proposal. Originally, I responded in the PR, but I'll move my response to here:

Setuptools works at a different cadence than Python. It has its own release schedule and semver-based versioning. As a result, users of Python 3.7-3.11 (and probably later) can depend on Setuptools < 65 for compatibility.

It's able to move much faster for some things, but still aims not to be disruptive. If you have a look at the history in pypa/distutils, you'll see there have been many changes that are technically API breaking, but which had little or no negative impact. At the same time, there are many improvements being introduced to limit the variation across platforms and eliminating the need for monkeypatches.

That said, because it's a build tool, I recognize that it's not a simple matter of pinning to a version for compatibility. Often a project has little control over which versions of Setuptools may be present in an environment (especially in the future).

---

There are benefits to not being frozen and users are reaping those benefits as distutils evolves carefully and selectively but provides a uniform implementation across all supported Python versions and providing continuous backward compatibility through older releases.

Although #3505 revealed an unintentional breakage, it was an intentional risk to reduce the legacy burden that distutils carries. Setuptools is able to adapt quickly and roll back unintentional breakage. For known breaking changes like these, I aim to have < 24h turnaround for a fix. I did check in on the project Sunday evening and Monday evening for any emergent issues, but didn't see any. I had not checked my email, so I missed the discussion in the PR until today. To be sure, if there's widespread breakage, do feel free to reach out in other channels.

Regarding maintaining compatibility with stdlib, there is an escape hatch. If users wish to drop back to stdlib behavior, they can still do so with SETUPTOOLS_USE_DISTUTILS=stdlib. I'm currently working on a plan to obviate that behavior, to wean users off of distutils entirely so that _distutils_hack can be removed, hopefully long before 2027.

[tiran]

It looks like the issue boils down to the question who "owns" the distutils package namespace.

You seem to have the understanding that the setuptools development team and PyPA now owns the distutils package namespace and can do as the setuptools team sees fit.

As a Python core developer I have a completely different opinion. In my understanding the distutils package namespace is still owned by Python core. Any changes to distutils must follow the same scrutiny and deprecation policy as any other Python core feature. My reasoning is:

• Python 3.11 and earlier ship distutils
• The official documentation is https://docs.python.org/3.11/library/distutils.html
• Python's sys modules declares that distutils is a part of the stdlib, "distutils" in sys.stdlib_module_names is True.
• It's surprising for users that the presence of setuptools package changes the behavior and feature set of distutils. It's only obvious for people that are familiar with internals of Python packaging and the history of setuptools.

Although #3505 revealed an unintentional breakage, it was an intentional risk to reduce the legacy burden that distutils carries.

This may sound overly harsh and aggressive, but if setuptools is unwilling or unable to shoulder the legacy burden of distutils, then you should explore a different approach to deal with distutils. For example you could still use distutils from Python's stdlib for <= 3.11 and only use your copy for 3.12+.

If setuptools wants to provide a modified distutils package namespace, then you should coordinate any change with Python core and ensure that the canonical documentation is updated accordingly. Core may object some changes or request a deprecation phase. For the #3505 changeset we would probably have asked you to include a deprecation warning for at least one year.

Regarding maintaining compatibility with stdlib, there is an escape hatch. If users wish to drop back to stdlib behavior, they can still do so with SETUPTOOLS_USE_DISTUTILS=stdlib.

It seems like the escape hatch does not work for anybody who uses pep517 build isolation. tox users need to change their config in order to pass the env var.

[jaraco]

It seems like the escape hatch does not work for anybody who uses pep517 build isolation.

That claim was a surprise to me, so I checked. The escape hatch does work with build isolation:

 draft $ cat > setup.py
import distutils
raise ValueError(distutils.__file__)
 draft $ env SETUPTOOLS_USE_DISTUTILS=stdlib pip install --use-pep517 .
Processing /Users/jaraco/draft
  Installing build dependencies ... done
  Getting requirements to build wheel ... error
  error: subprocess-exited-with-error
  
  × Getting requirements to build wheel did not run successfully.
  │ exit code: 1
  ╰─> [19 lines of output]
      Error in sitecustomize; set PYTHONVERBOSE for traceback:
      AssertionError:
      Traceback (most recent call last):
        File "/opt/homebrew/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 363, in <module>
          main()
        File "/opt/homebrew/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 345, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
        File "/opt/homebrew/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 130, in get_requires_for_build_wheel
          return hook(config_settings)
        File "/opt/homebrew/Cellar/python@3.10/3.10.6_1/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/setuptools/build_meta.py", line 177, in get_requires_for_build_wheel
          return self._get_build_requires(
        File "/opt/homebrew/Cellar/python@3.10/3.10.6_1/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/setuptools/build_meta.py", line 159, in _get_build_requires
          self.run_setup()
        File "/opt/homebrew/Cellar/python@3.10/3.10.6_1/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/setuptools/build_meta.py", line 281, in run_setup
          super(_BuildMetaLegacyBackend,
        File "/opt/homebrew/Cellar/python@3.10/3.10.6_1/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/setuptools/build_meta.py", line 174, in run_setup
          exec(code, locals())
        File "<string>", line 2, in <module>
      ValueError: /opt/homebrew/Cellar/python@3.10/3.10.6_1/Frameworks/Python.framework/Versions/3.10/lib/python3.10/distutils/__init__.py
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.

The issue in that comment was the inability of the installer to control which setuptools version is used in a build.

[hroncok]

Alternatively, if setuptools needs their modified version of distutils, they could do so without overriding the standard library one.

[hroncok]

Another case of a breaking change is #3143

[hodgestar]

I think we can all agree that historically the setuptools and distutils code bases were pretty messy and entangled and there was / is a lot of monkey patching that makes it hard to understand what is happening. @jaraco and company are trying to make these tools better, and as someone who has written a build_ext override for HPy, I fully support (that was a painful week of my like :).

By removing distutils from Python 3.11, and having had distutils be frozen in core for some time, the core Python development team are handing over responsibility for distutils (or at least its use within setuptools and the general Python build ecosystem) to the PyPA group.

As a maintainer of a moderately complex scientific Python package (qutip) I also don't want things to break for me or for the users of qutip though, and in that respect, I agree with @tiran that the breakages of the last week have been pretty bad.

It's worth acknowledging that this handover period from core to setuptools is just going be a bit tricky. Maybe the plan does need to change a bit, but we all want what's best for our ecosystem, so we agree on the big picture, even if we disagree on some details.

[tiran]

Alternatively, if setuptools needs their modified version of distutils, they could do so without overriding the standard library one.

+1, although that might break projects that mix distutils and setuptools in an unholy way, use distutils entry points, and rely on the fact that their subclasses of distutils commands are properly handled by setuptools. It's a special case. I'm not even sure that the problem occurs in the wild.

By removing distutils from Python 3.11 [...]

Small correction, Python 3.11 still ships distutils. distutils will be removed in 3.12. You are still good for another year.

Until recently I assumed that setuptools was going to take a different approach. I thought that the project was going to gobble up and consume distutils code entirely. It came as a surprise to me that setuptools started to override Python's namespace. I don't recall that this was discussed with Python core at all. As far as I remember we learned about the fact when setuptools started to break CPython builds and tests for users, e.g. #3007 and python/cpython#91169.

IMHO SETUPTOOLS_USE_DISTUTILS=stdlib should be the default and SETUPTOOLS_USE_DISTUTILS=local should be opt-in.

[lazka]

although that might break projects that mix distutils and setuptools in an unholy way

not sure how common that is, but I use the following in all my projects: https://github.com/pygobject/pycairo/blob/7a725bd1a4e7ba1777835aab63d73de1a5d97399/setup.py#L22 It was from a time when setuptools wasn't that common and didn't add much over distutils but it needed to be supported because pip would inject it either way.

[rgommers]

The current breakage seems to be concentrated in numpy.distutils, so let me say a few things as the maintainer of that package:

• numpy.distutils extends, relies on or monkey patches a very large part of all of distutils, and given that distutils has no public vs. private API distinction basically any change to distutils can break things,
• the setuptools maintainers have been quite clear about their plans, and we (NumPy maintainers) are supportive of those,
• users of numpy.distutils should pin setuptools to the last known working version, and we have communicated that for quite a while (since around setuptools 60.0), see BUG: numpy.distutils fails with setuptools==65 numpy/numpy#22135 (comment),
• this is in general a healthy thing to do for build-time only dependencies, with pip & co defaulting to build isolation that cannot break other packages and it's a good idea for large packages that depend on distutils or numpy.distutils in any way that deviates from directly relying on setuptools,

and as someone who has written a build_ext override for HPy

I already recommended this to another HPy maintainer elsewhere, but: we should all stop doing this. This is what we were forced to do when distutils was in the stdlib, but it would be much better to simply contribute HPy support to setuptools rather than propagate more extensions. Fixing monkey patch races when you're using multiple tools that all modify build_ext is no fun.

The same goes for Cython, Pythran, etc.

IMHO SETUPTOOLS_USE_DISTUTILS=stdlib should be the default and SETUPTOOLS_USE_DISTUTILS=local should be opt-in.

I suspect that will make it much harder on the setuptools maintainers to get anywhere, because now they have two versions of distutils to deal with. They already have a very hard job, so I don't like this suggestion much. There is no way to make something sane out of distutils without significant breakages regularly, so asking for stability until 2027 does not seem very reasonable.

The setuptools maintainers are very proactive when things break, so that is as much as we can ask for I'd say.

[jaraco]

IMHO SETUPTOOLS_USE_DISTUTILS=stdlib should be the default and SETUPTOOLS_USE_DISTUTILS=local should be opt-in.

This condition was the case from Setuptools 50 through Setuptools 59, for most of 2021. And yet, there were zero reports of issues relating to it, even with a specific call to action. Platforms continued to patch stdlib and rely on those patches. Projects continued to import and rely on distutils. It wasn't until Setuptools 60 was released that the platform-specific behaviors could even be identified.

I thought that the project was going to gobble up and consume distutils code entirely. It came as a surprise to me that setuptools started to override Python's namespace.

This approach was what I had in mind also, but it didn't take long to realize that because of monkepatches, import races, and a long-established history of reliance on distutils' interfaces, users were not going to have a happy time if Setuptools simply stopped relying on the distutils namespace. Such an approach would have been unbearably disruptive and caused many existing packages to break across all releases.

The _distutils_hack was the best technique I could identify to provide backward compatibility and a transition from a reliance on distutils. It provides a smoother transition across Python versions (instead of a harsh transition expected in Python 3.12). Most users would prefer their projects to have compatibility across Python versions and not have to have separate implementations across Python boundaries.

Some of these concerns were alluded to in bpo-41282, though not discussed directly. Moreover, CPython core devs have commented on the startup performance concerns of _distutils_hack, so it was my impression that core devs were aware of what's been happening here.

It was my understanding that Setuptools was adopting the distutils namespace and not just the code. If the core devs would like for Setuptools not to own that namespace, it would have been nice to have known that earlier in the preview of this behavior in late 2020. It would have been dramatically easier for Setuptools to simply adopt the code and break the prior expectations.

It's simply untenable at this stage for distutils to remain fully compatible with stdlib. The code has evolved and advanced too much to abandon that work.

In my opinion, if Python core wished to retain rights over the use of the distutils package name, someone should have raised that earlier. I recognize that maybe the approach wasn't advertised and thus scrutinized as much as it could have been. In any case, if Python would like Setuptools to drop support for supplying distutils, it can do that, and it will make the problem of adopting distutils much simpler at the cost of losing compatibility with all of the packages that rely on the current behaviors, compatibility that has taken a great deal of effort to retain while making this transition.

If you still feel like that's the best approach and wish to have that conversation with Python core, then let's do that.

[hodgestar]

@jaraco Thank you for writing all of that up and explaining all of the histories and challenges. It was very useful.

[hodgestar]

I already recommended this to another HPy maintainer elsewhere, but: we should all stop doing this. This is what we were forced to do when distutils was in the stdlib, but it would be much better to simply contribute HPy support to setuptools rather than propagate more extensions. Fixing monkey patch races when you're using multiple tools that all modify build_ext is no fun.

HPy would very much like to become officially supported in setuptools and would happily contribute patches, but it still feels slightly too early for that. And I doubt setuptools would accept such code unless HPy was a bit more mature / accepted as the future of the C API. If any setuptools maintainers think that it is not too soon, let me know.

Many projects use custom build extensions and most of them probably can't become part of setuptools. It would be helpful to know what these projects should aim for instead.

[jaraco]

I already recommended this to another HPy maintainer elsewhere, but: we should all stop doing this. This is what we were forced to do when distutils was in the stdlib, but it would be much better to simply contribute HPy support to setuptools rather than propagate more extensions. Fixing monkey patch races when you're using multiple tools that all modify build_ext is no fun.

HPy would very much like to become officially supported in setuptools and would happily contribute patches, but it still feels slightly too early for that. And I doubt setuptools would accept such code unless HPy was a bit more mature / accepted as the future of the C API. If any setuptools maintainers think that it is not too soon, let me know.

Many projects use custom build extensions and most of them probably can't become part of setuptools. It would be helpful to know what these projects should aim for instead.

It's not too early, but there are some caveats, the main one being that distutils and setuptools are still maintained independently (with the latter vendoring the former), so any concern that affects both may require some special handling. That also means that any changes require extra care because the process from merge to release is more complicated. That said, if the intended feature/behavior is dominantly implemented in distutils, I'd recommend to propose/design/contribute the changes in pypa/distutils, or if the behavior relies more heavily on Setuptools-specific behavior (and barely touches the code in distutils), then drive it here. If you're unsure, just pick one and we can move it.

[jaraco]

@tiran Thanks for the proposal, but I don't think this proposal has much merit. We can discuss more at the summit. I'll be there virtually.