[FR] Vendor distutils stubs

[Avasam]

What's the problem this feature will solve?

Type-checkers don't see distutils as available in Python 3.12+
This is already causing us much pain when it comes to type-checking, and is becoming a blocker.
One of the main issue is that importing from distutils, types are seen as Any, which breaks subclassing, removes tons of type-safety, and creates nasty inconsistencies when trying to type-check <3.12 vs >=3.12.

We also get some overriding issues when type-checkers don't understand we're using an updated distutils version vendored from pypa/distutils.

As long as _distutils_hack exists, we can't have type-checkers scan the vendored version instead. (if pypa/distutils was installed like a regular package, this wouldn't be an issue either)

Describe the solution you'd like

My suggestion is to vendor distutils-stubs, reflecting the types as the modernized pypa/distutils.
(concretely this means: Start with https://github.com/python/typeshed/tree/main/stdlib/distutils, then apply https://github.com/python/typeshed/tree/main/stubs/setuptools/distutils on top)

Those stubs should exist in the repo in a folder named distutils or distutils-stubs.
distutils-stubs should be installed in users' site-packages when installing setuptools (this can be locked behind an extra if you have concerns for vendors)

This also means we no longer need to wait on new mypy releases to get distutils typing fixes from typeshed AND we can be more accurate to what _distutils does.

Alternative Solutions

• Make pypa/distutils a real package found in site-packages
• Change how the hack works, and always import from ._distutils in setuptools instead.

Additional context

• Merge simple type annotations from typeshed #4504 (see failures for Python 3.12 and 3.13)
• Add type hints to setuptools #2345 (comment)

Code of Conduct

• I agree to follow the PSF Code of Conduct

[abravalheri]

Thank you very much @Avasam for having a look on this.

I will leave the final call on this to Jason.

But my first impression is that this is a bit too intrusive to be handled in this repo. (We don't even maintain distutils, so it would not make much sense to maintain distutils-stubs - if anywhere, pypa/distutils would be a best place for them).

Now, the approach in #4704 is better, but it still introduces a lot of noise and some level of extra maintenance. It would be nice if mypy instead had a configuration that allowed us to point where the distutils package should be loaded from, to workaround mypy's general limitation regarding meta path finders.

I wonder if there would be another approach...

Let's assume that we all are willing to move setuptools/_distutils to setuptools/_private/distutils or something similar. In this case, would simply adding the configuration mypy_path = $MYPY_CONFIG_FILE_DIR/setuptools/_private enough to solve the problem? (i.e. would mypy accept the setuptools/_private/distutils directory as a source of truth for the typings in distutils, or we would still need to add those .pyi files?).

(Moving setuptools/_distutils to a different directory is still something that we have to discuss and I don't know if Jason would be OK with that, but it is certainly a more lightweight alternative than adding 40/50-something extra .pyi files).

[Avasam]

I agree that in the form of #4691, setuptools isn't the place to host these stubs. It is however, the best place for a PR to show the effects. (which probably means I should mark it as draft).
If going with that solution, I agree that it should live somewhere else (preferably with pypa/distutils, if it doesn't introduce distribution issues).

---

#4704 is different though, as setuptools is responsible for the distutils hack. I think that from a separation of concerns pov, pypa/distutils shouldn't need to be aware that setuptools implements it using import hacks nor the exact folder name it lives in. In other words, setuptools is responsible for the distutils hack, and it should be responsible for the stub aliasing distutils to setuptools._distutils (or w/e folder location).

In the hypothetical that setuptools/_distutils is renamed to setuptools/_private/distutils, yes it means that locally for development, we no longer need a stub. But users of setuptools will still need one. And that's true as long as the distutils hack exists and as long as setuptools can't point to distutils as a regular dependency (so once Python 3.11 support is dropped ?). One advantage of the folder rename, is that the alias stubs could be generated on source install/build and included in wheels, so #4704 could potentially be updated to not have the stubs be part of the git source (it already includes the script to generate them)

---

For the sake of completeness, I'll also mention that pypa/distutils could also generate stubs from inline types that would exist in the pypa/distutils repo, get vendored and distributed by setuptools.

---

In all cases, setuptools should in the long term* be responsible to ship stubs-distutils (even if it doesn't "owns" it), so that it gets installed (and maybe uninstalled) with it, even if it's only though an extra.

* As long as types-setuptools is active, in the shorter term typeshed could make distutils-stubs a dependency of types-setuptools. Assuming distutils-stubs is a standalone distribution.