Support Homebrew patches

[jaraco]

From Homebrew's side we completely override the prefix (base var). This is because under our system, the Python install directory and the install directory that should be used for user-installed packages should be separate (or user packages are lost on upgrades).

We currently do this via distutils.cfg planted in the standard library distutils which will of course not work anymore.

Going forward we're looking at the system that will hopefully be introduced in Python 3.11 to add additional install schemes to sysconfig.py:

• https://gist.github.com/FFY00/625f65681fbcd7fc039dd4d727bb2c2f
• https://bugs.python.org/issue43976

Notably, the idea is schemes should be added rather than replaced. This is why sysconfig.get_preferred_scheme(key) was introduced in Python 3.10. This is also important when dynamic logic may be required. Ideally I reckon distutils should use this when selecting a scheme.

While the above will cover Python 3.10+, there'll still be a question about what we do for <= 3.9. sysconfig._prefix_addition is almost what we want, though I don't know if calculating a relative path and going up levels (..) is a good idea.

Originally posted by @Bo98 in pypa/setuptools#2896 (comment)

[jaraco]

The preferred and expected long-term supported way for any Python environment to customize the install behavior is to provide custom install schemes as solicited here, and this technique is supported on all supported Pythons (3.6+). So if homebrew can patch/extend the install schemes in sysconfig, that would be the best solution for old and new Pythons. If it's going to be more complicated than that, capture any shortcomings here and we can discuss.

[Bo98]

The current mechanism doesn't allow custom install schemes - it requires replacing default install schemes since the selection of schemes is hardcoded:

distutils/distutils/command/install.py

Line 488 in 92082ee

self.select_scheme("posix_prefix")

The mentioned sysconfig.get_preferred_scheme API was specifically introduced in Python 3.10 (https://bugs.python.org/issue43312) for distributors to specify a custom install scheme name:

Python implementers and redistributors may add their preferred schemes to the _INSTALL_SCHEMES module-level global value, and modify this function to return those scheme names, to e.g. provide different schemes for system and language package managers to use, so packages installed by either do not mix with those by the other.

To us, this appears to be the direction the future of sysconfig is taking. For example, while not final yet, the proposal at python/cpython#25718 would explicitly not allow default install schemes to be overridden.

Pip are already onboard and are using it: https://github.com/pypa/pip/blob/bbc7021cde307676a71e7a89e88a6fc20b8e2c26/src/pip/_internal/locations/_sysconfig.py#L70-L71

[jaraco]

It seems distutils needs to support that get preferred scheme, which presumably could be honored even on old Pythons. I can work on an implementation this weekend. If Homebrew can patch sysconfig to extend the schemes and provide a get preferred scheme function, would that provide foreword and backward compatibility?

[Bo98]

Yes, it seems likely we will backport get_preferred_scheme to older Pythons so that plan sounds good to me.