Allow Python distributors to add custom site install schemes

[FFY00]

BPO	43976
Nosy	@malemburg, @jaraco, @tiran, @encukou, @stefanor, @zooba, @FRidh, @hroncok, @frenzymadness, @FFY00, @jakirkham
PRs	bpo-43976: add vendor config #25718

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2021-04-29.16:19:02.410>
labels = ['type-feature', 'library', '3.11']
title = 'Allow Python distributors to add custom site install schemes'
updated_at = <Date 2022-02-15.23:11:21.853>
user = 'https://github.com/FFY00'

bugs.python.org fields:

activity = <Date 2022-02-15.23:11:21.853>
actor = 'stefanor'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2021-04-29.16:19:02.410>
creator = 'FFY00'
dependencies = []
files = []
hgrepos = []
issue_num = 43976
keywords = []
message_count = 38.0
messages = ['392326', '392350', '392367', '392391', '392823', '392828', '392832', '392884', '392887', '392941', '392942', '392946', '392948', '392950', '392951', '392984', '392986', '392987', '392989', '392994', '393022', '393041', '393044', '399095', '399127', '399129', '399131', '399132', '399159', '399363', '399364', '399365', '400833', '402339', '402470', '409677', '409808', '411657']
nosy_count = 12.0
nosy_names = ['lemburg', 'jaraco', 'christian.heimes', 'petr.viktorin', 'stefanor', 'steve.dower', 'Frederik Rietdijk', 'hroncok', 'frenzy', 'FFY00', 'jakirkham', 'xrcg']
pr_nums = ['25718']
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue43976'
versions = ['Python 3.11']

[FFY00]

As part of the distutils migration we plan to add a mechanism to let Python distributors to add site install schemes.

Currently, Python distributors are patching distutils to add custom install schemes for their packages. I think most of the reasoning boils down to them wanting to stop Python installers, such as pip, to modify/interfere with their packages.

With the distutils deprecation, and it becoming a 3rd party module, Python distributors can no longer patch it. Because of this, we made distutils use the sysconfig module instead, which fixes the issue at the moment -- Python distributors can now patch sysconfig itself -- but is not a long term solution.
To prevent Python distributors from having to patch implementation details, and have things break unexpectedly, we aim to introduce a system that distributors can use for this purpose.

The idea is that they have a config file, which they can pass to configure, and in that config file they can specify some extra install schemes. These install schemes will get added in sysconfig, and will be loaded in the site module initialization.

In practice, it will look something like this:

config.py

EXTRA_SITE_INSTALL_SCHEMES = {
    'posix_prefix': {
        'stdlib': '{installed_base}/{platlibdir}/python{py_version_short}',
        'platstdlib': '{platbase}/{platlibdir}/python{py_version_short}',
        'purelib': '{base}/lib/python{py_version_short}/vendor-packages',
        'platlib': '{platbase}/{platlibdir}/python{py_version_short}/vendor-packages',
        'include':
            '{installed_base}/include/python{py_version_short}{abiflags}',
        'platinclude':
            '{installed_platbase}/include/python{py_version_short}{abiflags}',
        'scripts': '{base}/bin',
        'data': '{base}',
    },
}

./configure --with-vendor-config=config.py

[zooba]

Any reason this couldn't be in sitecustomize.py? Either by poking values into sysconfig directly (for back-compat) or we train sysconfig to look inside sitecustomize for a well-known name.

[FFY00]

Making sysconfig look at sitecustomize seems like the wrong approach. It is behavior I would never expect, and there are use-cases where I still want the schemes to be present when the site module initialization is disabled.

I would also argue that having this mechanism available will be useful for other things.

[hroncok]

Cross referencing the discussion: https://discuss.python.org/t/mechanism-for-distributors-to-add-site-install-schemes-to-python-installations/8467

[zooba]

Making sysconfig look at sitecustomize seems like the wrong approach.

I mean, you're literally customizing the site, so having it be done from sitecustomize doesn't seem terribly wrong. But I agree, I'd rather see the code in sitecustomize poke paths into sysconfig, rather than the other way around.

The problem then would be that -S bypasses the path configuration entirely, which is likely going to point at non-existent paths. So yeah, for this case you need an override that isn't tied to the site module. Having a similar-but-different mechanism in sysconfig seems fine. I have a *slight* preference for non-executable code, mostly to avoid the risk of import hijacking, but it's only slight.

[FFY00]

FYI, I have change the implementation to split the extra install schemes and extra schemes activated on site. This still makes sense over sitecustomize because we want the packages to be included in site.getsitepackages -- we want the vendor packages to essentially be the same as site-packages.

I have also moved sysconfig._get_preferred_schemes to the vendor config, instead of asking distributors to patch sysconfig -- this is why I prefer having it as executable code, we customize using functions, etc.
https://docs.python.org/3.10/library/sysconfig.html#sysconfig.\_get_preferred_schemes

A config taking advantage of all these mechanisms should look like this:

EXTRA_INSTALL_SCHEMES = {
    'vendor': {
        'stdlib': '{installed_base}/{platlibdir}/python{py_version_short}',
        'platstdlib': '{platbase}/{platlibdir}/python{py_version_short}',
        'purelib': '{base}/lib/python{py_version_short}/vendor-packages',
        'platlib': '{platbase}/{platlibdir}/python{py_version_short}/vendor-packages',
        'include':
            '{installed_base}/include/python{py_version_short}{abiflags}',
        'platinclude':
            '{installed_platbase}/include/python{py_version_short}{abiflags}',
        'scripts': '{base}/bin',
        'data': '{base}',
    },
}

EXTRA_SITE_INSTALL_SCHEMES = [
    'vendor',
]

def get_preferred_schemes(...):
    ...

Do you have any thoughts on this?

[zooba]

Yes, I saw some of the latest changes in the PR.

My biggest concern is with the bare "import _vendor_config", which I'd prefer to have restricted to a fixed location, rather than being influenced by environment variables and other options. We already have an issue with readline being imported from anywhere it can be found.

A native flag to suppress it (i.e. something in sys.flags) could also become important for embedders, though it may matter more at a higher level (i.e. should an embedded CPython *ever* be using sysconfig? Probably not...). I wouldn't add a new flag for it right now, but I feel like sys.flags.isolated should probably imply that this should be ignored.

Though then we hit the issue again that these patches are about changing the "safe default" behaviour, which is what you want to get back when you run with -S or -I. And I'm not totally sure how to resolve this.

So basically, my concerns are:

• don't import arbitrary files
• ensure -S/-I options remain useful (or become even more useful)

[encukou]

Sorry for not getting to this sooner, but 5 days is really tight for such a change.

With -S/-I, It would be great if sys.path only included packages installed as part of the OS, and not those installed by sudo pip. (Or pip --user, but that's covered).

It seems that with the current patch, pip will install into site-packages and there's no way to disable/change site-packages. Is that the case?

[FFY00]

My biggest concern is with the bare "import _vendor_config", which I'd prefer to have restricted to a fixed location, rather than being influenced by environment variables and other options. We already have an issue with readline being imported from anywhere it can be found.

Oh, I share the same concern! Though users could already mess up Python pretty badly by shadowing/overwriting parts of it, so I didn't thought it would be that big of an issue. Is there a way to achieve this while still allowing us to do everything we want?

Sorry for not getting to this sooner, but 5 days is really tight for such a change.

No worries. It was my fault, I should have been more attentive to the Python release timeline.

With -S/-I, It would be great if sys.path only included packages installed as part of the OS, and not those installed by sudo pip. (Or pip --user, but that's covered).

Perhaps we could add an option to enable only vendor site schemes?

It seems that with the current patch, pip will install into site-packages and there's no way to disable/change site-packages. Is that the case?

I mean, there is, though not as straightforward as -S/-I. I was planning on using it to build the distro entrypoint scripts, so that they only include the distro packages.

$ python -S
> site.addsitedir(sysconfig.get_path('purelib', 'vendor'))
> site.addsitedir(sysconfig.get_path('platlib', 'vendor'))

As I mentioned above, we could add a cli flag to do essentially the same.

[zooba]

The best option for restricting the import while still having it be a Python import is to find the file (if it's present in the expected location under sys.whatever), and then use importlib to import it: https://docs.python.org/3/library/importlib.html#importing-a-source-file-directly

I'd rather not have a new option here, I would much prefer "-S" in this context to mean "run Python with only core libraries" and "-s" to mean "run Python with only core and distro libraries" (and neither to mean "run Python with core, distro and user libraries").

That may be a bigger change, but there's enough angst around this issue that we would be better off getting it right this time, even if it changes things, than continuing to preserve the system that people dislike so much.

[zooba]

I'd rather not have a new option here ...

Perhaps what I'm suggesting here is that I don't see any reason for "sudo pip install ..." into a distro-installed Python to ever need to work, and would be quite happy for it to just fail miserably every time (which is already the case for the Windows Store distro of Python).

Admin installed all-user packages is the expert scenario here, and can be as twisted as possible. Pip installed per-user packages and system-tool installed packages are the defaults, and the more easily those can be overridden by a file in the distro, the better.

[malemburg]

On 04.05.2021 22:07, Steve Dower wrote:

Perhaps what I'm suggesting here is that I don't see any reason for "sudo pip install ..." into a distro-installed Python to ever need to work, and would be quite happy for it to just fail miserably every time (which is already the case for the Windows Store distro of Python).

The "pip install" into a root environment approach is the standard way
to setup Docker (and similar) containers, so I think trying to break
this on purpose will not do Python a good service.

The pip warning about this kind of setup which apparently got added
in one of the more recent versions of pip already is causing a lot
of unnecessary noise when building containers and doesn't make Python
look good in that environment.

[zooba]

Would "pip install --user ..." in a Docker container also work, though? Presumably all the filesystem paths are being redirected anyway, so is there a difference?

(My assumption is that "--user" would essentially become the default if you're using the OS provided pip/Python. If you do your own build/install of it then you obviously get "default" behaviour, for better or worse.)

[malemburg]

On 04.05.2021 22:29, Steve Dower wrote:

Would "pip install --user ..." in a Docker container also work, though? Presumably all the filesystem paths are being redirected anyway, so is there a difference?

(My assumption is that "--user" would essentially become the default if you're using the OS provided pip/Python. If you do your own build/install of it then you obviously get "default" behaviour, for better or worse.)

More modern Docker setups run the application itself under a non-root
user, but still install the packages and other dependencies as root.

See eg. Zammad's Dockerfile:
https://github.com/zammad/zammad-docker/blob/master/Dockerfile

Not sure whether that answers your question, though.

It's rather uncommon to install venvs inside Docker containers: one of the
main reasons for using containers is the added isolation, but it doesn't
make a lot of sense to add another layer of isolation inside the container.

"pip install as root" will need to continue to work and thus distros
need to get a way to make sure that it doesn't corrupt the system
installed packages. And perhaps distros can also patch pip to not
output those silly warnings anymore when using the system pip package :-)

Regarding the proposed solution: I'm not sure whether a new configure
option is the right way to go about this. Distros could simply patch
sysconfig.py, since that's the golden source of this information from
Python 3.10 onward.

setuptools' distutils version (and other packages which ship distutils)
will have to use this information instead of the copy which is/was
backed into distutils/sysconfig.py on Python 3.10+

[zooba]

"pip install as root" will need to continue to work and thus distros
need to get a way to make sure that it doesn't corrupt the system
installed packages

Excuse my ignorance, but does "as root" imply that there's no user site-packages directory at all?

I'm not imagining a solution that doesn't require *users* to change their commands, so if they're currently running "sudo pip install" because they need to, but we change it so they shouldn't, then I'm okay with them having to remove the "sudo". (At least for this discussion - we can evaluate transition plans separately.)

And yeah, patching sysconfig.py seems easier. But then, adding a file to the distro is even easier, and if it's easiest for Linux distros to do that via configure than to add a copy step into their build (which is how I'll do it for Windows distros that need it), then I'll leave that to others to decide/implement.

[malemburg]

On 04.05.2021 22:58, Steve Dower wrote:

> "pip install as root" will need to continue to work and thus distros
> need to get a way to make sure that it doesn't corrupt the system
> installed packages

Excuse my ignorance, but does "as root" imply that there's no user site-packages directory at all?

Why should there be no site-packages dir ? All non-core packages get
installed into site-packages (or a similar dir which holds such packages)
by distutils / setuptools.

However, distros usually split this up further into packages which are
managed by the distro packager and ones which are managed by distutils /
setuptools and this is why the install schemes need to be patched.

I'm not imagining a solution that doesn't require *users* to change their commands, so if they're currently running "sudo pip install" because they need to, but we change it so they shouldn't, then I'm okay with them having to remove the "sudo". (At least for this discussion - we can evaluate transition plans separately.)

I'm not sure I understand what you're suggesting.

For Docker, the instructions from the Dockerfile are run as root, so
there is no sudo involved. Whether you use sudo or not or how pip is
invoked is really not relevant for the discussion. The main point is
that the target of the installation is the system installation, not
a local user installation or a venv. That installation layout is what
sysconfig.py defines in the install schemes.

And yeah, patching sysconfig.py seems easier. But then, adding a file to the distro is even easier, and if it's easiest for Linux distros to do that via configure than to add a copy step into their build (which is how I'll do it for Windows distros that need it), then I'll leave that to others to decide/implement.

You mean: put something like...

from _sysconfig_site import *
install_sysconfig_site()

at the end of sysconfig.py and then have distros add a
_sysconfig_site module ?

That would work as well, but details will have to be hashed out, since
this can be abused to hijack the system installation of Python (python
-S would have no effect on this). Patching sysconfig.py is definitely
safer.

[tiran]

"as root" imply that there's no user site-packages directory at all
^^^^^

Steve is talking about user site-packages, not global site-packages directory.

[malemburg]

On 05.05.2021 10:01, Christian Heimes wrote:

> "as root" imply that there's no user site-packages directory at all
^^^^^

Steve is talking about user site-packages, not global site-packages directory.

You mean "pip install --user" as root ? That's not how you typically
install Python packages as root in a Dockerfile, no, but, of course,
even as root, there is the possibility to install into /root/.local/.

The typical Unix way of installing non-system packages is either
into /usr/local, /opt/local or similar variants, not into /usr.
Python itself also defaults to /usr/local when running
"make install". System provided packages normally live
under /usr (or even directly under / for low level tools).

As a root user, I'd assume that "pip install" also installs into
a /usr/local based site-packages dir -- and that's what happens
at least on Debian based OSes. But it can only happen because
the distros patch the install scheme, since this would normally
install into the /usr based site-packages dir for a python binary
living in /usr/bin.

[tiran]

I mean that Steve and you are talking about different things.

Neither Steve nor you or I are are Linux distro packaging experts. I suggest that we listen to the expertise of downstream packagers like Filipe or Miro. They deal with packaging on a daily basis.

By the way you are assuming that all container solutions work like Docker and that all Docker and non-Docker based container solutions allow you to run code as unrestricted, unconfined root. That's a) a incorrect, and b) offtopic for this ticket.

[malemburg]

On 05.05.2021 10:29, Christian Heimes wrote:

I mean that Steve and you are talking about different things.

Could be. I was addressing the point Steve made about not allowing
or making it hard to run "pip install" as root user.

Neither Steve nor you or I are are Linux distro packaging experts. I suggest that we listen to the expertise of downstream packagers like Filipe or Miro. They deal with packaging on a daily basis.

Agreed.

By the way you are assuming that all container solutions work like Docker and that all Docker and non-Docker based container solutions allow you to run code as unrestricted, unconfined root. That's a) a incorrect, and b) offtopic for this ticket.

I gave the Docker example as proof that running "pip install" as
root is a rather common scenario and needs to be supported.

Linux distros have been supporting this for many years and just
because distutils is deprecated should not mean that we no longer
provide ways to support this kind of setup.

BTW: I'm aware that other container solutions work in different ways,
e.g. Podman, LXC, etc. but I have yet to find a solution that doesn't
offer root permissions inside the containers (I'm not talking
about how the container is run in the host system).

[FFY00]

We cannot change how sudo pip install fundamentally works because there are too many people depending on it, and even if we could, this is not the place :P

I think we went a little off-topic here, so let's get back to the discussion.

The best option for restricting the import while still having it be a Python import is to find the file (if it's present in the expected location under sys.whatever), and then use importlib to import it: https://docs.python.org/3/library/importlib.html#importing-a-source-file-directly

Right, though that requires also a new import, importlib, which may not be optimal. Considering that this module is meant to be private and basically all other private importable parts of Python suffer from the same issue, I am finding it hard to justify. If there's enough consensus that this approach would be better, I am more than happy to change the implementation.

I'd rather not have a new option here, I would much prefer "-S" in this context to mean "run Python with only core libraries" and "-s" to mean "run Python with only core and distro libraries" (and neither to mean "run Python with core, distro and user libraries").

I don't think having an option to start Python with only the vendor modules would be *necessary*, though it would certainly be helpful. Among other things, it would be super helpful to be able to tell users to run Python with the -D (made up) option to isolate issues with the vendor modules and the user Python environment.

That may be a bigger change, but there's enough angst around this issue that we would be better off getting it right this time, even if it changes things, than continuing to preserve the system that people dislike so much.

This may be completely wrong for other people, but is my understanding. AFAIK those these issues come from lack of separation between the distro, system and user environments, causing a hell of conflicts and silent module shadowing that neither the system package manager or pip can fix. Almost every time I help people with Python I have to tell them to use a virtual env, which most people aren't expecting, and would likely run into issues had I not suggested it.
Considering that, I think this approach, including the CLI option, would be a step forward. How big would that step be, I am not sure, but probably not *that* big.

But yeah, this is, of course, my experience, and that can vary for other people, so there may be different perspectives here. So I'd very much like to hear other people on this.

[zooba]

> The best option for restricting the import while still having it be a Python
> import is to find the file (if it's present in the expected location under
> sys.whatever), and then use importlib to import it:
> https://docs.python.org/3/library/importlib.html#importing-a-source-file-directly

Right, though that requires also a new import, importlib, which may not be
optimal. Considering that this module is meant to be private and basically all
other private importable parts of Python suffer from the same issue, I am
finding it hard to justify. If there's enough consensus that this approach would
be better, I am more than happy to change the implementation.

Another alternative would be to convert sysconfig into a directory and make the vendor patch a submodule. That's _very slightly_ more impactful for the unpatched case, but only really for scenarios where people are trying to do things they shouldn't. Or we can include the file in all distros and import it earlier (before taking environment variables, etc. into account).

In my opinion, the security implications alone suggest we shouldn't be importing this by name without knowing where it is coming from.

> I'd rather not have a new option here, I would much prefer "-S" in this
> context to mean "run Python with only core libraries" and "-s" to mean "run
> Python with only core and distro libraries" (and neither to mean "run Python
> with core, distro and user libraries").

I don't think having an option to start Python with only the vendor modules
would be *necessary*, though it would certainly be helpful. Among other things,
it would be super helpful to be able to tell users to run Python with the -D
(made up) option to isolate issues with the vendor modules and the user Python
environment.

But the user can already exclude their user-installed packages with -s, right? It's the site-installed packages that would require -S, but that also excludes vendor modules.

Why do we encourage users to install site-wide packages using pip? Why is it such an important scenario for a distro-provided Python to be able to modify its global install using non-distro-provided tools and non-distro-provided packages? What's wrong with saying "install for --user", or else "apt install some-different-python-bundle" first and use that?

(To be clear, I'm framing these as confrontational questions to help my understanding. I'm totally willing to accept an answer of "just because", provided whoever is giving that answer actually "owns" dealing with the fallout.)

[malemburg]

Steve: I think the point of discussing whether "pip install" can
be used to manage system wide packages is moot. It's been like that
for ages, not only for pip, but also for the distutils setup.py install
process and the old Makefile.pre.in approach before that. People
have their reasons, it's what you'd expect to work as a Unix sysadmin
and won't go away anytime soon :-)

So back to the original point...

Filipe: Could you please explain why patching sysconfig.py is not a
long term solution ?

This doesn't involve any changes on the CPython side, is as flexible
as you can get (you can also patch functions defined in sysconfig.py
to do the necessary magic, not only provide a static dict),
doesn't create overhead for Python's startup, works with all the
different command line options for limiting sys.path additions and
avoids security issues with the Python import logic.

It's already clear that sysconfig.py will be the new golden source
for installation related APIs and schemes (perhaps this could be
made even clearer in the docs), so 3rd party packages will adapt
to this once 3.10 is out.

[xrcg]

What is the probability that custom site install schemes will be supported without requiring a patch before distutils is removed?

[FFY00]

I find it very difficult. I would be a bit anxious merging the code this close to the final release, so I imagine the release managers would be much more.

Anyway, this is the current state of this issue from my perspective:

As far as I understand, progress is currently blocked because Matthias thinks this mechanism must allow Debian to replace all related downstream patches.
I disagree with him in the way Debian is patching Python, which IMO is way too intrusive and fragile. Given that this vendor config will essentially shift the burden of supporting these modifications to the Python upstream, I don't think it would be a good idea to allow the modifications Debian wants in this config.
For all other downstreams I have talked to, the proposal seems to be fine, though more testing would definitely be appreciated.

I think that at this point, we could really use a core dev to help push this to the state of being able to get merged, because I don't think I can do that alone.
I have already requested feedback from the community and the technical discussions on the PR went silent, so I do not see what else I can do :/
If needed, I would be available to brief anyone on my understanding of the issues and how this proposal handles that. So, any core dev, please feel free to reach out if you want to help move this forward.

[xrcg]

@FFY00: Thanks for the info.

From what I quickly read, 3.10 will only deprecate distutils, and the removal is scheduled for 3.12. Is that correct? If so, the impending release of 3.10 shouldn’t be a problem, as there are 2 more releases before distutils will be removed.

Or does deprecating distutils cause problems other than warnings?

[FFY00]

Yes, but in the process of deprecating disutils, some internals were changed to stop relying on it. This will cause a lot of downstream patches to break.
I you are not modifying CPython, you don't need to worry, but if you are, then you should have a look and figure out how/where to patch it now.

As far as the public distutils API is concerned, everything will still the same until 3.12, just with deprecation warnings.
Development has moved to [1] and it will eventually be merged into setuptools.

[1] https://github.com/pypa/distutils

[FFY00]

s/still/stay/

[jaraco]

Thanks Filipe for the summary. I was unsure the status.

Even if we grant that Debian is too intrusive in its patching of distutils, it's Python and distutils that are making the change here, so it's not unreasonable for Python to maintain feature parity with the current regime.

It's plausible that you or I could convince Matthias to adapt Debian to a less intrusive approach by adapting the whole Debian ecosystem to a new approach. I suspect achieving that change would be a bit of a challenge.

The only other alternative I see is for distutils to provide some mechanism to enable Debian to achieve its current expectations without patching.

It may not be necessary for CPython to support this mechanism. It may be the case that CPython can support the site install schemes, but that any additional customizations remain a contract between Setuptools/distutils. I'm unsure what implications that would have for other build systems not based on distutils, but presumably it would be up to those systems to support Debian.

I suggest one of three courses here:

1. Convince Debian to drop their customizations.
2. Add support to the CPython customizations to support Debian.
3. Draft support for Setuptools/distutils to support Debian customizations and get confirmation from Debian that satisfies their needs.

Are there any other options? What direction would you like to pursue?

[FFY00]

Hi Jason, thank you.

I already tried 1), but could not convince Matthias on an alternative way to achieve the result that Debian wants. If you want, maybe you and I could try restarting those discussions.

2. is fine, but I am not the one that will me maintaining it, so it is a question for the core devs. I mean, I could commit to working on it and dealing with issues, but I am not a core dev, so it would always need to involve someone else.

3. does not really make sense as of the distutils deprecation. What needs to be patched are the install schemes, which are now outside distutils/setuptools.
   I had hoped that we were able to have the mechanism proposed here in 3.10, so that people could just replace their distutils patches with it, but it was not possible.

I am not sure how to proceed.

[xrcg]

I haven’t read everything in this convo, but it looks like the changes proposed here cover all known downstream users other than Debian.

Is that correct?

Would these changes break Debian’s customizations substantially more than they’ll already be broken by the distutils deprecation refactoring?

If not, and if you don’t want to support the customizations that Debian wants, why not proceed with this enhancement to cover every other case, then deal with Debian later? In the short term, and possible for the long term, Debian can continue to patch the install routine, just in the new appropriate places to patch it. Debian wants something no one else wants, so they have to put in extra effort.

[xrcg]

s/possible/possibly/

[FFY00]

Matthias, can you check if bpo-44982 solves your issues related to the conflicts you were experiencing for the Debian patching?

If so, it would unblock this issue.

I am probably gonna discuss this proposal with the conda-forge folks next week, to get some feedback from the Conda perspective.
I would like to have this unblocked so that we can move forward, as far as I know the only issues are the ones Debian is experiencing about this proposal alone not being able to completely replace the downstream patching.

[jaraco]

In the short term, and possible for the long term, Debian can continue to patch the install routine...

The problem with this approach is Setuptools is attempting to adopt distutils by exposing its own vendored copy of distutils as distutils (top-level name). By doing this, it bypasses the Debian's patching of distutils as found in CPython. Because this bypass behavior breaks distutils for Debian users, the functionality has been disabled (opt-in).

Setuptools would like to be able to present a version of distutils that, unpatched, runs on all the major platforms, and thus make it default.

That won't be possible until Debian can stop relying on its patching of distutils.

[jaraco]

Here's what I propose:

1. In pypa/distutils, add support for honoring the proposed install schemes (based on PR 25718). Merge with Setuptools.
2. Add whatever ugly hacks are needed to pypa/distutils to honor other Debian-specific behaviors (revive Integrate Debian patches pypa/distutils#4 but properly match Debian expectations). Mark these ugly hacks as deprecated.
3. In Debian, Fedora, etc, provide patches that configure the install schemes. Test with latest Setuptools and SETUPTOOLS_USE_DISTUTILS=local.
4. Formalize the install schemes support in CPython (as in PR 25718).
5. Ask Debian to propose more a cleaner interface for Debian-specific needs.

[hroncok]

In Fedora 36+ / Python 3.10+ we now use an install_scheme that looks like this:

'purelib': '{base}/local/lib/python{py_version_short}/site-packages',
'platlib': '{platbase}/local/{platlibdir}/python{py_version_short}/site-packages',
'scripts': '{base}/local/bin',
'data': '{base}/local',
...

We got a user report [1] saying that pip install --root ... --prefix /usr the prefix is not respected at all.

That is, users expect that /usr/local is the prefix, and when they explicitly set it to /usr, the /local/ bit will not be there, while in reality, /local/ is not a part of the prefix, but it is a part of the installation scheme.

I can somehow relate to that assumption.

Now I wonder whether we should have adapted prefix instead of the installation scheme :/

Any ideas on how to approach this problem? I am quite clueless.

[1] https://bugzilla.redhat.com/show_bug.cgi?id=2026979

[jaraco]

I don't have a good answer, but given the title of this issue (which is specifically scoped to site install schemes), I'm tempted to say we should deal with prefixes in a separate, perhaps broader issue, and there address the reported issue (that a user's prefix override isn't honored by the scheme) and maybe more broadly the issue that there's not a design/spec for python installations (and probably there should be).

[FRidh]

In Nixpkgs we install every Python package under a unique prefix, a so-called Nix store path. If we were to use sysconfig for installing packages, then we'd need to be able to dynamically set the paths. This was also discussed as part of the Installer project. https://github.com/pradyunsg/installer/issues/98

We could use a custom scheme, however, we do need to be able to dynamically set a certain variable, e.g. base.

variables = {"installed_base": "$out", "base": "$out", "platbase": "$out", "installed_platbase": "$out"}
# Note there is no `sysconfig.get_default_scheme()`
sysconfig._expand_vars("posix_prefix", variables)

I could imagine we do something like

# check whether we're in a nix build and want to install in a prefix
if "IN_NIX_BUILD" in os.environ:
    base = os.environ["out"]

    scheme = {...}

We'd then need to update the base variable in sysconfig or partially expand our own scheme using this variable.

[hroncok]

I don't have a good answer, but given the title of this issue (which is specifically scoped to site install schemes), I'm tempted to say we should deal with prefixes in a separate, perhaps broader issue, and there address the reported issue (that a user's prefix override isn't honored by the scheme) and maybe more broadly the issue that there's not a design/spec for python installations (and probably there should be).

Technically, I guess we could (instead of redefining the default installation scheme) redefine the default {base} and {platbase} in sysconfig.get_config_vars(). However, I suspect that would require further changes to distutils and pypa/distutils to respect that :/

I'll start a discussion on https://discuss.python.org/ trying to sum up what went wrong with our custom installation scheme and what we want to achieve instead.

[FFY00]

From the feedback I have gathered so far, I think this suggestion is a hard sell for some people due to the performance impact, so I think that's the first thing we have to work on if we want to implement this. There is some duplication in getpath and sysconfig that could be removed, which seems like a good starting point.

[FFY00]

I created #98718 to keep track of sysconfig speed-up efforts.