Allow Python distributors to add custom site install schemes

[FFY00]

BPO	43976
Nosy	@malemburg, @jaraco, @tiran, @encukou, @stefanor, @zooba, @FRidh, @hroncok, @frenzymadness, @FFY00, @jakirkham
PRs	bpo-43976: add vendor config #25718

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2021-04-29.16:19:02.410>
labels = ['type-feature', 'library', '3.11']
title = 'Allow Python distributors to add custom site install schemes'
updated_at = <Date 2022-02-15.23:11:21.853>
user = 'https://github.com/FFY00'

bugs.python.org fields:

activity = <Date 2022-02-15.23:11:21.853>
actor = 'stefanor'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2021-04-29.16:19:02.410>
creator = 'FFY00'
dependencies = []
files = []
hgrepos = []
issue_num = 43976
keywords = []
message_count = 38.0
messages = ['392326', '392350', '392367', '392391', '392823', '392828', '392832', '392884', '392887', '392941', '392942', '392946', '392948', '392950', '392951', '392984', '392986', '392987', '392989', '392994', '393022', '393041', '393044', '399095', '399127', '399129', '399131', '399132', '399159', '399363', '399364', '399365', '400833', '402339', '402470', '409677', '409808', '411657']
nosy_count = 12.0
nosy_names = ['lemburg', 'jaraco', 'christian.heimes', 'petr.viktorin', 'stefanor', 'steve.dower', 'Frederik Rietdijk', 'hroncok', 'frenzy', 'FFY00', 'jakirkham', 'xrcg']
pr_nums = ['25718']
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue43976'
versions = ['Python 3.11']

[FFY00]

As part of the distutils migration we plan to add a mechanism to let Python distributors to add site install schemes.

Currently, Python distributors are patching distutils to add custom install schemes for their packages. I think most of the reasoning boils down to them wanting to stop Python installers, such as pip, to modify/interfere with their packages.

With the distutils deprecation, and it becoming a 3rd party module, Python distributors can no longer patch it. Because of this, we made distutils use the sysconfig module instead, which fixes the issue at the moment -- Python distributors can now patch sysconfig itself -- but is not a long term solution.
To prevent Python distributors from having to patch implementation details, and have things break unexpectedly, we aim to introduce a system that distributors can use for this purpose.

The idea is that they have a config file, which they can pass to configure, and in that config file they can specify some extra install schemes. These install schemes will get added in sysconfig, and will be loaded in the site module initialization.

In practice, it will look something like this:

config.py

EXTRA_SITE_INSTALL_SCHEMES = {
    'posix_prefix': {
        'stdlib': '{installed_base}/{platlibdir}/python{py_version_short}',
        'platstdlib': '{platbase}/{platlibdir}/python{py_version_short}',
        'purelib': '{base}/lib/python{py_version_short}/vendor-packages',
        'platlib': '{platbase}/{platlibdir}/python{py_version_short}/vendor-packages',
        'include':
            '{installed_base}/include/python{py_version_short}{abiflags}',
        'platinclude':
            '{installed_platbase}/include/python{py_version_short}{abiflags}',
        'scripts': '{base}/bin',
        'data': '{base}',
    },
}

./configure --with-vendor-config=config.py

[zooba]

Any reason this couldn't be in sitecustomize.py? Either by poking values into sysconfig directly (for back-compat) or we train sysconfig to look inside sitecustomize for a well-known name.

[FFY00]

Making sysconfig look at sitecustomize seems like the wrong approach. It is behavior I would never expect, and there are use-cases where I still want the schemes to be present when the site module initialization is disabled.

I would also argue that having this mechanism available will be useful for other things.

[hroncok]

Cross referencing the discussion: https://discuss.python.org/t/mechanism-for-distributors-to-add-site-install-schemes-to-python-installations/8467

[zooba]

Making sysconfig look at sitecustomize seems like the wrong approach.

I mean, you're literally customizing the site, so having it be done from sitecustomize doesn't seem terribly wrong. But I agree, I'd rather see the code in sitecustomize poke paths into sysconfig, rather than the other way around.

The problem then would be that -S bypasses the path configuration entirely, which is likely going to point at non-existent paths. So yeah, for this case you need an override that isn't tied to the site module. Having a similar-but-different mechanism in sysconfig seems fine. I have a *slight* preference for non-executable code, mostly to avoid the risk of import hijacking, but it's only slight.

[FFY00]

FYI, I have change the implementation to split the extra install schemes and extra schemes activated on site. This still makes sense over sitecustomize because we want the packages to be included in site.getsitepackages -- we want the vendor packages to essentially be the same as site-packages.

I have also moved sysconfig._get_preferred_schemes to the vendor config, instead of asking distributors to patch sysconfig -- this is why I prefer having it as executable code, we customize using functions, etc.
https://docs.python.org/3.10/library/sysconfig.html#sysconfig.\_get_preferred_schemes

A config taking advantage of all these mechanisms should look like this:

EXTRA_INSTALL_SCHEMES = {
    'vendor': {
        'stdlib': '{installed_base}/{platlibdir}/python{py_version_short}',
        'platstdlib': '{platbase}/{platlibdir}/python{py_version_short}',
        'purelib': '{base}/lib/python{py_version_short}/vendor-packages',
        'platlib': '{platbase}/{platlibdir}/python{py_version_short}/vendor-packages',
        'include':
            '{installed_base}/include/python{py_version_short}{abiflags}',
        'platinclude':
            '{installed_platbase}/include/python{py_version_short}{abiflags}',
        'scripts': '{base}/bin',
        'data': '{base}',
    },
}

EXTRA_SITE_INSTALL_SCHEMES = [
    'vendor',
]

def get_preferred_schemes(...):
    ...

Do you have any thoughts on this?

[zooba]

Yes, I saw some of the latest changes in the PR.

My biggest concern is with the bare "import _vendor_config", which I'd prefer to have restricted to a fixed location, rather than being influenced by environment variables and other options. We already have an issue with readline being imported from anywhere it can be found.

A native flag to suppress it (i.e. something in sys.flags) could also become important for embedders, though it may matter more at a higher level (i.e. should an embedded CPython *ever* be using sysconfig? Probably not...). I wouldn't add a new flag for it right now, but I feel like sys.flags.isolated should probably imply that this should be ignored.

Though then we hit the issue again that these patches are about changing the "safe default" behaviour, which is what you want to get back when you run with -S or -I. And I'm not totally sure how to resolve this.

So basically, my concerns are:

• don't import arbitrary files
• ensure -S/-I options remain useful (or become even more useful)

[encukou]

Sorry for not getting to this sooner, but 5 days is really tight for such a change.

With -S/-I, It would be great if sys.path only included packages installed as part of the OS, and not those installed by sudo pip. (Or pip --user, but that's covered).

It seems that with the current patch, pip will install into site-packages and there's no way to disable/change site-packages. Is that the case?

[FFY00]

My biggest concern is with the bare "import _vendor_config", which I'd prefer to have restricted to a fixed location, rather than being influenced by environment variables and other options. We already have an issue with readline being imported from anywhere it can be found.

Oh, I share the same concern! Though users could already mess up Python pretty badly by shadowing/overwriting parts of it, so I didn't thought it would be that big of an issue. Is there a way to achieve this while still allowing us to do everything we want?

Sorry for not getting to this sooner, but 5 days is really tight for such a change.

No worries. It was my fault, I should have been more attentive to the Python release timeline.

With -S/-I, It would be great if sys.path only included packages installed as part of the OS, and not those installed by sudo pip. (Or pip --user, but that's covered).

Perhaps we could add an option to enable only vendor site schemes?

It seems that with the current patch, pip will install into site-packages and there's no way to disable/change site-packages. Is that the case?

I mean, there is, though not as straightforward as -S/-I. I was planning on using it to build the distro entrypoint scripts, so that they only include the distro packages.

$ python -S
> site.addsitedir(sysconfig.get_path('purelib', 'vendor'))
> site.addsitedir(sysconfig.get_path('platlib', 'vendor'))

As I mentioned above, we could add a cli flag to do essentially the same.

[zooba]

The best option for restricting the import while still having it be a Python import is to find the file (if it's present in the expected location under sys.whatever), and then use importlib to import it: https://docs.python.org/3/library/importlib.html#importing-a-source-file-directly

I'd rather not have a new option here, I would much prefer "-S" in this context to mean "run Python with only core libraries" and "-s" to mean "run Python with only core and distro libraries" (and neither to mean "run Python with core, distro and user libraries").

That may be a bigger change, but there's enough angst around this issue that we would be better off getting it right this time, even if it changes things, than continuing to preserve the system that people dislike so much.