Adopting "working" scheme for every run

[pradyunsg]

Just carrying over my idea in #1056 (comment), for proper dedicated discussion.

AFAIK, there are 3 possible schemes for packages:

• "system" - for system/global packages (--system)
• "user" - for packages installed in user space (--user)
• "local" - for virtualenv packages (--local)

pip enforces a "working" scheme on every run. Outside a virtualenv, the default working scheme would be "system" (It should really be "user", that's another issue #1668). Inside a virtualenv, the default working scheme should be "local". Passing --system or --user or --local overrides the working scheme.

Only packages in the same scheme as the working scheme can be modified. By modifying, I mean installing or uninstalling a package. Trying to modify a package in a different scheme is not allowed and pip would print a message and error out.

So, modifying a package in system scheme with a "user" working scheme is not allowed. Nor is modifing a package in user scheme with a "system" working scheme. Niether are the other permutations with "local".

I think this results in a pretty simple behaviour model.

As @ncoghlan pointed out, this would need some logic to understand that user installs shadow system ones. Also, I would slightly change this to spell it like --scheme {global,user,venv} because I like how this signifies exclusiveness of the behaviours better than plain flags do and is consistent with some internal configuration stuff.

Additionally, while doing this:

• preparatory work of Default to --user  #1668 can also be done, meaning that defaulting to user would just be flipping the default (something like what we did for --upgrade-strategy).
• addresses pip install -U does not remember whether a package was installed with --user #1056 by requiring the user to specify the scheme that they are working with.

/ping @dstufft @pfmoore

[pradyunsg]

I would slightly change this to spell it like --scheme {global,user,venv}

Oh, and this would let you set a default scheme for yourself using the configuration toolchain.

[pradyunsg]

@dstufft @pfmoore I would like to know what you think about this. :)

[pfmoore]

In principle this sounds like a reasonable thing to do - but I'm afraid I've got some personal priorities at the moment that mean I don't really have time to think through the implications.

So count me as in favour in principle, but not really able to provide a detailed review at the moment, sorry.

[pradyunsg]

Cool. Thanks! :)

[ncoghlan]

Just noting that I briefly thought that --scheme may not be a good name for this option, due to the potential collision with the concept of installation schemes in sysconfig: https://docs.python.org/3/library/sysconfig.html#installation-paths

However, I subsequently realised that these uses are actually the same use case - the new pip level option is just a helper to select the desired scheme without having to specify the exact platform appropriate scheme name as defined in sysconfig.

[pradyunsg]

Linking to #2418 since I somehow always forget there was an attempt at making --user behaviour default.

---

concept of installation schemes in sysconfig

This is, actually, nice that the names here and there match. :)

[tuukkamustonen]

This came up in #4809 where I suggested being able to pip list only packges related to one scope (now it combines both system and user packages, making it impossible to say which package comes from where).

[pradyunsg]

I've gone ahead and made a PR for this -- #4871.

Thoughts @dstufft, @xavfernandez?

Maybe Barry Warsaw (his name's on the Debian patch to pip; #1668 would be fixed as a part of that PR) should be pinged for this discussion?

[piotr-dobrogost]

How does --target mode of operation relate to schemes? It feels like --target essentially is another type of scheme and treating it as such would potentially solve many bugs pertaining to --target option.

[nanonyme]

I personally think that fallback logic is too complicated. If you're operating under a badly configured distro, use virtualenv.

[ssbarnea]

@nanonyme The fallback has nothing to do with what you call "badly" configured distro. Here is a very simple use case: you have a bash script that call "pip install foo", which is needed for testing you code. You want to make this script usable regardless if user is inside an virtualenv or not.

The script could be called by tox so it would be inside a virtualenv, it may be called by user outside a virtualenv or by user after the activated a virtualenv.

All these use cases are not only valid but also wild spread. At this moment it requires adding a lot of extra logic inside that bash script in order to detect virtualenv presence and decide which params to give to pip.

I gave the bash script just as an example but in practice users may not even have a script and only a configuration line that accepts a "command to run", something quite common on CI systems (see travis.ini). Those CI systems may run that code inside or outside a virtualenv.

If pip is not able to detect and make use of virtualenv, it make any usage of it much harder because the user would be forced to write wrappers around that code in order to make it work in various contexts. Writing these wrappers is not even possible in some cases (or just hard and ugly due to the need to cope with multiple levels or quoting in order to use bash to implement that missing logic).

[nanonyme]

Basically with above you have

1. Explicit global, ignore virtualenv
2. Explicit user, ignore virtualenv
3. Explicit local, local set to global, virtualenv with fallback to global
4. Explicit local, local set to user, virtualenv with fallback to user

The fallback sequences are far simpler this way

[chrahunt]

Not sure if I missed it, how would this interact with the set of packages available for import in a setup.py-based install, or a PEP 517 install using --no-build-isolation?

Also, is the assumption that for determining whether a dependency is already installed we'd have some of these schemes "inherit" the packages from the packages that would be available in another? Like

• user considers packages in system
• local considers packages in user and system only if system site packages is enabled
• system doesn't consider any others
• target doesn't consider any others (or it may be configurable -I think we've seen use cases going both ways)

[pradyunsg]

No changes on that front. This would only affect how a package is installed / unpacked, not how it's built.

Things that are currently importable at build time, would stay importable -- basically anything that's on sys.path when running.

[pradyunsg]

Poking here for feedback on #7164.

[illume]

It looks like @takluyver in #7164 (comment) has a nice plan for stopping some of the damage of defaulting to write to system wide packages. #7002 That should hopefully stop much of the damage experienced by newbies. Thank you, thankyou, thanksssss to all involved! 🎈🎉

[pradyunsg]

(Re?)iterating that this change is still relevant in a post #7002 world, since this is an internal refactoring-related issue in pip.

I'd expect us to start by making refactors to decouple the scheme from the various parts of our codebase (like @chrahunt has started!) and once that's completely/fairly done; we'd start working on the user-facing changes.

[robeke]

I agree this will hopefully improve the current inconsistent behavior. For example using Python 3.11.3 venv with pip 22.3.1 on Linux, I am able to install a package with the --user option, but if I attempt to uninstall the same package using the same venv/pip, it refuses indicating package is "outside environment" and "Can't uninstall 'mypackage'. No files were found to uninstall." I would expect the uninstall to work given the install was permitted.

[hmkim]

If you have a sudo account, use this command.
sudo dnf remove python3-requests

For example, Amazon Linux 2023 have this issue.