pip install -U does not remember whether a package was installed with --user

[pfmoore]

If you do

pip install --user dist
pip install -U dist

then the upgrade will uninstall the original installation from the user site-packages and install the upgraded version in the system site-packages. It should not do this, but rather should remember where the package being upgraded was located, and reinstall to the same location.

[pradyunsg]

And print a message about the fact that it's doing this?

[piotr-dobrogost]

This would require finding out what scheme (here it's the user scheme) installed package uses. Is this feasible?

I think pip install --user -U dist is a workaround.

[RonnyPfannschmidt]

i think its valid to break there (early) instead of silently fixing user misstakes

[piotr-dobrogost]

It's not so clear this is a user mistake. Why does pip uninstall package installed using user scheme during pip install -U dist while there's no --user option given? This is very bad UX. The only sane behavior would be to operate on packages installed in user scheme only if --user was passed.

[RonnyPfannschmidt]

by break early i mean pip should bail out if install and uninstall locations don't match

the user mistake is not remembering --user
the pip ux failure is still working in unclear conditions

[pradyunsg]

I agree with @RonnyPfannschmidt.

If the user has installed a package with --user space, while upgrading/unistalling the package pip should require a --user. Basically, pip should not fiddle with the user space unless --user is passed.

[piotr-dobrogost]

the user mistake is not remembering --user

And the pip mistake is to uninstall package installed in user scheme although there was no --user option passed to it.

by break early i mean pip should bail out if install and uninstall locations don't match

I'd rather have pip operate on one specific installation scheme (user scheme if --user was passed) not across schemes so that install and uninstall locations would always match.

Basically, pip should not fiddle with the user space unless --user is passed.

Exactly and this is what I described saying The only sane behavior would be to operate on packages installed in user scheme only if --user was passed. I'm not sure @RonnyPfannschmidt agrees with this, however.

[RonnyPfannschmidt]

ignoring one of the locations when working on a specific package means there will be "interesting" effects when working with requirement files

which is why i prefer error out early, the later the user becomes aware of the mismatches,
the worse he/she will get hit with strange surprises

[pradyunsg]

I don't think pip should ignore it.

If the user tries to uninstall a package installed in system or venv scheme, with --user, pip should error out, with message about why it did so.

[pradyunsg]

@RonnyPfannschmidt What exactly do you mean the behavior should be like? Could you please give an example of it?

I'm not sure if I am correctly understanding what you are saying... Do you agree with the behavior in my previous comment?

[RonnyPfannschmidt]

i agree, pip should error out as soon as it sees a missmatch
this is in particular relevant when a requirements file installation finds a situation where bits are in user-site and other bits are in the system site

[pradyunsg]

Ohk. 💡 There seems to be consensus on that behavior them. I have an idea/proposal...

AFAIK, there are 3 possible schemes for packages:

• "system" - for system/global packages (--system)
• "user" - for packages installed in user space (--user)
• "local" - for virtualenv packages (--local)

pip enforces a "working" scheme on every run. Outside a virtualenv, the default working scheme would be "system" (It should really be "user", that's another issue #1668). Inside a virtualenv, the default working scheme should be "local". Passing --system or --user or --local overrides the working scheme.

Only packages in the same scheme as the working scheme can be modified. By modifying, I mean installing or uninstalling a package. Trying to modify a package in a different scheme is not allowed and pip would print a message and error out.

So, modifying a package in system scheme with a "user" working scheme is not allowed. Nor is modifing a package in user scheme with a "system" working scheme. Niether are the other permutations with "local".

I think this results in a pretty simple behaviour model.

Any comments?

(editted)

[piotr-dobrogost]

Yes, that's exactly how I think this should work.

[ncoghlan]

To go back to the original scenario (and assuming the current "system" default scheme) and check I'm understanding the proposal correctly:

$ pip install --user dist
# Successfully installs to user site-packages
$ pip install -U dist
# Error like "Upgrade of 'dist' failed: target scheme (--system) does not match existing installation scheme (--user)"

This would need some logic to understand that user installs shadow system ones (so "pip install --upgrade --user dist" is fine, even if the only current installation is at system level), but the general idea sounds good to me.

[pradyunsg]

This should be fine now -- since we default to --user when we can't write to the system. I'm gonna close this out on the basis that the lack of activity here reflects that this has been fixed.

If users are still seeing this be subtly broken, please file a new issue with a reproducer.