Disable upgrades to existing python modules which were not installed via pip

[nehaljwani]

What's the problem this feature will solve?

pip lets me upgrade python modules installed by the system package manager

Describe the solution you'd like

pip should not touch packages not installed by it ;-)

$ rpm -qf /usr/lib64/python2.7/site-packages/nacl
python2-pynacl-1.2.0-2.fc28.x86_64

$ pip freeze
cffi==1.11.5
ply==3.9
pycparser==2.14
PyNaCl==1.2.0
six==1.11.0

$ sudo pip install --upgrade pynacl
...
Successfully installed pycparser-2.18 pynacl-1.2.1

Alternative Solutions

🤔

Additional context

I initially posted it here. For starters, I have opened a PR to make pip_installed_by_pip() function more generic so that it can be used by whoever implements this feature.

[pradyunsg]

For starters, I have opened a PR to make pip_installed_by_pip() function more generic so that it can be used by whoever implements this feature.

I've gone ahead and merged the PR; mostly because it seems like a good improvement on it's own.

[pradyunsg]

This is an interesting idea - on the surface, ISTM that this would make pip's interactions much better when interacting with distro-installed packages and I don't notice any glaring issues here.

I'm definitely not super-familiar with the details here so I'll defer to others for inputs on this. :)

[hroncok]

Also read https://fedoraproject.org/wiki/Changes/Making_sudo_pip_safe for what we attempted in Fedora. There, however are issues currently with sudo pip install -U described in https://bugzilla.redhat.com/show_bug.cgi?id=1550368 and a for review stalled workaround at https://src.fedoraproject.org/rpms/python-pip/pull-request/10

[hroncok]

Also, from the example that seems like Python 2. We only did it for Python 3, bacause we feel like we don't want to kick a dead horse.

[dstufft]

I don't think pip should only touch files installed by pip, files installed by easy_install or any other python level package manager should be perfectly fine to modify.

What I think is the better test here, is if a package is system managed or not. We don't currently have a marker for that, but ideally we can add one and start to use that instead.

[nehaljwani]

How about adding a flag --installer to pip to mark the installer, so that one could do:

pip install <pkg-name> <extra-flags> --installer=rpm/deb/dnf/yum/conda/...

and if not specified, default to pip?

This is similar to https://www.python.org/dev/peps/pep-0376/#installer

[ncoghlan]

I agree with @dstufft that only touching files installed specifically by pip by default would be problematic, since it inhibits migrating to pip from other non-system installers.

What I do like though is the notion of being able to configure pip to be more opinionated about what it's willing to touch, and provide a reasonably helpful error message when refusing to modify things. That way a distro-provided pip can ship with that configuration set, and python3 -m pip --upgrade pip will fail with guidance on what the user should do instead.

I'm thinking along the lines of a python-installers.toml file that can appear alongside any pip.conf with entries like:

[installer.dnf]
modify = false
message = "This is a system managed package, use `dnf` to update it."

(Tangentially related: #5424 is a proposal to provide a way to add an extra "wrapper" identifier to the User Agent info that pip sends to PyPI. If that ends up being pursued, then it may make sense to structure it so that the wrapper info also ends up in the INSTALLER file)

[ChrisBarker-NOAA]

So an issue to be considered:

I don't know how yum, apt, homebrew, etc install python packages, but:

conda-build is kind of a wrapper around existing installation methods. So a conda recipe for a python package generally uses pip to install. And thus a python package installed by conda was, in fact, installed by pip, and we'd like to keep it that may.

So what would be nice is a way to tell pip whether you want it to manage the package in the future:

pip install --not_pip_managed a_package

and then, in the future, if someone did:

pip uninstall a_package

they get a "this package is not managed by pip" message.

This isn't that different than @ncoghlan's idea above, except perhaps conceptually -- the question is not whether pip was used to install it, it's whether pip is supposed to be used to manage it in the future.

[pfmoore]

Just to make a small point here - the distribution metadata format is standardised for precisely this reason - there's nothing stopping a tool like conda simply doing a pip install and then directly modifying the content of the INSTALLER file in the .dist-info directory to reflect that conda owns this distribution.

That's not to say that pip couldn't have an option to do this, just that it's perfectly acceptable to do it manually (or in this case, in conda-build).

[nehaljwani]

Instead of adding post-install hack specifically in conda-build, I would very much like to have an option which pip would provide, so that other package managers could use as well. Combining that along with @ncoghlan's proposal which would require the file to have a [installer.conda] section, and making pip honor it, would be pretty awesome.

[pfmoore]

I don't consider it a "post-install hack", but rather a supported way for installers to record their ownership of an installed package. Would it help if there was a library function (in some packaging library) that let you set the installer for a distribution? In some ways, I consider letting pip set INSTALLER to something other than "pip" to be the hack - needed by tools that want to use pip as a "subroutine", but not required for pip's core functionality.

Anyway, I'm not going to make a big deal of this, I just wanted to remind people that it's 100% possible and supported to do this without needing changes to pip.

[nehaljwani]

Well, when you put it that way, it makes more sense.

Fedora folks have been doing that for pip anyway: https://src.fedoraproject.org/rpms/python-pip/blob/f28/f/python-pip.spec#_360

[ChrisBarker-NOAA]

@pfmoore: thanks, that is helpful -- probably conda should alter than post-install.

but that means conda-build would have to have some (more) specific hackery, or every python package recipe would have to do it -- which would get messy and unreliable.

So it'd be nice if there was pip invocation that would do an "unmanaged" install.

[hroncok]

In Fedora, we mostly use setup.py install in rpm build. We could switch that to pip and do something like: pip install --root ... --installer dnf.
Later, what Nick says would be done. Possibly with an optional --force switch.

[eli-schwartz]

In Arch Linux we also use setup.py install as a matter of course -- there's no real reason to depend on the whole pip stack for this, and 99% of the functionality of pip is either not needed or unwanted behavior for distro packaging (build-time installation of unmet dependencies inside the wrong package?) so we really prefer to do this ourselves.

I guess we could write the INSTALLER file by hand either way (and a pip option would not make this easier for us), but I don't even see how that helps unless either:

• pip hardcodes a list of known "not system package managers" which it feels okay modifying
• pip hardcodes a list of known "system package managers" which it refuses to modify
• some standard string for "a system package manager did it!" is made, and everyone agrees to use that whether they're actually pacman, rpm, dnf, yum, apt, emerge, or whatever... which pip refuses to modify.

Option 1 means pip can't fix/update packages installed via legacy/minimal methods that didn't write the INSTALLER file, options 2 & 3 mean pip will continue to clobber system package managers unless distro packagers are careful (note that Arch Linux at least has a lot of user-created packages, many of which aren't high-quality to begin with, and standardization via packaging documentation can best be described as a work in progress). Option 2 has the additional disadvantage of needing to actually maintain the list.

Generally, system package managers don't operate together to begin with -- they cover lots more than just python, so it's not like one needs to know which "system package manager" it was. And the package manager will anyways be operating off of a strict manifest of files it installed, tracked in a global database.
Is there anything specific which is done with this information at the moment, other than e.g. pip checking to see if it says "pip"? Does this need to be extra-specific?

[ncoghlan]

Option 4: we define a mechanism for system package managers to let pip know the system package manager's name.

We're going to need a configuration system along those lines in order to make distro-specific Linux wheels viable on generally available index servers anyway (see pypa/packaging-problems#69, which I've finally gotten around to updating with a better title, and better reference links)

If we wanted to cover this as well, then instead of making that file binary-compatibility.toml (and then have a separate platform-installers.toml field for this use case), we could instead call it something like pyplatform.toml, and include a "protected-installers" setting, which would be a mapping from installer names that a Python level installer should leave alone to the error message that it should raise when it encounters them.

[pradyunsg]

This was discussed at PyCon US Packaging Mini-Summit. There is a follow up conversation happening at:

https://discuss.python.org/t/playing-nice-with-external-package-managers/1968

[pradyunsg]

This is now covered by https://peps.python.org/pep-0668/.

[uranusjr]

Should we still keep this issue open? PEP 668 implementation is already tracked in #11381.

[hroncok]

FWIW I belive that this already is implemented even without PEP 668, based on missing RECORD (using info from INSTALLER). See e.g. #11631

[szepeviktor]

Was this feature released in v22.3.1?