pip list looks at sys.path for listing installed packages

[agoose77]

Description

Currently, pip list will return a list of packages it finds on sys.path

$ pip list                                                                     
Package    Version
---------- -------
pip        22.0.2
setuptools 59.6.0

vs

$ PYTHONPATH="/some/other/venv/site-packages" pip list                                                                     
Package                   Version
------------------------- -----------
absl-py                   1.0.0
anyio                     3.6.1
argon2-cffi               21.3.0
argon2-cffi-bindings      21.2.0
attrs                     21.4.0
awkward                   1.8.0
Babel                     2.10.1
backcall                  0.2.0
beautifulsoup4            4.11.1
black                     22.3.0
bleach                    5.0.0
boost-histogram           1.3.1
certifi                   2021.10.8
cffi                      1.15.0
...

Expected behavior

I can see a benefit in giving users a sense of which packages Python is aware of, but I think the existing behaviour is confusing the responsibility of pip. My expectation is that pip should list the packages installed under its control i.e. site-packages.

There may be good reasons for the existing behaviour, but if not, I have a suggestion:

• pip list should first list all packages under paths that it can manage (i.e uninstall).
• An additional section can be added to list packages that it can't manage, but is aware of (to avoid the inevitable "why can't I uninstall this package" and "why is the package version different to pip list).
• pip should continue to accept additional paths to search, which are included in this section

There are some details that might need to be examined, i.e.

• should pip try and list all packages with the same identifier on the import path, to make it clear that there are shadowed packages? (i.e. search all paths for all packages)
• should we use an additional column for "pip managed" instead of a separate table etc.

pip version

22.0.2

Python version

3.10.4

OS

Ubuntu 22.04

How to Reproduce

1. Create two virtual environments A and B.
2. Install a set of packages into B
3. Run pip list with A's environment, and PYTHONPATH set to include B's site-packages
4. Note the inclusion of packages from B
5. Run `pip uninstall to remove one of B's packages from A's environment (fails)

Output

No response

Code of Conduct

• I agree to follow the PSF Code of Conduct.

[pfmoore]

pip list is working as intended here. I understand what you're saying that there's a certain confusion of purpose between where pip installs and what pip lists, but pip is a (relatively) low-level tool, and explicitly is not an environment manager. It sounds like you're looking for something that's higher level, working with "environments".

Changing how pip list works would risk breaking existing users, so we wouldn't do that lightly. And making pip into an environment manager would involve a lot more than just changing the output of pip list.

Personally, I'd rather see a dedicated environment management tool built on top of pip, instead of expanding pip's scope (which is already large) to cover this as well. (Also, there are already environment managers, one of them may already cover this).

[agoose77]

That was a quick update!

My thought was to only change the non-porcelain API, and have a flag that becomes opt-out over some period? But, I can see the hesitancy here - any pip behaviour change is likely breaking.

I actually use a project manager, but this came up in the context of a user of a compute cluster (which have habits of messing with PYTHONPATH) seeing a different version to the one that they thought they'd installed (which was in fact the case, they'd installed the newer version before the env var was set).

My motivation for saying that this isn't an environment feature is that pip uninstall currently fails to uninstall anything outside of the current environment. To me, that means pip already has some notion of "environment". I'm fairly sure this is not a lack of metadata issue - when passing a different site-packages directory via PYTHONPATH it still complains about removing those packages.

EDIT: I note that this only happens when you try to remote packages whilst inside a virtual environment.

However, you have a much better sense of what the best outcome is here, so I'll leave it up to you / the other maintainers to close if you feel it's out of scope :)

[pfmoore]

My thought was to only change the non-porcelain API

I don't really know what you mean by the "non-porcelain API". Were you meaning the JSON format output? Even so, there's a whole bunch of unanswered questions around what it even means to say whether pip can manage a location (there's --target, --prefix, etc, which muddy the waters here).

I don't think we're likely to do this without a fairly compelling and detailed real-world use case (and no promises even then 🙂). Theoretical arguments about what pip should do, or what makes sense, are likely to just go round in circles forever.

[agoose77]

I don't really know what you mean by the "non-porcelain API".

Ah, apologies - that's a Git term that I'm using to mean non-programmatic e.g. pip list vs pip list --json

I don't think we're likely to do this without a fairly compelling and detailed real-world use case

That's reasonable, let's close this issue then - I can't provide a strong argument that something is breaking - this behaviour is problematic for debugging, but is not breaking anything per se.

[piotr-dobrogost]

@agoose77 You might find #4575 (Adopting "working" scheme for every run) interesting.

[notatallshaw]

this behaviour is problematic for debugging, but is not breaking anything per se.

Particularly when I have an environment manager I call Pip using some form of python -m pip ... or whatever my standard practice for calling Python is. It definitely reduces the amount of confusion I can get in to.

[agoose77]

I call Pip using some form of python -m pip

Yes, I find it a lot easier to default to python -m pip these days as it reduces the number of failure modes by a factor N!