uv pip list / freeze / show --system do not show all system packages on Fedora #2500
Comments
charliermarsh
added
the
compatibility
|
Can you just confirm for me what packages should appear, and where they're located? |
$ pip list
❯ pip list
Package Version
------------------ ---------
argcomplete 2.0.0
Beaker 1.12.1
beautifulsoup4 4.12.3
blivet 3.8.2
blivet-gui 2.5.0
blurb 1.1.0
Brlapi 0.8.5
cffi 1.15.1
cfgv 3.4.0
charset-normalizer 3.2.0
click 8.1.3
colorama 0.4.6
ConfigArgParse 1.7
configobj 5.0.8
cryptography 41.0.7
cupshelpers 1.0
dasbus 1.7
dbus-python 1.3.2
distlib 0.3.7
distro 1.8.0
dnf 4.19.0
fedora-third-party 0.10
file-magic 0.4.0
filelock 3.13.1
fros 1.1
humanize 3.13.1
identify 2.5.33
idna 3.4
iniconfig 2.0.0
langtable 0.0.65
lexid 2021.1006
libcomps 0.1.20
libdnf 0.73.0
looseversion 1.3.0
lxml 4.9.3
Mako 1.2.3
MarkupSafe 2.1.3
nftables 0.1
nodeenv 1.8.0
olefile 0.46
packaging 23.1
Paste 3.5.3
pexpect 4.8.0
pid 2.2.3
pillow 10.2.0
pip 23.2.1
pipx 1.3.3
platformdirs 4.1.0
pluggy 1.3.0
ply 3.11
productmd 1.38
psutil 5.9.5
ptyprocess 0.7.0
pwquality 1.4.5
pycairo 1.25.1
pycparser 2.20
pycrypto 2.6.1
pycups 2.0.1
pycurl 7.45.2
PyGObject 3.46.0
pykickstart 3.48
pyOpenSSL 23.2.0
pyparted 3.13.0
PySocks 1.7.1
python-augeas 1.1.0
python-dateutil 2.8.2
python-meh 0.51
pyudev 0.24.1
PyYAML 6.0.1
requests 2.28.2
requests-file 1.5.1
requests-ftp 0.3.1
rpm 4.19.1.1
ruff 0.2.2
selinux 3.5
sepolicy 3.5
setools 4.4.3
setuptools 67.7.2
simpleline 1.9.0
six 1.16.0
sos 4.6.0
soupsieve 2.5
systemd-python 235
Tempita 0.5.2
terminator 2.1.3
tuna 0.5.11
urllib3 1.26.18
userpath 1.9.1
virtualenv 20.25.0(Note that pip list also list user packages in $ pip list --user
Package Version
-------------- ---------
blurb 1.1.0
cfgv 3.4.0
colorama 0.4.6
ConfigArgParse 1.7
distlib 0.3.7
filelock 3.13.1
identify 2.5.33
iniconfig 2.0.0
lexid 2021.1006
looseversion 1.3.0
nodeenv 1.8.0
pipx 1.3.3
platformdirs 4.1.0
pluggy 1.3.0
ruff 0.2.2
tuna 0.5.11
userpath 1.9.1
virtualenv 20.25.0 |
|
i can probably look into this sometime this week (maybe weekend) unless someone else is already on it :.) |
|
So testing with uv version < 0.1.19 actually reveals the error: Directory So I believe this should be labeled as "bug" not "compatibility". I believe the error is hidden in 0.1.19 above due to #2413 |
|
The error appears all the way in version 0.1.12 which introduced the system flag so this does not appear to be a regression in the original implementation. |
|
If you actually install something with My guess is that |
|
(To summarize, I think the fix here is to iterate over |
|
Indeed, installing something into system packages will install it to $ sudo uv pip install tuna --system
Resolved 1 package in 261ms
Downloaded 1 package in 99ms
Installed 1 package in 3ms
+ tuna==0.5.11
$ uv pip list
Package Version
------- -------
tuna 0.5.11
$ uv pip show tuna
Name: tuna
Version: 0.5.11
Location: /usr/local/lib/python3.12/site-packages
Requires:Here's how the site directories are configured in my Python (Fedora 39) $ python -c "import site;print(site.getsitepackages())"
['/usr/local/lib64/python3.12/site-packages', '/usr/local/lib/python3.12/site-packages', '/usr/lib64/python3.12/site-packages', '/usr/lib/python3.12/site-packages']
Yep, that seems to be the case. |
|
I've just checked that installing a package that already exist in |
|
I think pip would list both in that case, right? |
|
Nope, pip will (correctly imo) only show that which will be actually used when imported in the interpreter. e.g. Things installed in Note that the same logic applies to stuff installed in |
|
I don't really agree with that decision, but good to know, thank you! |
danielhollas
changed the title
|
confirmed making the change below (replace |
|
We may need to return all of sys.path from get_interpreter_info to make this work as expected? |
oh dang, that's a very helpful pointer (i should've looked where these interpreter-specific values were coming from :.)), yah that would do it, let me try that out. |
|
after spending a little too much effort and time i've noticed that this may have already been fixed by some other change...? ( where @danielhollas can you take a look to see if i'm delirious or this has indeed been fixed 🦀 the change i've been working on which uses |
|
@ChannyClaus thanks for the ping. I've tested with |
|
hmm it still seems to pick up the packages? where do you have more details around your setup? it would be ideal if you have a small reproducible example end-to-end |
|
Hmm, strange. I am running directly on Fedora 39, I am wondering if there are some differences between Fedora 39 Docker image and normal install. Also in your case you're running under root, not sure if that makes a difference. I'll try to dig around a bit more. |
|
I failed to make progress here due to lack of any useful logging from tl;dr It would be helpful to me if somebody could add logging to print which paths uv is looking at when running |
maybe the snippet from description in #2636 (comment) would help? (more specifically seems like the issue linked may not be completely green-lit but if/once it's green-lit i could maybe take on it? |
|
Note for future self: Turns out one can install only specific perl modules, and not the whole perl metapackage. I ended up doing: |
danielhollas
changed the title
@ChannyClaus to clarify, this has always worked. :-) In other words, this issue is about uv ignoring Fedora system packages that come with the Fedora installation, which are in a different directory from system packages installed by the user. (I am assuming they are doing this to prevent users from messing up the system python install).
I'd still argue that uv should list all the packages that are available to the Python interpreter. I noticed the following help text for the This is very much the same issue as here, but from the text it is not clear whether this was a conscious design decision or just stating the current limitation. If the former I'd argue to reconsider. As an example, in HPC context, admins may provide a base Python environment with basic scientific packages (numpy et al) compiled specifically for the machine to get the best performance. In this case, it would be quite wrong for uv to ignore these packages and reinstall them inside the venv. |
<!-- Thank you for contributing to uv! To help us out with reviewing, please consider the following: - Does this pull request include a summary of the change? (See below.) - Does this pull request include a descriptive title? - Does this pull request include references to any relevant issues? --> ## Summary likely necessary to resolve #2500 made this a separate PR in an attempt to make the changes as small as possible; let me know if it's preferred to keep them as a single PR. <!-- What's the purpose of the change? What does it do, and why? --> ## Test Plan - edited the test in `interpreter.rs` - tested manually via `println!` ``` $ cargo run --quiet pip show test ["/Users/chankang/Library/Caches/uv/.tmpKzNEPN", "/Users/chankang/.pyenv/versions/3.12.2/lib/python312.zip", "/Users/chankang/.pyenv/versions/3.12.2/lib/python3.12", "/Users/chankang/.pyenv/versions/3.12.2/lib/python3.12/lib-dynload", "/Users/chankang/repos/uv/.venv/lib/python3.12/site-packages"] warning: Package(s) not found for: test chankang@chans-Air ~/repos/uv - (syspath) $ git diff diff --git a/crates/uv-interpreter/src/environment.rs b/crates/uv-interpreter/src/environment.rs index 33b785ce..8ebf0864 100644 --- a/crates/uv-interpreter/src/environment.rs +++ b/crates/uv-interpreter/src/environment.rs @@ -106,6 +106,7 @@ impl PythonEnvironment { /// Some distributions also create symbolic links from `purelib` to `platlib`; in such cases, we /// still deduplicate the entries, returning a single path. pub fn site_packages(&self) -> impl Iterator<Item = &Path> { + println!("{:?}", self.interpreter.sys_path()); if let Some(target) = self.interpreter.target() { Either::Left(std::iter::once(target.root())) } else { chankang@chans-Air ~/repos/uv - (syspath) $ python -c "import sys; print(sys.path)" ['', '/Users/chankang/.pyenv/versions/3.12.2/lib/python312.zip', '/Users/chankang/.pyenv/versions/3.12.2/lib/python3.12', '/Users/chankang/.pyenv/versions/3.12.2/lib/python3.12/lib-dynload', '/Users/chankang/.pyenv/versions/3.12.2/lib/python3.12/site-packages'] chankang@chans-Air ~/repos/uv - (syspath) ``` <!-- How was it tested? -->
|
I just ran into a variant of this problem when working with layered environments that use I initially thought it was working correctly because Between Running Using I don't advise trying to use |
|
I just commented on #4466 as well, but @danielhollas and @ncoghlan's use cases resonate with me as well -- layered (and in my case, redistributable) virtual environments containing heavyweight packages such as torch and tensorflow. Considering that #3500 is merged, is it fair to assume that a solution to this would come at essentially zero performance cost, since |
i think you're right about the performance bit, but the tricky part (at least based on the last time i gave this a try) is to make this change without breaking the existing tests... it's possible some of the breaking tests need to be changed but it's quite risky to make changes to tests especially when it's not obviously correct to do so; additionally, this particular issue seemed to be less pressing / critical compared to some other reported issues. |
|
With regards to the test suite -- could this be as simple as changing the test fixture(s) to create the venvs with (otherwise I definitely see how this can make tests brittle -- if not even non-portable -- assuming that the contents of the interpreter-wide |
Yeah we can't do this.. we'd need to have a fake system interpreter available in the test context. We'd probably only have a small subset of test cases include handling packages in the "system" interpreter — the rest would be fully isolated. This is more likely to cause problems in the test suite when run by downstream packagers rather than here where we have full control over the CI environment. |
|
For the test suite, I'd recommend testing the
If that works as described, a venv with system-site-packages enabled should also work (without the hassle of trying to make tests that enable system-site-packages portable across different Python installations). |
|
I hit this today as I wanted to prepare a demo using uv for an upcoming talk. My use case is using a ubuntu:jammy / python 3.10 based docker image having heavy dependencies pre-installed in the system python. I was hoping to create a |
@ChannyClaus do you happen to have a draft PR/commit ref handy for this? If you have the logic side of the change ready to go, I'd be more than happy to give the test suite a whirl :-) |
things may have changed significantly since this was months ago but the main change necessary was https://github.com/astral-sh/uv/pull/2636/files#diff-f29dcb297374a7babe6be50ec410d36c52cb44c6e9ec5183b4b7d0b96229bd94R94-R109 i believe 🌵 |
|
I can confirm that this issue also exists on Databricks Runtime Images that are based on Ubuntu. The |
Executing
uv pip list --systemdoes not list any packages. Similarly forfreezeandshowsubcommands (I did not test any others).OS: Fedora 39, python 3.12, amd64
uv version: 0.1.21, installed via pipx
The text was updated successfully, but these errors were encountered: