pick up site packages from the interpreter's sys.path

[ChannyClaus]

Summary

Resolves #2500

This change got way bigger and more complex than initially thought 😭 (happy to break up into multiple smaller PRs if preferred!) - the main changes are as follows:

• Expose sys.path from the chosen interpreter to be able to pick up more import paths beyond purelib / platlib
• install from uv-dispatch to take an additional argument reinstall so that venv creation with --seed can always force reinstall (this seemed the least amount of change required to not break the behavior of --seed). This was necessary since uv will think pip already exists with this change (in the base interpreter) and would skip installing it for the venv.
• (WIP) uv pip install with reinstall option seems to pick up site package paths multiple times

$ cargo run pip install homeassistant --reinstall --cache-dir chantest --verbose |& grep site
DEBUG Site packages: ["/Users/chan.kang/repos/.venv/lib/python3.12/site-packages"]
DEBUG Site packages: ["/Users/chan.kang/repos/uv/chantest/.tmpdROCqV/.venv/lib/python3.12/site-packages", "/Users/chan.kang/repos/.venv/lib/python3.12/site-packages"]

probably it should be done once for the lifetime of a single command invocation? still looking into it.

Test Plan

$ docker run -v /Users/chankang/repos/uv:/uv  $(docker build -q .)
Package    Version  Location                            Installer
---------- -------- ----------------------------------- ---------
dnf        4.19.0   /usr/lib/python3.12/site-packages
libcomps   0.1.20   /usr/lib64/python3.12/site-packages
libdnf     0.73.0   /usr/lib64/python3.12/site-packages
pip        23.2.1   /usr/lib/python3.12/site-packages   rpm
pyparsing  3.0.9    /usr/lib/python3.12/site-packages   rpm
rpm        4.19.1.1 /usr/lib64/python3.12/site-packages
setuptools 67.7.2   /usr/lib/python3.12/site-packages   rpm
Resolved 5 packages in 700ms
Downloaded 5 packages in 92ms
Installed 5 packages in 21ms
 + certifi==2024.2.2
 + charset-normalizer==3.3.2
 + idna==3.6
 + requests==2.31.0
 + urllib3==2.2.1
Package    Version
---------- --------
certifi    2024.2.2
dnf        4.19.0
idna       3.6
libdnf     0.73.0
pip        23.2.1
pyparsing  3.0.9
requests   2.31.0
setuptools 67.7.2
urllib3    2.2.1

where

$ cat Dockerfile 
FROM fedora:39

RUN yum update -y
RUN yum install pip python cmake perl rust cargo -y

COPY reproduce.sh .
ENTRYPOINT [ "bash", "reproduce.sh" ]

$ cat reproduce.sh 
pip list -v

cd uv && cargo run --quiet pip install requests --system && cargo run --quiet pip list --system

[ChannyClaus]

fixing CI and probably will add tests (it's turning out to be kinda non-trivial hmm, still looking though), should be done sometime today or tomorrow.

[hauntsaninja]

is it desirable for cwd to be in here? (if not, it should be removed, accounting for sys.flags.safe_path)

[ChannyClaus]

probably will leave this value as is so that it still equals to sys.path (it may be confusing if it's called sys_path but the value returned isn't sys.path) - but added filtering via suffix site/dist packages where it's used.

[ChannyClaus]

taking longer than expected :.( - down to a couple of failing tests (having trouble reproducing it locally atm) + system setup jobs failing for windows, will continue to look.

[ChannyClaus]

@zanieb do you happen to know why incompatible_python_compatible_override_available_no_wheels might be failing here? it seems to have succeeded for macos after rebase, but the identical code on my fork fails here

also on CI machines i've noticed if i run it multiple times, it only fails the first time and succeeds in all subsequent runs... 🤔

[ChannyClaus]

😆 ok after rebasing this time it seems like the only failures are windows; i'll try to get these resolved this week.

[zanieb]

What kind of problems are you having with the test suite? Have you merged main lately? We've recently improved the test snapshot filtering (i.e. #2678)

[ChannyClaus]

What kind of problems are you having with the test suite? Have you merged main lately? We've recently improved the test snapshot filtering (i.e. #2678)

the most cryptic test started succeeding after rebasing a couple of times this week 🥲 i think i should be good once i figure out this windows issue (seemed like it can be resolved with something similar to fs::canonicalize)

thank you!

[ChannyClaus]

pip installed via yum install python3-pip has RECORD missing, which causes issues described in pypa/pip#11631 (this issue can be reproduced pulling the docker image used and running the command here)

[ChannyClaus]

closing since this PR is too stale at this point