Install uses wrong package location when a secondary Python installation is declared as venv

[adebrecht661]

On Ubuntu 22.04, with Python installed to ~/.local/bin/python through a symlink (-> /usr/bin/python -> /usr/bin/python3.11) and VIRTUAL_ENV=~/.local/:

With uv==0.1.11, uv pip install installs to ~/.local/lib/python3.11/site-packages. This makes sense.

With uv==0.1.12, uv pip install attempts to install to /usr/local/lib/python3.11/dist-packages (and fails because of insufficient permissions). I would expect this to still attempt to install to ~/.local, since that was declared as the virtual environment.

I believe #2353, #2059, #1584, #1526 are related (e.g. the --user flag would at least resolve this issue), but it seems like this may be a separate problem with the order of resolution of possible installation locations; that is, I'd expect uv to attempt to install to a virtual environment if one is declared, even if the virtual environment eventually resolves to a system(ish) Python installation.

[charliermarsh]

I'm not sure on this one... I think it'd be error-prone to treat ~/.local/bin/python as a virtual environment just because it's set as VIRTUAL_ENV. There's just no guarantee at all (AFAICT?) that ~/.local/lib/python3.11/site-packages would actually be on sys.path at runtime. So we'd end up installing packages that aren't available to the user.

[charliermarsh]

A virtual environment has a more "formal" definition (AFAIK), which is, a Python interpreter in which sys.prefix and sys.base_prefix are not equivalent.

[charliermarsh]

I think this is reasonably solved by creating a real virtual environment, or using --user once it's supported, but I don't think it's tenable to make assumptions about the paths based on VIRTUAL_ENV.

[adebrecht661]

There's just no guarantee at all (AFAICT?) that ~/.local/lib/python3.11/site-packages would actually be on sys.path at runtime.

Digging through the docs, I believe this is guaranteed for "UNIX and non-framework MacOS builds" (under the condition that ENABLE_USER_SITE is true), but not because of anything that could be relied upon in the case of a more general VIRTUAL_ENV path. Specifically, this relies on the fact that the user site-packages will be added to sys.path if it exists. But if, for example, VIRTUAL_ENV=~/.foo instead, ~/.foo/lib/python3.11/site-packages would not be on the path.

I agree this is solved by --user, so this can be closed in favor of that issue.

I will say that while I understand wanting to push users away from modifying the system Python, there are a lot of ways to install Python that are global but not the system Python, and passing both --user and --system in the command to get pip's default behavior seems... counterintuitive. I'm not sure if there's a straightforward resolution to this, though there is already also --break-system-packages.

[charliermarsh]

I think this would work now with --prefix ~/.local?