Virtual environments break on Homebrew upgrades due to using a Cellar link

[cjolowicz]

On Homebrew, virtual environments created by uv venv reference the Python installation under Cellar in their interpreter symlink and pyvenv.cfg, which has the full downstream version in its path. These virtual environments break when Homebrew upgrades the respective Python package to the next maintenance release. In recent versions of venv and virtualenv, this issue was resolved by using the stable link under $(brew --prefix)/opt/python@3.x/ instead. For example, on Python 3.11 macOS aarch64 this would be the interpreter in /opt/homebrew/opt/python@3.11/bin.

This shell session demonstrates the problem:

❯ uv venv
❯ readlink .venv/bin/python
/opt/homebrew/Cellar/python@3.11/3.11.7_1/Frameworks/Python.framework/Versions/3.11/bin/python3.11
❯ grep ^home .venv/pyvenv.cfg
home = /opt/homebrew/Cellar/python@3.11/3.11.7_1/Frameworks/Python.framework/Versions/3.11/bin

When Homebrew upgrades its python@3.11 package and cleans up the old installation under Cellar, those references in the virtual environment start to dangle.

For comparison, here's what I get with venv from Homebrew's python@3.11 installation:

❯ python3.11 -m venv .venv
❯ readlink .venv/bin/python
python3.11
❯ readlink .venv/bin/python3.11
/opt/homebrew/opt/python@3.11/bin/python3.11
❯ grep ^home .venv/pyvenv.cfg
home = /opt/homebrew/opt/python@3.11/bin

And with virtualenv:

❯ virtualenv --version
virtualenv 20.25.0 from ~/.local/pipx/venvs/virtualenv/lib/python3.12/site-packages/virtualenv/__init__.py
❯ virtualenv .venv
❯ readlink .venv/bin/python
/opt/homebrew/opt/python@3.12/bin/python3.12
❯ grep ^home .venv/pyvenv.cfg
home = /opt/homebrew/opt/python@3.12/bin

Affected platforms: Linux and macOS with Homebrew Python

❯ uv --version
uv 0.1.4
❯ gcc -dumpmachine
arm64-apple-darwin23.2.0

[charliermarsh]

Do you by any chance have a link to the relevant PRs or issues in venv and/or virtualenv?

[cjolowicz]

I don't, but FWIW venv derives the home key from sys._base_executable, which is the stable path.

[charliermarsh]

Hmm yeah, we're using sys._base_executable as of an open PR, but even that gives me:

❯ python3.8
Python 3.8.18 (default, Aug 24 2023, 19:48:18)
[Clang 15.0.0 (clang-1500.1.0.2.5)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys._base_executable
'/opt/homebrew/bin/python3.8'

[charliermarsh]

The problem is that we're calling canonicalize on the sys.executable.

[charliermarsh]

@konstin -- We may want to revisit this change (#965). I think we should only do this (canonicalize) if we're in a virtual environment?

[charliermarsh]

We may want something like...

• If we're not in a venv, use sys.executable.
• If we are in a venv, resolve symlinks once (not recursively) on sys.executable.

That's closer to virtualenv:

# if we're not in a virtual environment, this is already a system python, so return the original executable
# note we must choose the original and not the pure executable as shim scripts might throw us off
return self.original_executable

[ofek]

I don't know if it's quite relevant but they just fixed a bug by storing the resolved absolute path in the metadata file that is generated for virtual environments: pypa/virtualenv#2682

Also, I know all of you are basically Rust experts, but in case I am reading the source correctly please never ever use canonicalize directly as it is literally (I'm not joking) broken on Windows and will cause all of us bugs and unexpected behavior. Please don't use it. Instead, it's common to use the dunce crate or the normpath crate when you don't want to resolve symlinks. See even Armin talking about it.

[charliermarsh]

Haha. We do use fs::canonicalize but we then strip UNC paths everywhere. (I guess we could use dunce's canonicalize to simplify that process.)

[charliermarsh]

I'd actually expect that virtualenv change to break this case, if I'm reading it correctly:

❯ realpath /opt/homebrew/opt/python@3.8/bin/python3.8
/opt/homebrew/Cellar/python@3.8/3.8.18_2/Frameworks/Python.framework/Versions/3.8/bin/python3.8

So now virtualenv would also use the Cellar path, IIUC?

[ofek]

I don't own a macOS to test that for you unfortunately but that sounds right.

[charliermarsh]

On main, virtualenv uses home = /opt/homebrew/Cellar/python@3.8/3.8.18_2/bin. So, they might have the same problem @cjolowicz?

[charliermarsh]

\cc @gaborbernat -- it seems like virtualenv on main will resolve symlinks for system Pythons, which seems desirable in some cases (hence the issue in virtualenv) but not in others (hence this issue). I can't tell which is "correct" though.

[gaborbernat]

I do not have a good answer here. 😱

[konstin]

We may want to revisit this change (#965). I think we should only do this (canonicalize) if we're in a virtual environment?

Agreed, i think this is better, i'm also thinking about the use case where somebody might have intentional redirects for their python setup.

For the Cellar issue, note that technically different patch versions aren't compatible from a packaging perspective, technically a project could require python_full_version != "3.12.2". In practice projects i've only seen lower bounds for patch version (e.g. >3.8.1), upper or exact bounds would be against how python patch versions are maintained and we're building a more reliable tool by intentionally ignoring this detail.

[minusf]

It would be really nice if this was solved somehow, all my uv venvs break after a minor python update...

Besides the difference in the picked executable symlink as described above, I find it interesting that both tools pick the absolute symlink completely differently:

pip_venv/bin/python@ -> python3.12
pip_venv/bin/python3@ -> python3.12
pip_venv/bin/python3.12@ -> /opt/homebrew/opt/python@3.12/bin/python3.12

uv_venv/bin/python@ -> /opt/homebrew/Cellar/python@3.12/3.12.6/Frameworks/Python.framework/Versions/3.12/bin/python3.12
uv_venv/bin/python3@ -> python
uv_venv/bin/python3.12@ -> python

FWIW my OCD prefers the pip version, the most specific filename has the absolute symlink (python3.12) instead of uv's least specific (python) 😅

In case it helps anyone here is also the difference between the pyvenv.conf's:

--- pip_venv/pyvenv.cfg 2024-09-13 20:18:47.389618788 +0200
+++ uv_venv/pyvenv.cfg  2024-09-13 20:19:00.968279785 +0200
@@ -1,5 +1,6 @@
-home = /opt/homebrew/opt/python@3.12/bin
+home = /opt/homebrew/Cellar/python@3.12/3.12.6/Frameworks/Python.framework/Versions/3.12/bin
+implementation = CPython
+uv = 0.4.9
+version_info = 3.12.6
 include-system-site-packages = false
-version = 3.12.6
-executable = /opt/homebrew/Cellar/python@3.12/3.12.6/Frameworks/Python.framework/Versions/3.12/bin/python3.12
-command = /opt/homebrew/opt/python@3.12/bin/python3.12 -m venv /Volumes/sensitive/src/handmade/0/pip_venv
+relocatable = false

[charliermarsh]

I'm open to changing this since it's been reported multiple times. It might be considered breaking though.

[frostming]

I suggest changing it to resolve symbolic links until a non-venv python is found.

[lespea]

I think at the very least it would be nice to have a flag to force one behavior or the other, but maybe there's a concern of bloating commands with too many switches.

[charliermarsh]

I'm fine to change this but it probably requires 0.5.

[charliermarsh]

I'll look into this as part of v0.5.

[charliermarsh]

So just to summarize current behavior:

• python -m .venv: Does not resolve symlinks (seemingly, even if you're creating a venv from within a venv -- at least sometimes).
• virtualenv: Does fully resolve symlinks (matches uv's behavior). Older versions of virtualenv do not resolve symlinks, but newer versions do, and so suffer from this same problem.
• uv: Fully resolves symlinks.

[charliermarsh]

I don't fully understand the python -m .venv behavior. It may have changed ove rtime. For example, if I create a venv with the Homebrew Python, then create a venv from that venv, I get:

❯ cat .brew1/pyvenv.cfg
home = /opt/homebrew/opt/python@3.9/bin
include-system-site-packages = false
version = 3.9.19

❯ cat .brew2/pyvenv.cfg
home = /Users/crmarsh/workspace/uv/.brew1/bin
include-system-site-packages = false
version = 3.9.19

But if I do the same with my non-Homebrew Python, I get:

❯ cat .venv1/pyvenv.cfg
home = /Users/crmarsh/.local/share/rtx/installs/python/3.12.3/bin
include-system-site-packages = false
version = 3.12.3
executable = /Users/crmarsh/.local/share/rtx/installs/python/3.12.3/bin/python3.12
command = /Users/crmarsh/.local/share/rtx/installs/python/3.12.3/bin/python -m venv /Users/crmarsh/workspace/uv/.venv1

❯ cat .venv2/pyvenv.cfg
home = /Users/crmarsh/.local/share/rtx/installs/python/3.12.3/bin
include-system-site-packages = false
version = 3.12.3
executable = /Users/crmarsh/.local/share/rtx/installs/python/3.12.3/bin/python3.12
command = /Users/crmarsh/workspace/uv/.venv1/bin/python -m venv /Users/crmarsh/workspace/uv/.venv2

[charliermarsh]

Ok, if I use Homebrew's 3.12:

❯ cat .venv12-1/pyvenv.cfg
home = /opt/homebrew/opt/python@3.12/bin
include-system-site-packages = false
version = 3.12.7
executable = /opt/homebrew/Cellar/python@3.12/3.12.7_1/Frameworks/Python.framework/Versions/3.12/bin/python3.12
command = /opt/homebrew/opt/python@3.12/bin/python3.12 -m venv /Users/crmarsh/workspace/uv/.venv12-1

❯ cat .venv12-2/pyvenv.cfg
home = /opt/homebrew/Cellar/python@3.12/3.12.7_1/Frameworks/Python.framework/Versions/3.12/bin
include-system-site-packages = false
version = 3.12.7
executable = /opt/homebrew/Cellar/python@3.12/3.12.7_1/Frameworks/Python.framework/Versions/3.12/bin/python3.12
command = /Users/crmarsh/workspace/uv/.venv12-1/bin/python -m venv /Users/crmarsh/workspace/uv/.venv12-2

So it fully resolves the symlink when inside a virtualenv, and doesn't resolve it when outside a virtualenv.

[charliermarsh]

What we're suggesting is different than all of these: resolve until we see a non-virtualenv. But I think it makes sense.

[charliermarsh]

Ok, I'm guessing that the Homebrew Pythons didn't used to set sys._base_executable? And now it's set to /opt/homebrew/Cellar/python@3.12/3.12.7_1/Frameworks/Python.framework/Versions/3.12/bin/python3.12 so they always use that.

[charliermarsh]

(Gonna put together a table documenting all this.)

[charliermarsh]

Collated here: https://docs.google.com/spreadsheets/d/1Vw5ClYEjgrBJJhQiwa3cCenIA1GbcRyudYN9NwQaEcM/edit?usp=sharing (sorry, the formatting is terrible)

[charliermarsh]

uv's current behavior is closest to the new virtualenv behavior (which also suffers from the problem described in this issue; prior versions did not). The venv behavior is closest to optimal but not quite optimal IMO because it uses sys._base_executable which leads to strange outcomes for the nested virtual environment with Brew.

[charliermarsh]

PR here: #8433

[minusf]

Thank you for looking into this. Out of curiosity, what is the use case of

then create a venv from that venv

Admittedly I am a very vanilla venv user, always use a homebrew python (or now uv) to create single simple venv's per projects, or until uv tool install came along, some handmade ~/usr/venvs/... for running python apps not installed with brew (esp after 3.12 python started enforcing not to abuse the system wide site-packages). btw uv tool install is a real game changer for these and I even uninstalled some brew packages that were lagging a bit behind.

TBH while waiting for this feature to land I was even content just recreating and repopulating the venv's after a python upgrade, it's just so fast from the cache... Although I was a bit surprised that uv venv is a no-questions-asked destructive operation, and an existing .venv means nothing to it 🤣

[charliermarsh]

We refined the behavior a bit in #9723.