pip install virtualenv embeds absolute path to Python interpreter

[jameshfisher]

Pip version:

$ pip --version
pip 9.0.1 from /usr/local/lib/python2.7/site-packages (python 2.7)

Python version:

$ python --version
Python 2.7.14
$ which python
/usr/local/opt/python@2/libexec/bin/python

Operating system: macOS 10.13.3

Description:

This morning, when I ran virtualenv, I got this:

$ virtualenv
-bash: /usr/local/bin/virtualenv: /usr/local/opt/python/bin/python2.7: bad interpreter: No such file or directory

virtualenv only started behaving this way today. It worked yesterday. It broke because virtualenv is a Python script which embeds the absolute path to a Python interpreter which went away:

$ which virtualenv
/usr/local/bin/virtualenv
$ head -1 /usr/local/bin/virtualenv
#!/usr/local/opt/python/bin/python2.7

It appears that this absolute path is generated dynamically! After pip uninstall virtualenv && pipinstall virtualenv, I get a new absolute path to a new Python interpreter:

$ head -1 /usr/local/bin/virtualenv
#!/usr/local/opt/python@2/bin/python2.7

This behavior is not idiomatic, and causes pip-installed packages to break in confusing ways when the path to the interpreter changes!

The idiomatic behavior is for virtualenv to resolve Python from my PATH at runtime, not at install time. For example, the script could begin with a portable interpreter command, e.g. #!/usr/bin/env python. Why is this not the case?

[pfmoore]

That's correct and by design. Resolving from PATH is a potential source of errors, as the Python interpreter active on your PATH may not even have virtualenv installed (it's perfectly acceptable and supported to run the virtualenv script via its absolute path, without it being on PATH at all).

the absolute path to a Python interpreter which went away

You don't explain how the Python interpreter "went away", but if you uninstall and reinstall a Python installation, you should be uninstalling and reinstalling any packages as well. And moving a Python installation (via OS commands rather than reinstall) isn't supported by Python.

[jameshfisher]

For future visitors, the immediate fix is:

pip uninstall virtualenv && pip install virtualenv

I have since found that my Python interpreter at /usr/local/opt/python/bin/python2.7 went away because it was part of the Homebrew python package, which on 1st March 2018 was upgraded to Python 3.x. Here's a related discussion in Homebrew support.

Apparently, brew considers it acceptable to make breaking changes to absolute paths, but virtualenv does not consider this acceptable. Thus these two tools are not stable when used together.

(Arguably, the root problem is the Python community's decision to release a new programming language under a name which was already taken.)

[pradyunsg]

This is something that has occurred due to a change downstream, by Homebrew, to your system configuration. I don't see anything actionable here from pip's end.

Feel free to close this @jameshfisher! :)

[pradyunsg]

See also: Homebrew/homebrew-core#25060

[jameshfisher]

Homebrew have reversed their decision to make python point to Python 3, which fixes this immediate issue.

I still see it as weird that pip trawls my PATH at install time, instead of following convention; and I don't understand the reasoning for it. But it's clear that this is not going to change, so I'm closing.

[MikeMcQuaid]

Hi, Homebrew lead maintainer here 👋. Obviously it's up to pip how this is handled but in our case we generally appreciate applications who consume our binaries to not realpath their way to the location because every version is installed into a separate prefix. The path to the symlink e.g. /usr/local/opt/python/bin/python or /usr/local/bin/python will be consistent across python upgrades and removals where /usr/local/Cellar/python/3.6.4_3/bin/python will go away if a user runs brew cleanup and will obviously not receive security updates.

Homebrew does all it can to address this by ensuring that applications are exposed to the symlinked paths and rewriting things which refer to the Cellar but obviously we cannot do this for applications outside Homebrew. This is also not something we can change on our end without essentially starting the project again from scratch.

[dstufft]

Hmm, I don't remember calling realpath on hashbang lines, I think we just use the value of sys.executable. I can't dig into that right now and I don't have Python installed from Homebrew, does /usr/local/bin/python -c "import sys; print(sys.executable)" return /usr/local/bin/python or /usr/local/Cellar/python/3.6.4_3/bin/python (or similar).

[pfmoore]

What we (technically distlib, as we just vendor distlib for console script handling) do is put sys.executable into the script. We do this because the script references the Python module(s) that were installed alongside the script, so if we were to use a different Python interpreter, the script wouldn't work anyway, as it would have a different site-packages.

If Homebrew wants to have scripts link to a generic name, they could patch sys.executable to reflect that - but in doing so, they'd need to make sure site-packages was managed appropriately. In the original issue here, even if the shebang line had pointed to the new interpreter, the command still would't have worked, as virtualenv wasn't installed in that interpreter (if it had been, the script would have been overwritten with the new interpreter).

[dentarg]

@dstufft I have python and python@2 installed from Homebrew, here's what it returns

$ brew -v
Homebrew 1.5.10-6-g700a4b4
Homebrew/homebrew-core (git revision 6a709; last commit 2018-03-12)

$ ls -ahl =python
lrwxr-xr-x  1 dentarg  admin    38B Mar 12 10:34 /usr/local/bin/python@ -> ../Cellar/python@2/2.7.14_3/bin/python

$ /usr/local/bin/python -c "import sys; print(sys.executable)"
/usr/local/opt/python@2/bin/python2.7

$ ls -ahl =python3
lrwxr-xr-x  1 dentarg  admin    36B Mar 12 10:33 /usr/local/bin/python3@ -> ../Cellar/python/3.6.4_4/bin/python3

$ /usr/local/bin/python3 -c "import sys; print(sys.executable)"
/usr/local/opt/python/bin/python3.6

[bjay1404]

It look me a long time to stumble upon this! I'm having the same problem with clean installs

ipython
zsh: /usr/local/bin/ipython: bad interpreter: /usr/local/opt/python/bin/python2.7: no such file or directory

[bjay1404]

Looks like this is why

 $ /usr/local/bin/python -c "import sys; print(sys.executable)"
/usr/local/opt/python@2/bin/python2.7

[MikeMcQuaid]

I think we just use the value of sys.executable

Thanks, this is helpful and explains the issues here 👍

[ilovezfs]

Sorry but I don't think it's using sys.executable or that anything has been explained yet. As you can see above, sys.executable is printing the opt path but virtualenvs are actually using the realpath.

bash-3.2$ virtualenv -ppython3 venv
Running virtualenv with interpreter /usr/local/bin/python3
Using base prefix '/usr/local/Cellar/python/3.6.4_4/Frameworks/Python.framework/Versions/3.6'
New python executable in /Users/joe/virtualenvs/venv/bin/python3.6
Also creating executable in /Users/joe/virtualenvs/venv/bin/python
Installing setuptools, pip, wheel...done.
bash-3.2$ ls -l venv/.Python 
lrwxr-xr-x  1 joe  staff  80 Mar 13 01:56 venv/.Python -> /usr/local/Cellar/python/3.6.4_4/Frameworks/Python.framework/Versions/3.6/Python
bash-3.2$ cat venv/lib/python3.6/orig-prefix.txt
/usr/local/Cellar/python/3.6.4_4/Frameworks/Python.framework/Versions/3.6bash-3.2$

but

bash-3.2$ python3 -c "import sys; print(sys.executable)"
/usr/local/opt/python/bin/python3.6

[bumperjeep]

To be clear, I'm having a problem outside of virtualenv

[pradyunsg]

I think we should have a new issue for any further discussions here.

[foresmac]

If anyone finds their way here by searching for the error, there is an easier solution if you keep both Python 2.x and Python 3.x on your system. Homebrew Python 2 keeps your pip installed packages in /usr/local/lib/python2.7/site-packages, and if you installed virtualenv with Python 2 it's still there. So all I did was rewrite the hash-bang in /usr/local/bin/virtualenv like so:

> which python
/usr/local/bin/python

or

> which python2
/usr/local/bin/python2

Update the hash-bang to one of the above values and everything works.

[exipilis]

Kudos to @foresmac . Editing /usr/local/bin/virtualenv file works like a charm.

[LUSUDEV]

Antes de instalar Django vamos a crear nuestro entorno virtual, para ello realizamos el siguiente
comando: mkvirtualenv -p /usr/bin/python2.7 proyecto. Asegúrense de que la
dirección de Python 2.7 en sus sistemas estén correctas.

#mkvirtualenv -p /usr/bin/python2.7 proyecto. listo en mi caso fue estos que me decías que The path /usr/local/bin/python2.7 (from --python=/usr/local/bin/python3.6) does not exist me faltava el puntu al final.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.