Running scripts with /usr/bin/env and 2021.11.5.post0 fails to find correct python

[mgmarino]

Issue description

This is another regression of the recent release. The problem is running scripts using a she-bang + /usr/bin/env python to reference python. For example:

#!/usr/bin/env python

import sys

print(sys.version)

Running this as executable with the recent stable version of pipenv yields what is expected:

> pipenv --version
pipenv, version 2021.5.29
> pipenv run ./test.py
3.7.9 (default, Nov 16 2020, 11:56:37)
[Clang 12.0.0 (clang-1200.0.32.21)]

With the new version, it keeps the "default" environment (here 2.7 on a mac)

> pipenv --version
pipenv, version 2021.11.5.post0
> pipenv run ./test.py
2.7.17 (default, Dec 23 2019, 21:25:34)
[GCC 4.2.1 Compatible Apple LLVM 11.0.0 (clang-1100.0.33.16)]

[danielpoonwj]

I think it might be because pipenv run does not add the virtual env to $PATH.

This was what I got running in a python:3.8-buster docker container.

2021.11.5.post0

$ pipenv run printenv
PATH=/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

2021.5.29

$ pipenv run printenv
PATH=/root/.local/share/virtualenvs/app-4PlAip0Q/bin:/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

Regardless, this definitely looks like a regression and is causing some issues for me too.

[NorthWindH]

I'm seeing another issue here that I think has the same root cause:

• When calling which or shutil.which using pipenv run to find the path of a python command installed using pipenv, that command is not found

Note: when calling the same under pipenv shell, the command does get found. Our CI expects the former so it's breaking our builds for the time being.

Reproduction steps follow:

Using python:3.7-slim image for the following steps:

$ docker image ls python:3.7-slim
REPOSITORY   TAG        IMAGE ID       CREATED      SIZE
python       3.7-slim   3e12d0db6381   9 days ago   120MB

With old pipenv:

root@05a0d5fab621:~# pipenv --version
pipenv, version 2021.5.29
root@05a0d5fab621:~# pipenv install alembic
Creating a virtualenv for this project...
Pipfile: /root/Pipfile
Using /usr/local/bin/python3 (3.7.12) to create virtualenv...
⠹ Creating virtual environment...created virtual environment CPython3.7.12.final.0-64 in 772ms
  creator CPython3Posix(dest=/root/.local/share/virtualenvs/root-BuDEOXnJ, clear=False, no_vcs_ignore=False, global=False)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/root/.local/share/virtualenv)
    added seed packages: pip==21.3.1, setuptools==58.3.0, wheel==0.37.0
  activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator

✔ Successfully created virtual environment!
Virtualenv location: /root/.local/share/virtualenvs/root-BuDEOXnJ
Creating a Pipfile for this project...
Installing alembic...
Adding alembic to Pipfile's [packages]...
✔ Installation Succeeded
Pipfile.lock not found, creating...
Locking [dev-packages] dependencies...
Locking [packages] dependencies...
Building requirements...
Resolving dependencies...
✔ Success!
Updated Pipfile.lock (628381)!
Installing dependencies from Pipfile.lock (628381)...
  🐍   ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 0/0 — 00:00:00
To activate this project's virtualenv, run pipenv shell.
Alternatively, run a command inside the virtualenv with pipenv run.
root@05a0d5fab621:~# pipenv run which alembic
/root/.local/share/virtualenvs/root-BuDEOXnJ/bin/alembic
root@05a0d5fab621:~# pipenv run python -c 'import shutil; print(shutil.which("alembic"))'
/root/.local/share/virtualenvs/root-BuDEOXnJ/bin/alembic

with new pipenv:

root@534ff8a7cfec:~# pipenv --version
pipenv, version 2021.11.5.post0
root@534ff8a7cfec:~# pipenv install alembic
Creating a virtualenv for this project...
Pipfile: /root/Pipfile
Using /usr/local/bin/python (3.7.12) to create virtualenv...
⠙ Creating virtual environment...created virtual environment CPython3.7.12.final.0-64 in 729ms
  creator CPython3Posix(dest=/root/.local/share/virtualenvs/root-BuDEOXnJ, clear=False, no_vcs_ignore=False, global=False)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/root/.local/share/virtualenv)
    added seed packages: pip==21.3.1, setuptools==58.3.0, wheel==0.37.0
  activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator

✔ Successfully created virtual environment!
Virtualenv location: /root/.local/share/virtualenvs/root-BuDEOXnJ
Creating a Pipfile for this project...
Installing alembic...
Adding alembic to Pipfile's [packages]...
✔ Installation Succeeded
Pipfile.lock not found, creating...
Locking [dev-packages] dependencies...
Locking [packages] dependencies...
Building requirements...
Resolving dependencies...
✔ Success!
Updated Pipfile.lock (628381)!
Installing dependencies from Pipfile.lock (628381)...
  🐍   ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 0/0 — 00:00:00
To activate this project's virtualenv, run pipenv shell.
Alternatively, run a command inside the virtualenv with pipenv run.
root@534ff8a7cfec:~# pipenv run which alembic
root@534ff8a7cfec:~# pipenv run python -c 'import shutil; print(shutil.which("alembic"))'
None
root@534ff8a7cfec:~# pipenv shell
Launching subshell in virtual environment...
root@534ff8a7cfec:~#  . /root/.local/share/virtualenvs/root-BuDEOXnJ/bin/activate
((root) ) root@534ff8a7cfec:~# which alembic
/root/.local/share/virtualenvs/root-BuDEOXnJ/bin/alembic
((root) ) root@534ff8a7cfec:~# python -c 'import shutil; print(shutil.which("alembic"))'
/root/.local/share/virtualenvs/root-BuDEOXnJ/bin/alembic

[wjmelements]

We are experiencing the same issue. Seems the PATH is broken for pipenv run.

Your deployment broke our CI. Since this breaking change made it into your release, I want to see integration tests verifying the PATH. @frostming

[stewartmiles]

Seeing the same thing with 2021.11.5 as well. I'm guessing the pipenv shell regression was the reason to release 2021.11.5.post0 :(

$ pipenv run which python
Loading .env environment variables...
/usr/bin/python  # This should point at the venv instead.
$ pipenv shell
Loading .env environment variables...
Traceback (most recent call last):
  File "/home/stewart/.local/bin/pipenv", line 8, in <module>
    sys.exit(cli())
  File "/home/stewart/.local/pipx/venvs/pipenv/lib/python3.8/site-packages/pipenv/vendor/click/core.py", line 1137, in __call__
    return self.main(*args, **kwargs)
  File "/home/stewart/.local/pipx/venvs/pipenv/lib/python3.8/site-packages/pipenv/vendor/click/core.py", line 1062, in main
    rv = self.invoke(ctx)
  File "/home/stewart/.local/pipx/venvs/pipenv/lib/python3.8/site-packages/pipenv/vendor/click/core.py", line 1668, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/stewart/.local/pipx/venvs/pipenv/lib/python3.8/site-packages/pipenv/vendor/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/stewart/.local/pipx/venvs/pipenv/lib/python3.8/site-packages/pipenv/vendor/click/core.py", line 763, in invoke
    return __callback(*args, **kwargs)
  File "/home/stewart/.local/pipx/venvs/pipenv/lib/python3.8/site-packages/pipenv/vendor/click/decorators.py", line 84, in new_func
    return ctx.invoke(f, obj, *args, **kwargs)
  File "/home/stewart/.local/pipx/venvs/pipenv/lib/python3.8/site-packages/pipenv/vendor/click/core.py", line 763, in invoke
    return __callback(*args, **kwargs)
  File "/home/stewart/.local/pipx/venvs/pipenv/lib/python3.8/site-packages/pipenv/cli/command.py", line 419, in shell
    do_shell(
  File "/home/stewart/.local/pipx/venvs/pipenv/lib/python3.8/site-packages/pipenv/core.py", line 2309, in do_shell
    shell = choose_shell()
TypeError: choose_shell() missing 1 required positional argument: 'project'

[mdekstrand]

This also broke my workflow, using Invoke to call other scripts in the virtualenv (specifically, jb, the jupyter-book build script) using c.run. invoke would run, but couldn't execute jb.

I have worked around by locking to pipenv 2021.5.29 in my CI pipeline.