pipenv install --system fails when python points to Python 2.

[enku]

Issue description

Since 2022.8.15 pipenv install --system will fail on systems that have both Python 2 and Python 3 installed but which python resolves to Python 2. In commit dbea3f5 it switched from using project._which() which uses sys.executable to determine which python to run to using python_project() which, when using --system first checks for python and then python3. So if, e.g. your project/system is using Python 3 which is installed as /usr/local/bin/python3 but your system Python package us Python 2 which is at /usr/bin/python then it will try to execute using Python 2, resulting in

[pipenv.exceptions.InstallError]: File "/git-test/pipenv/patched/pip/__pip-runner__.py", line 21
[pipenv.exceptions.InstallError]:     fullname: str,
[pipenv.exceptions.InstallError]:             ^
[pipenv.exceptions.InstallError]: SyntaxError: invalid syntax

Expected result

Expected result is pipenv install --system installs the dependencies using Python 3. Although I'm not sure it's really a bug since technically it is using the "system" Python. Nevertheless it is a regression.

Actual result

# python3 -m pipenv install  --system --verbose
Installing dependencies from Pipfile.lock (89d195)...
Writing supplied requirement line to temporary file: "dataclasses==0.8; python_version < '3.7' --hash=sha256:8479067f342acf957dc82ec415d355ab5edb7e7646b90dc6e2fd1d96ad084c97 --hash=sha256:0201d89fa866f68c8ebd9d08ee6ff50c0b255f8ec63a71c16fda7af82bb887bf"
Installing 'dataclasses'
$ /usr/bin/python /usr/local/lib/python3.9/site-packages/pipenv/patched/pip/__pip-runner__.py install --verbose --upgrade --require-hashes --no-deps --exists-action=i -r /tmp/pipenv-v0jrsbzs-requirements/pipenv-_x_y9mvb-requirement.txt
Writing supplied requirement line to temporary file: "importlib-metadata==4.8.2; python_version < '3.8' --hash=sha256:53ccfd5c134223e497627b9815d5030edf77d2ed573922f7a0b8f8bb81a1c100 --hash=sha256:75bdec14c397f528724c1bfd9709d660b33a4d2e77387a3358f20b848bb5e5fb"
Installing 'importlib-metadata'
$ /usr/bin/python /usr/local/lib/python3.9/site-packages/pipenv/patched/pip/__pip-runner__.py install --verbose --upgrade --require-hashes --no-deps --exists-action=i -r /tmp/pipenv-v0jrsbzs-requirements/pipenv-y_uqgaka-requirement.txt
Writing supplied requirement line to temporary file: "typing-extensions==4.0.1; python_version < '3.8' --hash=sha256:7f001e5ac290a0c0401508864c7ec868be4e701886d5b573a9528ed3973d9d3b --hash=sha256:4ca091dea149f945ec56afb48dae714f21e8692ef22a395223bcd328961b6a0e"
Installing 'typing-extensions'
$ /usr/bin/python /usr/local/lib/python3.9/site-packages/pipenv/patched/pip/__pip-runner__.py install --verbose --upgrade --require-hashes --no-deps --exists-action=i -r /tmp/pipenv-v0jrsbzs-requirements/pipenv-k8di4opm-requirement.txt
  🐍   ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 3/3 — 00:00:00
File "/usr/local/lib/python3.9/site-packages/pipenv/patched/pip/__pip-runner__.py", line 21
    fullname: str,
            ^
SyntaxError: invalid syntax
An error occurred while installing dataclasses==0.8; python_version < '3.7' --hash=sha256:8479067f342acf957dc82ec415d355ab5edb7e7646b90dc6e2fd1d96ad084c97 --hash=sha256:0201d89fa866f68c8ebd9d08ee6ff50c0b255f8ec63a71c16fda7af82bb887bf! Will try again.
File "/usr/local/lib/python3.9/site-packages/pipenv/patched/pip/__pip-runner__.py", line 21
    fullname: str,
            ^
SyntaxError: invalid syntax
An error occurred while installing importlib-metadata==4.8.2; python_version < '3.8' --hash=sha256:53ccfd5c134223e497627b9815d5030edf77d2ed573922f7a0b8f8bb81a1c100 --hash=sha256:75bdec14c397f528724c1bfd9709d660b33a4d2e77387a3358f20b848bb5e5fb! Will try again.
File "/usr/local/lib/python3.9/site-packages/pipenv/patched/pip/__pip-runner__.py", line 21
    fullname: str,
            ^
SyntaxError: invalid syntax
An error occurred while installing typing-extensions==4.0.1; python_version < '3.8' --hash=sha256:7f001e5ac290a0c0401508864c7ec868be4e701886d5b573a9528ed3973d9d3b --hash=sha256:4ca091dea149f945ec56afb48dae714f21e8692ef22a395223bcd328961b6a0e! Will try again.
Installing initially failed dependencies...
Writing supplied requirement line to temporary file: "dataclasses==0.8; python_version < '3.7' --hash=sha256:8479067f342acf957dc82ec415d355ab5edb7e7646b90dc6e2fd1d96ad084c97 --hash=sha256:0201d89fa866f68c8ebd9d08ee6ff50c0b255f8ec63a71c16fda7af82bb887bf"
Installing 'dataclasses'
$ /usr/bin/python /usr/local/lib/python3.9/site-packages/pipenv/patched/pip/__pip-runner__.py install --verbose --upgrade --require-hashes --no-use-pep517 --no-deps --exists-action=i -r /tmp/pipenv-v0jrsbzs-requirements/pipenv-x1ek7nrg-requirement.txt
Writing supplied requirement line to temporary file: "importlib-metadata==4.8.2; python_version < '3.8' --hash=sha256:53ccfd5c134223e497627b9815d5030edf77d2ed573922f7a0b8f8bb81a1c100 --hash=sha256:75bdec14c397f528724c1bfd9709d660b33a4d2e77387a3358f20b848bb5e5fb"
Installing 'importlib-metadata'
$ /usr/bin/python /usr/local/lib/python3.9/site-packages/pipenv/patched/pip/__pip-runner__.py install --verbose --upgrade --require-hashes --no-use-pep517 --no-deps --exists-action=i -r /tmp/pipenv-v0jrsbzs-requirements/pipenv-iyfa0tgp-requirement.txt
Writing supplied requirement line to temporary file: "typing-extensions==4.0.1; python_version < '3.8' --hash=sha256:7f001e5ac290a0c0401508864c7ec868be4e701886d5b573a9528ed3973d9d3b --hash=sha256:4ca091dea149f945ec56afb48dae714f21e8692ef22a395223bcd328961b6a0e"
Installing 'typing-extensions'
$ /usr/bin/python /usr/local/lib/python3.9/site-packages/pipenv/patched/pip/__pip-runner__.py install --verbose --upgrade --require-hashes --no-use-pep517 --no-deps --exists-action=i -r /tmp/pipenv-v0jrsbzs-requirements/pipenv-jmqhat8h-requirement.txt
File "/usr/local/lib/python3.9/site-packages/pipenv/patched/pip/__pip-runner__.py", line 21
    fullname: str,
            ^
SyntaxError: invalid syntax
[pipenv.exceptions.InstallError]: File "/usr/local/lib/python3.9/site-packages/pipenv/patched/pip/__pip-runner__.py", line 21
[pipenv.exceptions.InstallError]:     fullname: str,
[pipenv.exceptions.InstallError]:             ^
[pipenv.exceptions.InstallError]: SyntaxError: invalid syntax
ERROR: Couldn't install package: dataclasses
 Package installation failed...
/usr/local/lib/python3.9/subprocess.py:1052: ResourceWarning: subprocess 742 is still running
  _warn("subprocess %s is still running" % self.pid,
ResourceWarning: Enable tracemalloc to get the object allocation traceback
sys:1: ResourceWarning: unclosed file <_io.FileIO name=6 mode='rb' closefd=True>
ResourceWarning: Enable tracemalloc to get the object allocation traceback
sys:1: ResourceWarning: unclosed file <_io.FileIO name=9 mode='rb' closefd=True>
ResourceWarning: Enable tracemalloc to get the object allocation traceback
  ☤  ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 2/3 — 00:00:00
/usr/local/lib/python3.9/subprocess.py:1052: ResourceWarning: subprocess 741 is still running
  _warn("subprocess %s is still running" % self.pid,
ResourceWarning: Enable tracemalloc to get the object allocation traceback
sys:1: ResourceWarning: unclosed file <_io.TextIOWrapper name=4 encoding='utf-8'>
ResourceWarning: Enable tracemalloc to get the object allocation traceback
sys:1: ResourceWarning: unclosed file <_io.TextIOWrapper name=7 encoding='utf-8'>
ResourceWarning: Enable tracemalloc to get the object allocation traceback

Steps to replicate

• Prepare a container, e.g. a CentOS container that has Python 2.7 as the system python.
• Install Python 3 into /usr/local but ensure there is no /usr/local/bin/python
• Create a simple Pipfile then pipenv install --system

---

Please run $ pipenv --support, and paste the results here. Don't put backticks (`) around it! The output already contains Markdown formatting.

I can not paste the actual output but please let me know if you still need it and I will try to come up with a trivial reproducer.

[enku]

I haven't tried to see if this breaks anything else, but this simple change worked for me:

diff --git a/pipenv/utils/shell.py b/pipenv/utils/shell.py
index 362290fa..f6e39bd0 100644
--- a/pipenv/utils/shell.py
+++ b/pipenv/utils/shell.py
@@ -445,7 +445,7 @@ def project_python(project, system=False):
     if not system:
         python = project._which("python")
     else:
-        interpreters = [system_which(p) for p in ("python", "python3")]
+        interpreters = [system_which(p) for p in ("python3", "python")]
         interpreters = [i for i in interpreters if i]  # filter out not found interpreters
         python = interpreters[0] if interpreters else None
     if not python:

[matteius]

Thanks for your report @enku -- I was wondering also if putting python3 in the ordering first would solve for this. I agree that there is a subtle case where maybe someone else is expecting python because python3 happens to be some other version of python, but it would be worth opening a PR for that change and we can explore further if it makes the most sense. Let me know @enku If this is something you have time to do, otherwise I'll take it on.

[enku]

@matteius I'll open a PR for this.

[enku]

Ok, in the process of trying to create a PR I'm having an issue running the tests. I'm following the docs/dev/contributing.rst document but I perhaps overlooked something? The steps I have done are:

1. Create a container instance from a fresh python:3.10 image, then from inside the container:
2. Clone the pipenv repo, then from the root of the repo:
3. pip install -e .
4. pipenv install --dev
5. mkdir tests/pypi/fixtures # else the test collector fails
6. pipenv run pytest -x
   Which results in the following:

======================================= test session starts ========================================
platform linux -- Python 3.10.6, pytest-7.1.2, pluggy-1.0.0
rootdir: /root/work/pipenv, configfile: setup.cfg, testpaths: tests
plugins: pypi-0.1.1, cov-3.0.0, timeout-2.1.0, forked-1.4.0, xdist-2.5.0, flaky-3.7.0
collected 302 items                                                                                

tests/integration/test_cli.py .......F

============================================= FAILURES =============================================
____________________________________ test_pipenv_graph_reverse _____________________________________

PipenvInstance = functools.partial(<class 'tests.integration.conftest._PipenvInstance'>, capfd=<_pytest.capture.CaptureFixture object at 0x7f5731ee6ce0>)

    @pytest.mark.cli
    def test_pipenv_graph_reverse(PipenvInstance):
        with PipenvInstance() as p:
            c = p.pipenv('install tablib==0.13.0')
            assert c.returncode == 0
            c = p.pipenv('graph --reverse')
            assert c.returncode == 0
            output = c.stdout
    
            c = p.pipenv('graph --reverse --json')
            assert c.returncode == 1
            assert 'Warning: Using both --reverse and --json together is not supported.' in c.stderr
    
            requests_dependency = [
                ('backports.csv', 'backports.csv'),
                ('odfpy', 'odfpy'),
                ('openpyxl', 'openpyxl>=2.4.0'),
                ('pyyaml', 'pyyaml'),
                ('xlrd', 'xlrd'),
                ('xlwt', 'xlwt'),
            ]
    
            for dep_name, dep_constraint in requests_dependency:
                pat = fr'^[ -]*{dep_name}==[\d.]+'
                dep_match = re.search(pat, output, flags=re.MULTILINE)
                assert dep_match is not None, f'{pat} not found in {output}'
    
                # openpyxl should be indented
                if dep_name == 'openpyxl':
                    openpyxl_dep = re.search(r'^openpyxl', output, flags=re.MULTILINE)
                    assert openpyxl_dep is None, f'openpyxl should not appear at beginning of lines in {output}'
    
                    assert '  - openpyxl==2.5.4 [requires: et-xmlfile]' in output
                else:
                    dep_match = re.search(fr'^[ -]*{dep_name}==[\d.]+$', output, flags=re.MULTILINE)
>                   assert dep_match is not None, f'{dep_name} not found at beginning of line in {output}'
E                   AssertionError: odfpy not found at beginning of line in backports.csv==1.0.7
E                       - tablib==0.13.0 [requires: backports.csv]
E                     defusedxml==0.7.1
E                       - odfpy==1.4.1 [requires: defusedxml]
E                         - tablib==0.13.0 [requires: odfpy]
E                     et-xmlfile==1.1.0
E                       - openpyxl==3.0.10 [requires: et-xmlfile]
E                         - tablib==0.13.0 [requires: openpyxl>=2.4.0]
E                     pip==22.2.2
E                     pyyaml==6.0
E                       - tablib==0.13.0 [requires: pyyaml]
E                     setuptools==63.4.1
E                     wheel==0.37.1
E                     xlrd==2.0.1
E                       - tablib==0.13.0 [requires: xlrd]
E                     xlwt==1.3.0
E                       - tablib==0.13.0 [requires: xlwt]
E                     
E                   assert None is not None

tests/integration/test_cli.py:134: AssertionError
--------------------------------------- Captured stdout call ---------------------------------------
$ pipenv graph --reverse --json

Command failed...
--------------------------------------- Captured stderr call ---------------------------------------
Warning: Using both --reverse and --json together is not supported. Please select one of the two options.

Traceback (most recent call last):
  File "/root/.local/share/virtualenvs/pipenv-VlIzyezL/lib/python3.10/site-packages/click/testing.py", line 408, in invoke
    return_value = cli.main(args=args or (), prog_name=prog_name, **extra)
  File "/root/work/pipenv/pipenv/cli/options.py", line 56, in main
    return super().main(*args, **kwargs, windows_expand_args=False)
  File "/root/work/pipenv/pipenv/vendor/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "/root/work/pipenv/pipenv/vendor/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/root/work/pipenv/pipenv/vendor/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/root/work/pipenv/pipenv/vendor/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/root/work/pipenv/pipenv/vendor/click/decorators.py", line 84, in new_func
    return ctx.invoke(f, obj, *args, **kwargs)
  File "/root/work/pipenv/pipenv/vendor/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/root/work/pipenv/pipenv/cli/command.py", line 580, in graph
    do_graph(state.project, bare=bare, json=json, json_tree=json_tree, reverse=reverse)
  File "/root/work/pipenv/pipenv/core.py", line 2802, in do_graph
    sys.exit(1)
SystemExit: 1

========================================= warnings summary =========================================
../../.local/share/virtualenvs/pipenv-VlIzyezL/lib/python3.10/site-packages/_pytest/config/__init__.py:1129
  /root/.local/share/virtualenvs/pipenv-VlIzyezL/lib/python3.10/site-packages/_pytest/config/__init__.py:1129: PytestAssertRewriteWarning: Module already imported so cannot be rewritten: __editable___pytest_pypi_0_1_1_finder
    self._mark_plugins_for_rewrite(hook)

../../.local/share/virtualenvs/pipenv-VlIzyezL/lib/python3.10/site-packages/_pytest/config/__init__.py:1252
  /root/.local/share/virtualenvs/pipenv-VlIzyezL/lib/python3.10/site-packages/_pytest/config/__init__.py:1252: PytestConfigWarning: Unknown config option: plugins
  
    self._warn_or_fail_if_strict(f"Unknown config option: {key}\n")

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
===================================== short test summary info ======================================
FAILED tests/integration/test_cli.py::test_pipenv_graph_reverse - AssertionError: odfpy not found...
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
============================= 1 failed, 7 passed, 2 warnings in 31.28s =============================

Sorry this is the first time working on this repo. Any help would be appreciated.

[matteius]

@enku I believe remove step 5 and replace it with:

          git submodule sync
          git submodule update --init --recursive

The test runner command can be found in the .github/workflows/ci.yaml for reference, those are the exact steps the github test runner uses.

[matteius]

Just a heads up that there are currently two flaky tests that might fail for you, as they sometimes fail in the main test runner, those are:
FAILED tests/unit/test_utils.py::test_convert_deps_to_pip[deps0-requests] - A...
FAILED tests/unit/test_utils.py::test_convert_deps_to_pip[deps1-requests[socks]]

[enku]

Ahh, cool. I knew I left something out. And thanks for the heads up about the flaky tests.

[matteius]

pipenv, version 2022.8.30 has been released.