Fix install failures that silently pass #5046

matteius · 2022-04-14T03:36:11Z

The issue

Pipenv can fail to install dependencies but the return code doesn't represent this fact.

The fix

This checks at the end of the install routine after it has retried and cleaned up if there are still failed dependencies in the queue. If so it logs an error and exits with error.

The checklist

Associated issue
A news fragment in the news/ directory to describe this fix with the extension .bugfix, .feature, .behavior, .doc. .vendor. or .trivial (this will appear in the release changelog). Use semantic line breaks and name the file after the issue number or the PR #.

…r not and exit with error when there are still failed dependencies

oz123

I think we discussed this click\crayons issue before without a conclusion. Maybe we should consider not adding more of those.

oz123 · 2022-04-14T15:31:56Z

pipenv/core.py

+                crayons.red(
+                    f"Failed to install some dependency or packages.  "
+                    f"The following have failed installation and attempted retry: {failed_list}"
+                )


Why do we keep using click.echo(...crayons.{color}... when click has native color support?
I believe click has a huge user base and good support for Linux and Windows terminals.
pipenvs own crayons has some fixes to issues with crayons (which hasn't seen an update for a while).
However, I think we shoud consider it obsolete and not use it further. These fixes are probably for issues
which don't exist in click.

I'm not versed in click colors and was copying an existing pattern, but I am open to changing it here. I think you were talking about at one point being the one that would tackle a PR to remove crayons?

Yes. I wanted to tackle that. But instead of a big PR, I think we should simply remove this pattern slowly, and avoid it in new code.
Click uses colorama and has easy API:
https://click.palletsprojects.com/en/7.x/utils/#ansi-colors

matteius · 2022-04-17T21:46:03Z

Tested the latest changes with the demo example:

matteius@matteius-VirtualBox:~/pipenv-triage$ sudo docker run --rm -it python:3.9 bash
[sudo] password for matteius: 
root@3523e9fc4f6c:/# apt remove -y gcc
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following packages were automatically installed and are no longer required:
  g++-10 libstdc++-10-dev
Use 'apt autoremove' to remove them.
The following packages will be REMOVED:
  g++ gcc
0 upgraded, 0 newly installed, 2 to remove and 0 not upgraded.
After this operation, 61.4 kB disk space will be freed.
(Reading database ... 23373 files and directories currently installed.)
Removing g++ (4:10.2.1-1) ...
Removing gcc (4:10.2.1-1) ...
root@3523e9fc4f6c:/# pip install git+https://github.com/pypa/pipenv.git@issue-5031-fix-silent-install-failures
Collecting git+https://github.com/pypa/pipenv.git@issue-5031-fix-silent-install-failures
  Cloning https://github.com/pypa/pipenv.git (to revision issue-5031-fix-silent-install-failures) to /tmp/pip-req-build-scjd1uz1
  Running command git clone --filter=blob:none --quiet https://github.com/pypa/pipenv.git /tmp/pip-req-build-scjd1uz1
  Running command git checkout -b issue-5031-fix-silent-install-failures --track origin/issue-5031-fix-silent-install-failures
  Switched to a new branch 'issue-5031-fix-silent-install-failures'
  Branch 'issue-5031-fix-silent-install-failures' set up to track remote branch 'issue-5031-fix-silent-install-failures' from 'origin'.
  Resolved https://github.com/pypa/pipenv.git to commit 6d5f1b0a25ca2244e8017567a52a960dfb1c934a
  Running command git submodule update --init --recursive -q
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Collecting certifi
  Downloading certifi-2021.10.8-py2.py3-none-any.whl (149 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 149.2/149.2 KB 1.1 MB/s eta 0:00:00
Requirement already satisfied: pip>=18.0 in /usr/local/lib/python3.9/site-packages (from pipenv==2022.4.9.dev0) (22.0.4)
Collecting virtualenv-clone>=0.2.5
  Downloading virtualenv_clone-0.5.7-py3-none-any.whl (6.6 kB)
Requirement already satisfied: setuptools>=36.2.1 in /usr/local/lib/python3.9/site-packages (from pipenv==2022.4.9.dev0) (58.1.0)
Collecting virtualenv
  Downloading virtualenv-20.14.1-py2.py3-none-any.whl (8.8 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 8.8/8.8 MB 7.8 MB/s eta 0:00:00
Collecting platformdirs<3,>=2
  Downloading platformdirs-2.5.1-py3-none-any.whl (14 kB)
Collecting filelock<4,>=3.2
  Downloading filelock-3.6.0-py3-none-any.whl (10.0 kB)
Collecting six<2,>=1.9.0
  Downloading six-1.16.0-py2.py3-none-any.whl (11 kB)
Collecting distlib<1,>=0.3.1
  Downloading distlib-0.3.4-py2.py3-none-any.whl (461 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 461.2/461.2 KB 14.7 MB/s eta 0:00:00
Building wheels for collected packages: pipenv
  Building wheel for pipenv (pyproject.toml) ... done
  Created wheel for pipenv: filename=pipenv-2022.4.9.dev0-py2.py3-none-any.whl size=3563993 sha256=222bed184f4fbb5942d1d12bb6071578e5f39fd57a0711c1a0e5c8aee4720f3f
  Stored in directory: /tmp/pip-ephem-wheel-cache-o8qga8o6/wheels/e0/b4/bd/c4c1e011d05e7b7490683aa6b09901882353be06a94fe8a47a
Successfully built pipenv
Installing collected packages: distlib, certifi, virtualenv-clone, six, platformdirs, filelock, virtualenv, pipenv
Successfully installed certifi-2021.10.8 distlib-0.3.4 filelock-3.6.0 pipenv-2022.4.9.dev0 platformdirs-2.5.1 six-1.16.0 virtualenv-20.14.1 virtualenv-clone-0.5.7
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
root@3523e9fc4f6c:/# cat <<EOF > Pipfile
[[source]]
url = "https://pypi.python.org/simple"
verify_ssl = true
name = "pypi"

[packages]
pycocotools = "*"

[dev-packages]

[requires]
python_version = "3.9"
EOF
root@3523e9fc4f6c:/# pipenv lock
Creating a virtualenv for this project...
Pipfile: /Pipfile
Using /usr/local/bin/python3.9 (3.9.12) to create virtualenv...
⠋ Creating virtual environment...created virtual environment CPython3.9.12.final.0-64 in 622ms
  creator CPython3Posix(dest=/root/.local/share/virtualenvs/-x-v5uFv0, clear=False, no_vcs_ignore=False, global=False)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/root/.local/share/virtualenv)
    added seed packages: pip==22.0.4, setuptools==62.1.0, wheel==0.37.1
  activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator

✔ Successfully created virtual environment! 
Virtualenv location: /root/.local/share/virtualenvs/-x-v5uFv0
Locking [dev-packages] dependencies...
Locking [packages] dependencies...
Building requirements...
Resolving dependencies...
✔ Success! 
Updated Pipfile.lock (ebc52c)!
root@3523e9fc4f6c:/# pipenv install
Installing dependencies from Pipfile.lock (ebc52c)...
An error occurred while installing pycocotools==2.0.4 --hash=sha256:2ab586aa389b9657b6d73c2b9a827a3681f8d00f36490c2e8ab05902e3fd9e93! Will try again.
  🐍   ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 11/11 — 00:00:24
Installing initially failed dependencies...
  ☤  ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 1/1 — 00:00:01
Failed to install some dependency or packages.  The following have failed installation and attempted retry: [Requirement(_name='pycocotools', vcs=None, req=NamedRequirement(name='pycocotools', version='==2.0.4', req=Requirement.parse('pycocotools==2.0.4'), extras=[], editable=False, _parsed_line=<Line (editable=False, name=pycocotools, path=None, uri=None, extras=(), markers=None, vcs=None, specifier===2.0.4, pyproject=None, pyproject_requires=None, pyproject_backend=None, ireq=pycocotools==2.0.4)>), markers=None, _specifiers='==2.0.4', index='pypi', editable=False, hashes=frozenset({'sha256:2ab586aa389b9657b6d73c2b9a827a3681f8d00f36490c2e8ab05902e3fd9e93'}), extras=(), abstract_dep=None, _line_instance=<Line (editable=False, name=pycocotools, path=None, uri=None, extras=(), markers=None, vcs=None, specifier===2.0.4, pyproject=None, pyproject_requires=None, pyproject_backend=None, ireq=pycocotools==2.0.4)>, _ireq=None)]
root@3523e9fc4f6c:/# echo $?
1

…e tests run.

oz123 · 2022-04-18T05:24:32Z

It seems to me that pipenv can't install from VCS on Windows. Can you test this assumption?
The errors on all Windows versions are repeating:

The following have failed installation and attempted retry: [Requirement(_name='six', vcs='git', ...

matteius · 2022-04-18T07:11:07Z

It seems to me that pipenv can't install from VCS on Windows. Can you test this assumption? The errors on all Windows versions are repeating:

@oz123 It works fine though without any changes the install commands from the test run fine on windows CLI. The tests themselves initially failed in pytest failed on my Windows because of the spinner, so I committed change that gets them to pass locally in pytest but they still fail in the CI for some unknown reason. The actual error the CI is getting back is meeting the condition if "does not match installed location" in err: both the initial and attempted retry, but this is not reproducible outside of the CI.

matteius · 2022-04-18T11:30:39Z

@oz123 There is a chance that the new pip vendoring would actually fix these windows tests: pypa/pip#9452

matteius · 2022-04-20T22:06:33Z

Well unfortunately those 3 windows tests still fail in the CI and they pass locally in a terminal with these changes. I think the path forward is to mark skipping those 3 tests on windows and open a ticket to revisit getting them to work in the CI. To be clear, the install in the windows CI was always failing but the exit code was 0 because of this bug and fixing it revealed that behavior.

matteius · 2022-04-20T22:13:21Z

Weird, the python 3.7 windows build actually passed in the CI last run, but the other 3 did not ... very odd:

Check at the end of isntall if the failed dependency queue is empty o…

c5c626b

…r not and exit with error when there are still failed dependencies

matteius requested a review from oz123 April 14, 2022 03:36

matteius mentioned this pull request Apr 14, 2022

PEP517 packages can silently fail to install #5031

Closed

Add news fragment.

479c9f3

matteius changed the title ~~Check at the end of isntall if the failed dependency queue is empty o…~~ Check at the end of install if the failed dependency queue is empty o… Apr 14, 2022

matteius changed the title ~~Check at the end of install if the failed dependency queue is empty o…~~ Fix install failures that silently pass Apr 14, 2022

oz123 reviewed Apr 14, 2022

View reviewed changes

matteius added 3 commits April 17, 2022 15:29

Merge branch 'main' into issue-5031-fix-silent-install-failures

eb5a6a9

Fix windows tests that fail because of showing the progress bar.

dce1d50

Address PR feedback about crayons.

6d5f1b0

Special attempt at cleaning up the environment for the CI before thes…

eecae2b

…e tests run.

Special attempt at fixing CI environment for these tests to run.

3902c0b

matteius added 2 commits April 18, 2022 08:28

change this back.

cc909ae

Merge branch 'main' into issue-5031-fix-silent-install-failures

35894ea

Merge branch 'main' into issue-5031-fix-silent-install-failures

7910425

matteius mentioned this pull request Apr 21, 2022

Skipping 3 windows tests in the CI now that we fixed the install return code #5064

Closed

Skip these tests on the windows CI for now.

e5b2830

matteius merged commit 5b3c55d into main Apr 21, 2022

matteius deleted the issue-5031-fix-silent-install-failures branch April 21, 2022 01:52

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix install failures that silently pass #5046

Fix install failures that silently pass #5046

matteius commented Apr 14, 2022 •

edited

Loading

oz123 left a comment

oz123 Apr 14, 2022

matteius Apr 14, 2022

oz123 Apr 14, 2022

matteius commented Apr 17, 2022

oz123 commented Apr 18, 2022

matteius commented Apr 18, 2022

matteius commented Apr 18, 2022

matteius commented Apr 20, 2022

matteius commented Apr 20, 2022

Fix install failures that silently pass #5046

Fix install failures that silently pass #5046

Conversation

matteius commented Apr 14, 2022 • edited Loading

The issue

The fix

The checklist

oz123 left a comment

Choose a reason for hiding this comment

oz123 Apr 14, 2022

Choose a reason for hiding this comment

matteius Apr 14, 2022

Choose a reason for hiding this comment

oz123 Apr 14, 2022

Choose a reason for hiding this comment

matteius commented Apr 17, 2022

oz123 commented Apr 18, 2022

matteius commented Apr 18, 2022

matteius commented Apr 18, 2022

matteius commented Apr 20, 2022

matteius commented Apr 20, 2022

matteius commented Apr 14, 2022 •

edited

Loading