Handle invalidation of corrupt cache files

[abn]

Poetry
Version: 1.1.0a3
Python:  3.8.3

Virtualenv
Python:         3.8.3
Implementation: CPython
Path:           /path/to/project/.venv
Valid:          True

System
Platform: linux
OS:       posix
Python:   /usr

Issue

In cases where the a download fails, invalid or incomplete content downloaded, there seems to be a scenario where the cache file exists but was not cleaned up. Causing any following attempts to fail with an unpack error inside the pip subprocess. The below output is from an issue that happened when the underlying network changed during initial install and the process was killed. Then an install command was issued again, causing the issue.

  EnvCommandError

  Command ['/path/to/project/.venv/bin/python', '-m', 'pip', 'install', '--no-deps', 'file:///home/user/.cache/pypoetry/artifacts/c8/9d/8a/0ac7b9700197ce07d35f1f4db4c2afa79c83149e7ad6fafd4e82aa1143/traitlets-4.3.3-py2.py3-none-any.whl'] errored with the following return code 2, and output: 
  Processing /home/user/.cache/pypoetry/artifacts/c8/9d/8a/0ac7b9700197ce07d35f1f4db4c2afa79c83149e7ad6fafd4e82aa1143/traitlets-4.3.3-py2.py3-none-any.whl
  ERROR: Exception:
  Traceback (most recent call last):
    File "/path/to/project/.venv/lib64/python3.8/site-packages/pip/_internal/cli/base_command.py", line 153, in _main
      status = self.run(options, args)
    File "/path/to/project/.venv/lib64/python3.8/site-packages/pip/_internal/commands/install.py", line 401, in run
      resolver.resolve(requirement_set)
    File "/path/to/project/.venv/lib64/python3.8/site-packages/pip/_internal/legacy_resolve.py", line 202, in resolve
      self._resolve_one(requirement_set, req)
    File "/path/to/project/.venv/lib64/python3.8/site-packages/pip/_internal/legacy_resolve.py", line 368, in _resolve_one
      abstract_dist = self._get_abstract_dist_for(req_to_install)
    File "/path/to/project/.venv/lib64/python3.8/site-packages/pip/_internal/legacy_resolve.py", line 315, in _get_abstract_dist_for
      abstract_dist = self.preparer.prepare_linked_requirement(
    File "/path/to/project/.venv/lib64/python3.8/site-packages/pip/_internal/operations/prepare.py", line 191, in prepare_linked_requirement
      unpack_url(
    File "/path/to/project/.venv/lib64/python3.8/site-packages/pip/_internal/download.py", line 465, in unpack_url
      unpack_file_url(link, location, download_dir, hashes=hashes)
    File "/path/to/project/.venv/lib64/python3.8/site-packages/pip/_internal/download.py", line 429, in unpack_file_url
      unpack_file(from_path, location, content_type)
    File "/path/to/project/.venv/lib64/python3.8/site-packages/pip/_internal/utils/unpacking.py", line 249, in unpack_file
      unzip_file(
    File "/path/to/project/.venv/lib64/python3.8/site-packages/pip/_internal/utils/unpacking.py", line 114, in unzip_file
      zip = zipfile.ZipFile(zipfp, allowZip64=True)
    File "/usr/lib64/python3.8/zipfile.py", line 1268, in __init__
      self._RealGetContents()
    File "/usr/lib64/python3.8/zipfile.py", line 1335, in _RealGetContents
      raise BadZipFile("File is not a zip file")
  zipfile.BadZipFile: File is not a zip file
  WARNING: You are using pip version 19.3.1; however, version 20.1.1 is available.
  You should consider upgrading via the 'pip install --upgrade pip' command.
  

  at ~/.local/lib/python3.8/site-packages/poetry/utils/env.py:937 in _run
       933│                 output = subprocess.check_output(
       934│                     cmd, stderr=subprocess.STDOUT, **kwargs
       935│                 )
       936│         except CalledProcessError as e:
    →  937│             raise EnvCommandError(e, input=input_)
       938│ 
       939│         return decode(output)
       940│ 
       941│     def execute(self, bin, *args, **kwargs):

[bratao]

Hi @abn , maybe #2388 is related to this bug?
#2388 is a very important scenario ( using a private pypi) that is failing in multiple machines to me.

[abn]

Could be part of the reason. However, I suspect that one is more to do with not being able to download the artifact (eg: html error page being downloaded). Fixing this will simply invalidate it and download again, which might again return the bad content.

[Eugeny]

Also, clearing the cache doesn't remove these corrupted downloads

[noritada]

I encountered the same issue and found this issue report.
In my case, interruption of package installation by signal creates a broken cache file, and subsequent installation trials all failed.

I worked around by manually downloading the tarball and putting it to the path shown in the error message.

[earonesty]

disabling the experimental new installer "fixes" this, but really just need to download to tmp, validate, and rename later instead of polluting the cache : poetry config experimental.new-installer false

[earonesty]

using this fork, also fixes this for me: #3301

[neersighted]

Solved in cc3f994 -- we now don't make a file available in the cache until the download is complete.

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.