BadZipFile on trying to install package from private repository (Gemfury)

[licht1stein]

• I am on the latest Poetry version.

• I have searched the issues of this repo and believe that this is not a duplicate.

• If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

• OS version and name: Windows 10 Pro x64

• Poetry version: 1.0.5

• Link of a Gist with the contents of your pyproject.toml file: pyproject.toml

Issue

I get BadZipFile when trying to install a package from private repository - gemfury.

1. The package itself is built and published using poetry.
2. The package is successfully installed using pip.

Here's the full console output from running poetry add PACKAGE -vvv

This bug has already been referenced in a closed issue #1483

[costasgambit]

I can confirm I have the same error, also with Gemfury and Poetry 1.0.5, but on Ubuntu 20.04.

[tharwan]

I believe I have the same problem with our azure PyPI. I think it has something to do with the pip.conf and the [[tool.poetry.source]] entry.

There seems to be one config that works in our build job and one that works for me locally.

One source of error seems to be the pip version used by poetry another is the repository entry in the poetry config. But I have not yet figured it.

[bratao]

This error is also frequently occurring to me. @sdispater @finswimmer do you have any insight what can we do about it?

[simonschmidt]

I managed to work around this by putting some junk in the password part of the URL:

url = "https://TOKEN:dummy@pypi.fury.io/REPO_NAME/"

[IMBlues]

Same error here with private pypi config:

[[tool.poetry.source]]
name = "some-private"
url = "https://pypi.PRIVATE.com/simple/"
default = true

Error message shows up with every poetry install :

[BadZipfile]
File is not a zip file

[dmder]

Encountered the same error (Poetry version 1.0.8, MacOS 10.15.5)

Some observations:

It works if I configure the secrets using poetry config http-basic.gemfury <TOKEN> <any_fake_non_empty_password>

It doesn't work if I configure everything identical to step 1, but using environment variables. I get:

 File "/opt/anaconda3/envs/.../lib/python3.6/site-packages/poetry/installation/pip_installer.py", line 62, in install
    index_url = repository.authenticated_url
  File "/opt/anaconda3/envs/.../lib/python3.6/site-packages/poetry/repositories/legacy_repository.py", line 224, in authenticated_url
    password=quote(self._auth.auth.password, safe=''),
  File "/opt/anaconda3/envs/.../lib/python3.6/urllib/parse.py", line 825, in quote
    return quote_from_bytes(string, safe)
  File "/opt/anaconda3/envs/.../lib/python3.6/urllib/parse.py", line 850, in quote_from_bytes
    raise TypeError('quote_from_bytes() expected bytes')

It doesn't work if I configure the secrets inside the repo url, like https://<TOKEN>:@pypi.fury.io/<repo>/ or https://<TOKEN>:fake@pypi.fury.io/<repo>/ - get the same error as above.

Does anyone know why there is a discrepancy in behaviour between secrets configuration and env variables?

[Persedes]

thank you @dmder ! We've had that same problem with azure / vsts, except it manifested itself with the error:

[EnvCommandError]
Command ['/usr/local/bin/python', '-m', 'pip', 'install', '--no-deps', '--index-url', ........
Looking in indexes: ....
Collecting docutils==0.15.2
    self.link = finder.find_requirement(self, upgrade)
  File "/usr/local/lib/python3.6/site-packages/pip/_internal/index.py", line 494, in find_requirement
    all_candidates = self.find_all_candidates(req.name)
  File "/usr/local/lib/python3.6/site-packages/pip/_internal/index.py", line 452, in find_all_candidates
    for page in self._get_pages(url_locations, project_name):
  File "/usr/local/lib/python3.6/site-packages/pip/_internal/index.py", line 597, in _get_pages
    page = self._get_page(location)
  File "/usr/local/lib/python3.6/site-packages/pip/_internal/index.py", line 715, in _get_page
    return HTMLPage.get_page(link, session=self.session)
  File "/usr/local/lib/python3.6/site-packages/pip/_internal/index.py", line 824, in get_page
    "Cache-Control": "max-age=600",
  File "/usr/local/lib/python3.6/site-packages/pip/_vendor/requests/sessions.py", line 525, in get
    return self.request('GET', url, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/pip/_internal/download.py", line 396, in request
    return super(PipSession, self).request(method, url, *args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/pip/_vendor/requests/sessions.py", line 512, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python3.6/site-packages/pip/_vendor/requests/sessions.py", line 629, in send
    r = dispatch_hook('response', hooks, r, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/pip/_vendor/requests/hooks.py", line 31, in dispatch_hook
    _hook_data = hook(hook_data, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/pip/_internal/download.py", line 188, in handle_401
    username = six.moves.input("User for %s: " % parsed.netloc)
EOFError: EOF when reading a line

... and broke all builds using poetry.
The builds are using poetry 1.0.3 (via pip install) and were running fine until a few weeks ago.

[peldszus]

I also ran into this issue for a while now, currently also with poetry 1.0.10.

What did the trick for me is to clear (not the pip cache in ~/.cache/pip/ as I first thought, but) poetry's cache for that specific custom repository:

rm -rf ~/.cache/pypoetry/cache/repositories/my-custom-pypi/

[RoelantStegmann]

Having the same here with an azure devop repo. It does solve the versions (so it must see the repo), but it doesn't install

[abn]

This seems to be related to auth failures causing wheels not being downloaded. The situation will be more visiable thanks to #2576, resolved via #2577 (available in 1.1.0rc1). This also means that the empty/error html file won't be downloaded.

In the interim, you can remove any affected caches by doing the following:

poetry cache clear --all <repo-name>

@dmder your issue seems to be a URL parsing issue. When using config, poetry sends auth as an Authorization header instead of as basic auth credentials in the url. Additionally, I would recommend not doing that since this information (url) can get logged and poetry does not attempt to determin if your url contains credentials.

[peldszus]

Nice!

PS:
Thanks! I wasn't aware of poetry cache clear. Probably, because poetry cache is not yet visible in the https://python-poetry.org/docs/cli/, even though it is already there in my 1.0.10 version.

I see there was documentation added on the master branch for cache list (https://github.com/python-poetry/poetry/blob/master/docs/docs/cli.md#cache), but not yet for cache clear.

[abn]

but not yet for cache clear.

Happy to accept a PR. :)

[peldszus]

Will do, if you can wait for the weekend. :)

[peldszus]

There you go: Issue #3065, PR #3066

[alimantu]

I'm using poetry to create quite huge virtualenvs - 80+ packages and after fresh reinstall from poetry 1.0.5 to 1.1.2 I faced with this zipfile.BadZipFile issue. I tried to use poetry cache clear --all <pypi_name>, but in my case it didn't help.

What did?

1. Check the listing, search for the path of the specific corrupted wheel/tar.gz. It's usually starts with Command ...../pip.exe install --no-deps file:://<file_path>
2. Remove folder from pypoetry/Cache/artifacts/ with corrupted cached version of the package. See this documentation to find Cache folder in your OS.
3. Rerun poetry install command.

Example:
For this specific error listing you need to delete ~/AppData/Local/pypoetry/Cache/atrifacts/2f folder.

  • Installing notebook (6.1.4): Failed

  EnvCommandError

  Command C:\Users\asalynskiy\Documents\Python\MinorProjects\bjb-fx-forwards\.venv\Scripts\pip.exe install --no-deps file:///C:/Users/asalynskiy/AppData/Local/pypoetry/Cache/artifacts/2f/08/b7/b1d12be93495af1612f5533bfcb5ce5f1b83c281bfd093ed17538042e0/notebook-6.1.4-py3-none-any.whl errored with the following return code 2, and output:
  Processing c:\users\asalynskiy\appdata\local\pypoetry\cache\artifacts\2f\08\b7\b1d12be93495af1612f5533bfcb5ce5f1b83c281bfd093ed17538042e0\notebook-6.1.4-py3-none-any.whl
  ERROR: Exception:
  Traceback (most recent call last):
    File "C:\Users\asalynskiy\Documents\Python\MinorProjects\bjb-fx-forwards\.venv\lib\site-packages\pip\_internal\cli\base_command.py", line 216, in _main
      status = self.run(options, args)
    File "C:\Users\asalynskiy\Documents\Python\MinorProjects\bjb-fx-forwards\.venv\lib\site-packages\pip\_internal\cli\req_command.py", line 182, in wrapper
      return func(self, options, args)
    File "C:\Users\asalynskiy\Documents\Python\MinorProjects\bjb-fx-forwards\.venv\lib\site-packages\pip\_internal\commands\install.py", line 325, in run
      reqs, check_supported_wheels=not options.target_dir
    File "C:\Users\asalynskiy\Documents\Python\MinorProjects\bjb-fx-forwards\.venv\lib\site-packages\pip\_internal\resolution\legacy\resolver.py", line 183, in resolve
      discovered_reqs.extend(self._resolve_one(requirement_set, req))
    File "C:\Users\asalynskiy\Documents\Python\MinorProjects\bjb-fx-forwards\.venv\lib\site-packages\pip\_internal\resolution\legacy\resolver.py", line 391, in _resolve_one
      dist = abstract_dist.get_pkg_resources_distribution()
    File "C:\Users\asalynskiy\Documents\Python\MinorProjects\bjb-fx-forwards\.venv\lib\site-packages\pip\_internal\distributions\wheel.py", line 29, in get_pkg_resources_distribution
      with ZipFile(self.req.local_file_path, allowZip64=True) as z:
    File "C:\Users\asalynskiy\AppData\Local\Programs\Python\Python37\lib\zipfile.py", line 1222, in __init__
      self._RealGetContents()
    File "C:\Users\asalynskiy\AppData\Local\Programs\Python\Python37\lib\zipfile.py", line 1289, in _RealGetContents
      raise BadZipFile("File is not a zip file")
  zipfile.BadZipFile: File is not a zip file
  WARNING: You are using pip version 20.2.2; however, version 20.2.3 is available.
  You should consider upgrading via the 'C:\Users\asalynskiy\Documents\Python\MinorProjects\bjb-fx-forwards\.venv\Scripts\python.exe -m pip install --upgrade pip' command.


  at ~\.poetry\lib\poetry\utils\env.py:948 in _run
       944│                 output = subprocess.check_output(
       945│                     cmd, stderr=subprocess.STDOUT, **kwargs
       946│                 )
       947│         except CalledProcessError as e:
    →  948│             raise EnvCommandError(e, input=input_)
       949│
       950│         return decode(output)
       951│
       952│     def execute(self, bin, *args, **kwargs):

And yes, it took me about 4 iterations to download all the packages - each time poetry had some troubles with one of the downloaded wheels/tar.gz's. And it really makes me sad, because with 1.0.5 I have never faced issues like that and now 4 times in a row.

• OS version and name: Windows 10 Pro x64
• Poetry version: 1.1.2

@sdispater can you please take a closer look on it?

[abn]

@alimantu appreciate the details provided. The new installer is a significant shift from what was being done in 1.0.z, and it does come with it a new set of concerns. We do anticipate bugs along the way till the implementation stabilises.

The particular issue you seem to be hit with is a failure to discard corrupt/invalid cached artifacts #2674. The cache clear does not invalidate the artifact caches at the moment. We will be hardening this and issues related to fetching artifacts in the coming releases. Since this issue was previously closed, it would be helpful if errors like yours are either reported in a new issue or added to an already open issue.

If the experimental nature of the installer is a concern, I'd recommend that you disable the installer untill it stabilises in a future release as instructed here.

[nukopy]

This issue possibly related:

#3496
#3326
#3065
#2674
#2388
#1483
#155

Solution

I resolve this issue like below:

1. Remove cache(2 way)
   • remove cache with the command poetry cache clear --all pypi
   • remove cache manually
     • show cache directory with poetry config --list and remove manually
     • e.g.

       • cache-dir = "/Users/[username]/Library/Caches/pypoetry"

       • then execute rm -rf ~/Library/Caches/pypoetry/**
2. Execute poetry add [package name] again

[LSaldyt]

For me, this occurred when cancelling a download, leaving an unusable wheel file. The fix was to manually remove the offending wheel file, the path to which is visible in the logs, in the line below "EnvCommandError" with "Command":
rm -rf /home/user/.cache/pypoetry/artifacts/id/id/id/id/package.whl.
Removing the entire cache would also work, but could result in large redownloads

[earonesty]

this PR will resolve this issue: #3301

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.