Have a way to bypass TooManyRedirects

[pierresouchay]

• I am on the poetry version 1.1.13
• I have searched the issues of this repo and believe that this is not a duplicate.
• If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

• OS version and name: Mac OS or Linux, any version
• Poetry version: 1.1.13

Issue

Hello,

Since today, it seems that some packages on pypi return some kind of infinite loop, example:

curl -L https://pypi.org/pypi/Werkzeug/json
curl: (47) Maximum (50) redirects followed

Such package cause poetry to fail:

poetry add Werkzeug

  TooManyRedirects

  Exceeded 30 redirects.

  at ~/hello-pip/venv/lib/python3.10/site-packages/requests/sessions.py:191 in resolve_redirects
      187│             except (ChunkedEncodingError, ContentDecodingError, RuntimeError):
      188│                 resp.raw.read(decode_content=False)
      189│
      190│             if len(resp.history) >= self.max_redirects:
    → 191│                 raise TooManyRedirects(
      192│                     f"Exceeded {self.max_redirects} redirects.", response=resp
      193│                 )
      194│
      195│             # Release the connection back into the pool.

But this issue also appear on existing projects with existing lock file, eg:

poetry install
Installing dependencies from lock file

Package operations: 1 install, 0 updates, 0 removals

  • Installing werkzeug (2.0.3): Failed

  TooManyRedirects

  Exceeded 30 redirects for https://pypi.org/pypi/Werkzeug/2.0.3/json.

  at ~/work/python-dev/butterfly/venv/lib/python3.9/site-packages/requests/sessions.py:166 in resolve_redirects
      162│             except (ChunkedEncodingError, ContentDecodingError, RuntimeError):
      163│                 resp.raw.read(decode_content=False)
      164│
      165│             if len(resp.history) >= self.max_redirects:
    → 166│                 raise TooManyRedirects('Exceeded {} redirects for {}.'.format(self.max_redirects, url), response=resp)
      167│
      168│             # Release the connection back into the pool.
      169│             resp.close()
      170│

=> If I simply remove the package from venv, it get broken.

Is the is a way to avoid such behavior? (Currently, all of our CIs are broken because of this package)

Note that pip does not have this behavior:

pip install Werkzeug
Looking in indexes: https://pypi.python.org/simple/
Collecting Werkzeug
  Downloading Werkzeug-2.1.2-py3-none-any.whl (224 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 224.9/224.9 KB 4.5 MB/s eta 0:00:00
Installing collected packages: Werkzeug
Successfully installed Werkzeug-2.1.2

[rubenaranamorera]

Experiencing this also, is there any known workarround?

[iamhatesz]

Same situation for me, can't do poetry lock. Clearing caches doesn't help. I'm using 1.2.0b1 version.

[johnthagen]

PyPA status update

This is a known issue with PyPI that is being addressed.

• Redirect error on pypi.python.org/.../json pypi/warehouse#11554 (comment)
• https://status.python.org/incidents/lgpr13fy71bk

[studio3104]

I'm facing completely the same situation.
In my case, poetry install doesn't work if pyproject.toml contains TA-Lib, works if it's commented out. But of cource the project doesn't work if I do so...
Clearing caches doesn't help also for my case, using 1.1.13.

[olegZastavnyi]

I've got the same problem for poetry v.1.1.6.

[paul-tharun]

@olegZastavnyi it is not an issue with poetry. Pypi is facing some issues so it would be fixed in 1-2 hours

[YongWookHa]

Same problem here.
poetry add, poetry install, any commands don't work.
Is it a server problem?

[gmagannaDevelop]

@paul-tharun are you sure about this? To me it seems to be an issue related to poetry because using pip and conda do not fail when trying to install pillow, while poetry does.

[roma-glushko]

Hey people, I have just faced the same issue with the deprecated lib:

Have we come up any fix for this issue? 🙏

[lars-reimann]

@paul-tharun are you sure about this? To me it seems to be an issue related to poetry because using pip and conda do not fail when trying to install pillow, while poetry does.

poetry uses a different endpoint than pip/conda. See also https://status.python.org/incidents/lgpr13fy71bk.

[neersighted]

Hi, there's no way to work around a broken PyPI in Poetry as the API endpoints we need returning junk data mean we can't find packages to install. I'm going to close this for now since the relevant PyPI issue has been linked -- this should be resolved shortly according to upstream.

[txels]

This is how I have implemented falling back to pip in my CI from poetry:

poetry export --dev > requirements.txt
pip install -r requirements.txt

pip seems to use different PyPI endpoints so it's not affected by this specific PyPI issue.

[dstufft]

FWIW, PyPI considers the Simple API to be our "Tier 1" API, and we will typically go to great lengths to try not to break that.

Any other part of PyPI we consider to be a best effort "Tier 2" endpoint, and we're more willing to have short term breakages if needed. It's not recommended for any sort of critical system to rely on anything but the Simple API. If something needs more than that, we suggest writing a PEP to help promote more pieces into a "Tier 1" endpoint.

[roma-glushko]

Based on the @dstufft comment, Poetry is better to use tier 1 APIs, right?
@neersighted to your point about junk responses, this is something to consider in order to avoid the same things happening over and over again.

[sithembiso]

Facing the same issue here:

...
  • Installing pillow (8.4.0)
  • Installing pycodestyle (2.8.0)
  TooManyRedirects
  Exceeded 30 redirects.
  at /usr/local/lib/python3.9/site-packages/requests/sessions.py:191 in resolve_redirects
      187│                 url = urljoin(resp.url, requote_uri(url))
      188│             else:
      189│                 url = requote_uri(url)
      190│ 
    → 191│             prepared_request.url = to_native_string(url)
      192│ 
      193│             self.rebuild_method(prepared_request, resp)
      194│ 
      195│             # https://github.com/requests/requests/issues/1084
  • Installing python-crontab (2.6.0)
...

[roma-glushko]

@sithembiso the latest update from pypi:

Seems like libs that starts with "p" is still impacted by the issue.

[sithembiso]

Thank you @roma-glushko that's very helpful.

[dstufft]

To be clear, I have no idea what data poetry is pulling from the JSON API, so it's entirely possible that the data poetry is using doesn't exist in the simple api. In that case you're kinda stuck picking between:

1. Status quo, and accepting the increased chances of breakage every now and then.
   • I don't want to overstate this, we still do try not to break it or have downtime, but with no standard or documentated expectations for things like the JSON API, changes there tend to be at a larger risk to falling to Hyrum's Law.
2. Stop using that data, and use only the data available in the Simple API.
3. Go through the process to standardize an API that has the data you need or to add it to an existing standardized API.

We're OK with people picking any of those options :) I just wanted to call it out explicitly that if you're able to pick (2) or (3), you're less likely to run into random problems, but if you want to pick (1), that's totally fine.

[neersighted]

Based on the @dstufft comment, Poetry is better to use tier 1 APIs, right? @neersighted to your point about junk responses, this is something to consider in order to avoid the same things happening over and over again.

Poetry uses the JSON API to avoid the need to download distfiles in order to gather metadata. While we do still have support for the Legacy API in our codebase and we could technically switch PyPI back to using it by default, it would represent a significant regression for Poetry as our use case requires us to have package metadata during solving/locking and not merely at install time like pip.

Similarly, it could be possible to fall back to the Legacy API, but I am really not sure that it's a good idea to maintain an untested/rarely used code path for the rare cases that PyPI is down. Falling back would also make Poetry much slower and download more data with seemingly no explanation.

Ultimately, I think that if the JSON API is less supported than the Legacy API, upstream should call that out in docs for downstreams first, and we should then reason about what kind of failure is acceptable if there is an issue with PyPI itself.

Facing the same issue here:

...
  • Installing pillow (8.4.0)
  • Installing pycodestyle (2.8.0)
  TooManyRedirects
  Exceeded 30 redirects.
  at /usr/local/lib/python3.9/site-packages/requests/sessions.py:191 in resolve_redirects
      187│                 url = urljoin(resp.url, requote_uri(url))
      188│             else:
      189│                 url = requote_uri(url)
      190│ 
    → 191│             prepared_request.url = to_native_string(url)
      192│ 
      193│             self.rebuild_method(prepared_request, resp)
      194│ 
      195│             # https://github.com/requests/requests/issues/1084
  • Installing python-crontab (2.6.0)
...

Per upstream this is not fully resolved yet.

[dstufft]

Poetry uses the JSON API to avoid the need to download distfiles in order to gather metadata. While we do still have support for the Legacy API in our codebase and we could technically switch PyPI back to using it by default, it would represent a significant regression for Poetry as our use case requires us to have package metadata during solving/locking and not merely at install time like pip.

I'm sure you're aware, but just in case you're not, the metadata in the JSON API is historically very unreliable, and has a lot of caveats to trusting it. It will semi regularly just be missing and/or wrong depending on a number of factors.

Ultimately, I think that if the JSON API is less supported than the Legacy API, upstream should call that out in docs for downstreams first, and we should then reason about what kind of failure is acceptable if there is an issue with PyPI itself.

The "Tier 1" thing isn't an official designation or anything, just in practice that's how we've typically regarded the APIs. We've primarily been concerned with getting people off of things like the XMLRPC api which are like, "please never use this" and haven't though too hard about it. Some of it is the nature of the JSON API being a wholly unstandardized API, so there's not as good of a sense of what we can change without breaking someone.

I do agree we should formalize this though, so I opened pypi/warehouse#11559 :)

[johnthagen]

Resolved 🎉

The upstream PyPI issue has been resolved, and I can report that our poetry installs are working again in CI.

Thanks to the PyPI team work resolving this quickly. ❤️

• https://status.python.org/incidents/lgpr13fy71bk

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.