[pypi] pytorch index urls support

[Kakadus]

How are you running Renovate?

Mend Renovate hosted app on github.com

If you're self-hosting Renovate, tell us what version of Renovate you run.

No response

If you're self-hosting Renovate, select which platform you are using.

None

Was this something which used to work for you, and then stopped?

I have not seen this working

Describe the problem

The pytorch pypi indices such as https://download.pytorch.org/whl/cpu/ only support the PEP 503 simple api and answers non-existent urls with 403 Forbidden by default, such as https://download.pytorch.org/whl/cpu/torch/json. Thus, renovate does not retry as a simple index.
To reproduce, simply add torch to your dependencies as well as https://download.pytorch.org/whl/cpu/ as an extra index url. We use the poetry manager, but it should not be poetry specific.
As a fix, you could add pytorchs url scheme to

renovate/lib/modules/datasource/pypi/index.ts

Lines 44 to 45 in d2536a7

	// not all simple indexes use this identifier, but most do
	if (hostUrl.endsWith('/simple/') || hostUrl.endsWith('/+simple/')) {

or add 403 to the statusCode check in

renovate/lib/modules/datasource/pypi/index.ts

Line 60 in d2536a7

if (err.statusCode !== 404) {

Relevant debug logs

Logs

DEBUG: hostRules: no authentication for pypi.org
DEBUG: GET https://download.pytorch.org/whl/cpu/torch/json = (code=ERR_NON_2XX_3XX_RESPONSE, statusCode=403 retryCount=0, duration=102)
DEBUG: Datasource unauthorized
{
  "datasource": "pypi"
  "packageName": "torch"
  "url": "https://download.pytorch.org/whl/cpu/torch/json"
}

DEBUG: Failed to look up pypi package torch
{
  "dependency": "torch"
  "packageFile": "pyproject.toml"
}

DEBUG: GET https://download.pytorch.org/whl/cpu/torchaudio/json = (code=ERR_NON_2XX_3XX_RESPONSE, statusCode=403 retryCount=0, duration=106)
DEBUG: Datasource unauthorized
{
  "datasource": "pypi"
  "packageName": "torchaudio"
  "url": "https://download.pytorch.org/whl/cpu/torchaudio/json"
}

DEBUG: Failed to look up pypi package torchaudio
{
  "dependency": "torchaudio"
  "packageFile": "pyproject.toml"
}
...
DEBUG: http statistics
{
  "urls": {
     ....
    "https://download.pytorch.org/whl/cpu/torch/json (GET,403)": 1,
    "https://download.pytorch.org/whl/cpu/torchaudio/json (GET,403)": 1,
  }
  "hostStats": {
    "download.pytorch.org": {
      "requestCount": 2,
      "requestAvgMs": 104,
      "queueAvgMs": 1
    }
   ....
  }
}

Have you created a minimal reproduction repository?

I have linked to a minimal reproduction in the description above

[Kakadus]

Created a PR for the statusCode check in #26386

[rarkins]

I'm not sure I like the idea of retrying on a 403 unless that's the only way. If a tool like pip or poetry encounters such a server, what is their logic for determining to use the simple endpoint? If they try and get a 403 and then retry with simple then that's probably the only way, but it does not sound like a good idea because will get a lot of false positives

[Kakadus]

you are definitely right!

I tried with pip and poetry, and they just used the simple api right away

[rarkins]

So now we need to work out how they know this URL is simple

[Kakadus]

as far as I can see, pip will always use the simple interface, as per docs

pip searches for packages on PyPI using the HTTP simple interface, which is documented here and there.
https://pip.pypa.io/en/stable/cli/pip_install/#finding-packages

[rarkins]

I believe we prefer to be API-first because of the metadata available

[TheKevJames]

Hitting the same issue and can't get around it without disabling abortOnError / adding 403 to the abortIgnoreStatusCodes, which isn't ideal. Maybe we could add a new option to hostRules? eg.

  "hostRules": [
    {
      "hostType": "pypi",
      "matchHost": "download.pytorch.org",
      "forceSimpleApi": true
    }
  ]

That's a bit specific to pypi, but maybe a suitably general-enough-to-warrant-being-a-config-option idea would be:

  "hostRules": [
    {
      "hostType": "pypi",
      "matchHost": "download.pytorch.org",
      "attemptFallbackAfterError": true
    }
  ]

[github-actions[bot]]

Hi there,

Get your discussion fixed faster by creating a minimal reproduction. This means a repository dedicated to reproducing this issue with the minimal dependencies and config possible.

Before we start working on your issue we need to know exactly what's causing the current behavior. A minimal reproduction helps us with this. Discussions without reproductions are less likely to be converted to Issues.

Please follow these steps:

1. Read our guide on creating a minimal reproduction.
2. Go to our minimal reproduction template repository.
3. Select the Use this template button to create a new repository based on the template.
4. Work on the minimal reproduction in your own repository.
5. Fill out the information in your repository's README.md.
6. Add the link to your reproduction to the first post of your Discussion. If you are not the original author, you can post a new comment with the link.

If you need help with running renovate on your minimal reproduction repository, please refer to our Running Renovate guide.
Good luck,

The Renovate team