Allow opting out the pypi JSON API

[ralbertazzi]

• I have searched the issues of this repo and believe that this is not a duplicate.

Probably the opposite request of #5337 😄

I searched the Poetry CHANGELOG and, although I read some mentions about PyPI deprecated JSON APIs, I believe Poetry is still using the JSON API when targeting the public pypi repository. If that's not the case, then we can risolve the issue immediately.

As the PyPI JSON API is inherently bugged to due dependencies being specified at the package level rather than at the specific version level, I'd like to have a way to completely opt out of that feature and consider PyPI as a PEP 503 Simple API repository. Note how in PDM this is even the default behaviour: pdm-project/pdm#112.

I didn't find in the documentation an explicit way to disable this behaviour, although I'd like to ask you if through some hack this is already achievable - like defining a custom repository with default = true that points to https://pypi.org/simple/. If there is a solution and is not too hackish (aka we should expect the behaviour to stay consistent across Poetry releases), then I would consider this issue resolved without further changes.

Thank you a lot for maintaining Poetry 👏

[dimbleby]

like defining a custom repository with default = true that points to https://pypi.org/simple/.

did you try this? what happened?

regardless of the API that it queries, poetry assumes that all distributions of a package have the same metadata: so merely "not querying the JSON API" will make no difference to that.

You didn't say what problem you're trying to solve that has sent you down this path; possibly if you said something about that then more useful advice could be offered.

[ralbertazzi]

Hi @dimbleby , thanks for the response.

I did try this and it actually seems to work! However, I don't have full observability on whether the JSON API is totally avoided or not. I will try some network profiling

[[tool.poetry.source]]
default = true
name = "pypi-simple"
url = "https://pypi.org/simple"

regardless of the API that it queries, poetry assumes that all distributions of a package have the same metadata

Could you elaborate on that? What I was assuming is that Poetry will correctly fetch the dependencies of packages published on a Simple API repository. Say the repository contains

• package-a 1.0, depends on package-b >= 2.0
• package-a 1.1, depends on package-b < 2.0
  Will Poetry correctly fetch the sub-dependencies of package-a based on its version? This is what the JSON API does not seem to offer, as the request_dist field could only contain either package-b >= 2.0 or package-b < 2.0.

You didn't say what problem you're trying to solve that has sent you down this path; possibly if you said something about that then more useful advice could be offered.

It's not a specific problem, merely making sure that the right dependencies are selected for a given package. The JSON API doesn't seem to offer such functionality because of the "common" requires_dist field.

[dimbleby]

If not using another API, poetry will download some arbitrary distribution of the package whose metadata it needs. That might be the windows distribution or the linux one; or a py38 distribution or a py310 distribution, or whatever.

Then it will read the metadata from that distribution and assume that it is the same for all distributions of that version of the package.

(This is analogous to reading the same data from the JSON API, which is populated by whichever distribution happens to be uploaded first)

You cannot solve the problem that you are trying to solve by avoiding the JSON API.

On the other hand since you don't have an actual problem that you are trying to solve, that doesn't seem so important.

[dimbleby]

As the PyPI JSON API is inherently bugged to due dependencies being specified at the package level rather than at the specific version level,

We may be at cross-purposes: this is simply not true

The JSON API offers per-version information, which poetry uses. So perhaps you're happy all along.

What it does not offer is per-distribution information, which can in principle vary. But it nearly always doesn't.

[ralbertazzi]

If not using another API, poetry will download some arbitrary distribution of the package whose metadata it needs. That might be the windows distribution or the linux one; or a py38 distribution or a py310 distribution, or whatever.

That makes total sense, and I'm not here to question this :) I'm only interested about different dependencies for different versions of the same package, not different distributions.

The JSON API offers per-version information, which poetry uses. So perhaps you're happy all along.

Ok, maybe I completely missed it and if you're right (I'm sure you are!) then we can consider this issue solved. Could you just point me to that out of curiosity? If I run this

curl -s https://pypi.org/pypi/aio-pika/json | jq '.info.requires_dist'

Then I only see a single requires_dist entity. How does Poetry fetch the information of another version of the same library?

On the other hand since you don't have an actual problem that you are trying to solve, that doesn't seem so important.

I'm considering moving from a private repository that mirrors pypi to calling pypi directly. Since the behaviour is different (as the private repository does not offer the JSON API), I just want to verify that dependency resolution will keep being correct. I didn't get into an issue yet, but I wanted to investigate :)

[dimbleby]

How does Poetry fetch the information of another version of the same library?

eg https://pypi.org/pypi/aio-pika/8.2.5/json

[ralbertazzi]

Oooo then consider myself fully satisfied. The possibility of specifying the version in the JSON API completely slipped off me. Thanks for the quick reply!

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.