Get hashes from PyPI JSON API #1109

atugushev · 2020-04-19T10:31:22Z

Resolves #543.

Before

$ time echo web3 | pip-compile - -qo- --generate-hashes > /dev/null

real	1m32.754s
user	0m10.959s
sys	0m3.083s

After

$ time echo web3 | pip-compile - -qo- --generate-hashes > /dev/null

real	0m4.824s
user	0m2.603s
sys	0m0.141s

Changelog-friendly one-liner: Get hashes from PyPI JSON API.

Contributor checklist

Provided the tests for the changes.
Gave a clear one-line description in the PR (that the maintainers can add to CHANGELOG.md on release).
Assign the PR to an existing or new milestone for the target version (following Semantic Versioning).

piptools/repositories/pypi.py

codecov · 2020-04-19T10:45:56Z

Codecov Report

Merging #1109 into master will increase coverage by 0.01%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master    #1109      +/-   ##
==========================================
+ Coverage   99.44%   99.46%   +0.01%     
==========================================
  Files          36       36              
  Lines        2540     2631      +91     
  Branches      312      318       +6     
==========================================
+ Hits         2526     2617      +91     
  Misses          8        8              
  Partials        6        6

Impacted Files	Coverage Δ
piptools/repositories/pypi.py	`94.52% <100.00%> (+1.26%)`	⬆️
tests/test_repository_pypi.py	`100.00% <100.00%> (ø)`

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a33e653...82d093f. Read the comment docs.

graingert · 2020-04-19T11:47:51Z

it might be worth always getting the hashes now rather than skipping them if the hashes are already in the requirements.txt (this behaviour causes me problems, eg when new wheels are uploaded to an old version I don't get the hash of the new wheel

atugushev · 2020-04-19T11:53:11Z

We could ignore current hashes using --rebuild. What do you think?

atugushev · 2020-04-19T11:53:38Z

But if think it deserves a discussion in a separate issue.

graingert · 2020-04-19T12:42:56Z

piptools/repositories/pypi.py

+        for package_index in package_indexes:
+            url = "{url}/{name}/json".format(url=package_index.pypi_url, name=ireq.name)
+            try:
+                data = self.session.get(url).json()


we should probably remember which URLs support the PyPI JSON api

How would we know that JSON API of a custom index URL is supported?

if this URL doesn't 404

If it raises 404 then it falls back to the files hash checking.

Perhaps 404 should be handled explicitly because currently, it falls back on every HTTP/JSON error.

Sorry, I might be missing your point, but I'll try to describe mine in detail.

Using this heuristic, we can't be sure that JSON API is not supported. Consider the following case:

package1 from https://pypi1-server/pypi/package1/json

package2 from https://pypi2-server/pypi/package2/json

package3 from https://pypi1-server/pypi/package1/json

Suppose we run the following command:

$ pip-compile --index-url=https://pypi1-server/ --extra-index-url=https://pypi2-server/

Step 1. When pip-tools processes package1 everything is okay because it hits https://pypi1-server/pypi/package1/json which responses 200.

Step2. After, pip-tools compiles package2 and hit the first index URL, which is https://pypi1-server/pypi/package2/json. But this URL raises 404 because the package is on pypi2-server. So it has to go to the second index URL, which is https://pypi2-server/pypi/package2/json, which responses 200.

Step3. Then pip-tools processes package3 the same as in step 1 with the first index URL.

How would we know that JSON API of a custom index URL is supported?
If this URL doesn't 404.

Considering the above statement the test case will be failed on step 3 because pypi1-server would be marked as "doesn't support JSON API" on step 2 because pip-tools couldn't find package2 on pypi1-server due to it responded 404.

Am I missing something?

if the simple index and the JSON API 404s then you don't know
if the simple index 200s and the JSON API 404s then you know the API is unsupported

I see. Yes, that would work. Assuming that suggestion is to hit less the PyPI servers now we have to hit the servers twice on 404. So what's the point of the optimization?

graingert: You have to hit it twice to download the hashes from indexes that don't support the JSON API

Sorry, I don't understand what are you trying to achieve by that. Could you elaborate?

If you think this should be fixed anyway please feel free to open an issue/PR. Thank you for the help!

graingert · 2020-04-19T14:28:51Z

You have to hit it twice to download the hashes from indexes that don't support the JSON API

…

On Sun, 19 Apr 2020, 15:23 Albert Tugushev, ***@***.***> wrote: ***@***.**** commented on this pull request. ------------------------------ In piptools/repositories/pypi.py <#1109 (comment)>: > - try: - data = self.session.get(url).json() - except (ValueError, HTTPError) as e: - log.error( - "Fetch package info from PyPI failed: {url}: {e}".format(url=url, e=e) - ) - return None - return data + package_indexes = [ + PackageIndex(url=index_url, file_storage_domain="") + for index_url in self.finder.search_scope.index_urls + ] + for package_index in package_indexes: + url = "{url}/{name}/json".format(url=package_index.pypi_url, name=ireq.name) + try: + data = self.session.get(url).json() I see. Yes, that would work. Assuming that suggestion is to hit less the PyPI servers now we have to hit the servers twice on 404. So what's the point? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1109 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AADFATBWV3HX4WUC6BEFYY3RNMCNHANCNFSM4MLXNCFQ> .

atugushev · 2020-04-19T18:55:00Z

Okay, it's ready for review now.

AndydeCleyre · 2020-04-19T20:52:30Z

I may be missing something in pip's internal vendored requests.session business, but AFAIU there's no timeout getting set during the self.session.get(url) call, which could be trouble.

If so, I notice there's one other case like that in pypi.py that predates this PR (from 627dbaf).

I'll take a closer look at this whole thing tonight, but it looks really nice, especially that timed example you've provided.

Thanks!

atugushev · 2020-04-19T20:56:31Z

@AndydeCleyre

It respects pip's options.timeout, which is default 15 seconds.

piptools/repositories/pypi.py

AndydeCleyre · 2020-04-20T03:07:20Z

Great! This is a much appreciated improvement.

If you like, minor English edits in the docstrings:

"by a given InstallRequirement" -> "for a given..." (x2)
"from every release files" -> "of all release files" or "for all release files"

atugushev · 2020-04-20T06:32:13Z

Thanks @AndydeCleyre for reviewing this! Your suggestions addressed in a03f3fd17dafd0e8477f11785f03099e224f222c.

atugushev · 2020-04-21T18:37:56Z

Rebased onto the master and squashed commits.

atugushev · 2020-04-21T20:17:45Z

@AndydeCleyre @auvipy @graingert

Thanks for reviewing this! 🙏

graingert · 2020-04-27T08:20:27Z

@atugushev the speedup here is going to be incredible! ❤️ ❤️ ❤️ ❤️ 🎉

I was just rehashing a requirements that uses numpy pandas and lxml and thinking longingly of this PR.

@atugushev I think it's worth collecting some benchmarks for the changelog, like "generate-hashes in 5.1 will take 0.001x the time 5.0 used to"

atugushev · 2020-04-27T13:29:51Z

pip-tools v5.1.0 is released. Thanks @graingert!

graingert · 2020-04-27T14:25:59Z

@atugushev fyi this PR means you now get .egg hashes in the requirements.txt, eg:

 toml==0.10.0 \
     --hash=sha256:229f81c57791a41d65e399fc06bf0848bab550a9dfd5ed66df18ce5f05e73d5c \
     --hash=sha256:235682dd292d5899d361a811df37e04a8828a5b1da3115886b73cf81ebc9100e \
+    --hash=sha256:f1db651f9657708513243e61e6cc67d101a39bad662eaa9b5546f789338e07a3 \

atugushev · 2020-04-27T17:11:04Z

Erm, I wonder should we skip egg hashes?

graingert · 2020-04-27T17:16:20Z

@atugushev what mechanism skipped them before?

atugushev · 2020-04-27T17:27:04Z

pip doesn't support installation from eggs, so it picks wheels and source dists.

atugushev · 2020-04-27T17:28:42Z

So we could skip "packagetype": "bdist_egg" digests, since they aren't useful or leave it as is.

graingert · 2020-04-27T17:31:43Z

@atugushev #1121

atugushev added the enhancement Improvements to functionality label Apr 19, 2020

atugushev mentioned this pull request Apr 19, 2020

use sha256 hashes from warehouse #543

Closed

atugushev commented Apr 19, 2020

View reviewed changes

piptools/repositories/pypi.py Outdated Show resolved Hide resolved

auvipy approved these changes Apr 19, 2020

View reviewed changes

atugushev marked this pull request as draft April 19, 2020 11:14

graingert reviewed Apr 19, 2020

View reviewed changes

atugushev marked this pull request as ready for review April 19, 2020 18:54

atugushev requested review from vphilippon and AndydeCleyre April 19, 2020 18:55

atugushev commented Apr 19, 2020

View reviewed changes

piptools/repositories/pypi.py Outdated Show resolved Hide resolved

atugushev mentioned this pull request Apr 19, 2020

Indent logs with a context manager #1110

Merged

3 tasks

AndydeCleyre approved these changes Apr 20, 2020

View reviewed changes

Get hashes from PyPI JSON API

82d093f

atugushev force-pushed the hashes-from-pypi-json-api-543 branch from a03f3fd to 82d093f Compare April 21, 2020 18:36

atugushev added this to the 5.1.0 milestone Apr 21, 2020

atugushev merged commit 93454b7 into jazzband:master Apr 21, 2020

atugushev deleted the hashes-from-pypi-json-api-543 branch April 21, 2020 20:17

atugushev mentioned this pull request Apr 22, 2020

Show basename of hashing URL in debug logs #1113

Merged

3 tasks

This was referenced Apr 27, 2020

pip-compile --generate-hashes including bdist_egg hashes #1121

Closed

support the PyPI JSON api devpi/devpi#801

Open

get sha256 hash from /simple (PEP503) endpoint #1135

Open

graingert mentioned this pull request Jun 25, 2020

get sha256 hash from /simple (PEP503) endpoint peterbe/hashin#120

Open

This was referenced Jul 9, 2020

Question: Are there risks when hashes couldn't be gotten from PyPI? #1170

Closed

ignore current hashes using --no-reuse-hashes #1176

Closed

AndydeCleyre mentioned this pull request Aug 5, 2022

problems installing torch-geometric #1662

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Get hashes from PyPI JSON API #1109

Get hashes from PyPI JSON API #1109

atugushev commented Apr 19, 2020 •

edited

Loading

codecov bot commented Apr 19, 2020 •

edited

Loading

graingert commented Apr 19, 2020

atugushev commented Apr 19, 2020

atugushev commented Apr 19, 2020 •

edited

Loading

graingert Apr 19, 2020

atugushev Apr 19, 2020 •

edited

Loading

graingert Apr 19, 2020

atugushev Apr 19, 2020

atugushev Apr 19, 2020 •

edited

Loading

atugushev Apr 19, 2020 •

edited

Loading

graingert Apr 19, 2020

atugushev Apr 19, 2020 •

edited

Loading

atugushev Apr 19, 2020 •

edited

Loading

atugushev Apr 21, 2020

graingert commented Apr 19, 2020 via email

atugushev commented Apr 19, 2020

AndydeCleyre commented Apr 19, 2020

atugushev commented Apr 19, 2020

AndydeCleyre commented Apr 20, 2020

atugushev commented Apr 20, 2020

atugushev commented Apr 21, 2020

atugushev commented Apr 21, 2020

graingert commented Apr 27, 2020 •

edited

Loading

atugushev commented Apr 27, 2020

graingert commented Apr 27, 2020

atugushev commented Apr 27, 2020

graingert commented Apr 27, 2020

atugushev commented Apr 27, 2020 •

edited

Loading

atugushev commented Apr 27, 2020 •

edited

Loading

graingert commented Apr 27, 2020

Get hashes from PyPI JSON API #1109

Get hashes from PyPI JSON API #1109

Conversation

atugushev commented Apr 19, 2020 • edited Loading

Before

After

Contributor checklist

codecov bot commented Apr 19, 2020 • edited Loading

Codecov Report

graingert commented Apr 19, 2020

atugushev commented Apr 19, 2020

atugushev commented Apr 19, 2020 • edited Loading

graingert Apr 19, 2020

Choose a reason for hiding this comment

atugushev Apr 19, 2020 • edited Loading

Choose a reason for hiding this comment

graingert Apr 19, 2020

Choose a reason for hiding this comment

atugushev Apr 19, 2020

Choose a reason for hiding this comment

atugushev Apr 19, 2020 • edited Loading

Choose a reason for hiding this comment

atugushev Apr 19, 2020 • edited Loading

Choose a reason for hiding this comment

graingert Apr 19, 2020

Choose a reason for hiding this comment

atugushev Apr 19, 2020 • edited Loading

Choose a reason for hiding this comment

atugushev Apr 19, 2020 • edited Loading

Choose a reason for hiding this comment

atugushev Apr 21, 2020

Choose a reason for hiding this comment

graingert commented Apr 19, 2020 via email

atugushev commented Apr 19, 2020

AndydeCleyre commented Apr 19, 2020

atugushev commented Apr 19, 2020

AndydeCleyre commented Apr 20, 2020

atugushev commented Apr 20, 2020

atugushev commented Apr 21, 2020

atugushev commented Apr 21, 2020

graingert commented Apr 27, 2020 • edited Loading

atugushev commented Apr 27, 2020

graingert commented Apr 27, 2020

atugushev commented Apr 27, 2020

graingert commented Apr 27, 2020

atugushev commented Apr 27, 2020 • edited Loading

atugushev commented Apr 27, 2020 • edited Loading

graingert commented Apr 27, 2020

atugushev commented Apr 19, 2020 •

edited

Loading

codecov bot commented Apr 19, 2020 •

edited

Loading

atugushev commented Apr 19, 2020 •

edited

Loading

atugushev Apr 19, 2020 •

edited

Loading

atugushev Apr 19, 2020 •

edited

Loading

atugushev Apr 19, 2020 •

edited

Loading

atugushev Apr 19, 2020 •

edited

Loading

atugushev Apr 19, 2020 •

edited

Loading

graingert commented Apr 27, 2020 •

edited

Loading

atugushev commented Apr 27, 2020 •

edited

Loading

atugushev commented Apr 27, 2020 •

edited

Loading