Avoid downloading multiple candidates of a same version #9432
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
uranusjr
added
skip news
uranusjr
force-pushed
the
new-resolver-dedup-on-backtracking
branch
from
05cfa6f
to
9c12314
Compare
Since 41a3008, Candidate objects prepare their underlying distribution eagerly on creation, so the error can be caught deterministically to trigger backtracking. This however has a negative side-effect. Since dist preparation involves metadata validation, a remote distribution must be downloaded on Candidate creation. This means that an sdist will be downloaded, validated, and discarded (since its version is known to be incompatible) during backtracking. This commit moves version deduplication of candidates from indexes to before the Candidate object is created, to avoid unneeded preparation. Note that we still need another round of deduplication in FoundCandidates to remove duplicated candidates when a distribution is already installed.
uranusjr
force-pushed
the
new-resolver-dedup-on-backtracking
branch
from
9c12314
to
79cbe6b
Compare
uranusjr
changed the title
|
This one's gonna be a fun one to merge on 20.3.3. |
|
... which reminds me, are you still planning on doing a 20.3.4 release? If so, are you expecting to do it before or after 21.0? I doubt it makes much difference in practice, but I don't want to mess things up for you if I start looking at 21.0 stuff (e.g. vendoring updates, which I should probably do sooner rather than later...) |
|
Yea... I was figuring that out just now. I'm thinking of cutting it sometime this week. I'm basically going to be cherry picking the corresponding commits onto 20.3.3, so feel free to do whatever in the master branch. It won't really affect that work. I do think we'll need to create some sort of tracker for the legacy resolver removal though, so I've made a milestone for that. We'll want to populate that soon enough and let downstream users who're being noisy know that. Also, I just realized that I'm supposed to have dinner too, so, well, wheeeeeee! |
pradyunsg
added a commit
to pradyunsg/pip
that referenced
this pull request
Jan 23, 2021
…tracking Avoid downloading multiple candidates of a same version
jsirois
added a commit
to pex-tool/pip
that referenced
this pull request
Feb 1, 2021
* Merge pull request pypa#9289 from uranusjr/new-resolver-lazy-insert * Merge pull request pypa#9315 from pradyunsg/better-search-errors * Merge pull request pypa#9369 from uranusjr/resolvelib-054 * Merge pull request pypa#9320 from uranusjr/wheel-check-valid Verify built wheel contains valid metadata * Merge pull request pypa#9432 from uranusjr/new-resolver-dedup-on-backtracking Avoid downloading multiple candidates of a same version * Revert "Remove on_returncode parameter from call_subprocess" This reverts commit ab3ee71. * Revert "Remove show_stdout from run_command args" This reverts commit 94882fd. * Revert "Create call_subprocess just for vcs commands" This reverts commit 8adbc21. * Revert "Improve check for svn version string" This reverts commit 1471897. * Revert "Bubble up SubProcessError to basecommand._main" This reverts commit e9f738a. * Additional revert of 7969 Revert additional changes that were made after 7969 and depended on it. * add stdout_only to call_subprocess * vcs: capture subprocess stdout only * Add test for call_subprocess stdout_only * Bump for release * Fix test bitrot. Setuptools released 52.0.0 which killed easy_install support and thus caused tests exercising easy_install to fail. Co-authored-by: Pradyun Gedam <3275593+pradyunsg@users.noreply.github.com> Co-authored-by: Stéphane Bidoul <stephane.bidoul@gmail.com> Co-authored-by: Pradyun Gedam <pradyunsg@users.noreply.github.com>
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Since 41a3008 (#9289), Candidate objects prepare their underlying distribution eagerly on creation, so the error can be caught deterministically to trigger backtracking.
This however has a negative side-effect. Since dist preparation involves metadata validation, a remote distribution must be downloaded on candidate creation. This means that an sdist will be downloaded, validated, and discarded (since its version is known to be incompatible) during backtracking.
This commit moves version deduplication of candidates from indexes to before the Candidate object is created, to avoid unneeded preparation.
Note that we still need another round of deduplication in
FoundCandidatesto remove duplicated candidates when a distribution is already installed.News fragment omitted since the commit introducing this issue has not been released.