use .metadata distribution info when possible #11512
Conversation
When performing `install --dry-run` and PEP 658 .metadata files are available to guide the resolve, do not download the associated wheels. Rather use the distribution information directly from the .metadata files when reporting the results on the CLI and in the --report file.
|
pre-commit.ci autofix |
for more information, see https://pre-commit.ci
|
The legacy resolver does not support the |
|
Hi 👋 With the warehouse PR now merged: is there animo to move this PR forward? cc @pradyunsg @pfmoore I would love to experience near-instant reports for big wheels like: I can also offer my support if that helps! |
|
This is not the PR you are looking for. PEP 658 support is alreay merged in #11111. |
|
I think without this PR, my command above will still download the torch wheel despite metadata being available. Or did I misunderstand? $ pip install --no-cache-dir --ignore-installed --dry-run --no-deps --report - botocore
Collecting botocore
Downloading botocore-1.29.138-py3-none-any.whl (10.8 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━━━━━━━━━━━ 7.5/10.8 MB 2.4 MB/s eta 0:00:02
ERROR: Operation cancelled by user |
|
@uranusjr is there a cli flag I'm missing in my command above, such that there is no wheel downloaded? that's what this PR achieves right: early return because the wheel is not needed for the report generation, due to PEP 658 now being live on both pip and warehouse? |
|
There's linked bugs in pip and warehouse - warehouse is serving the data under the wrong name so pip can't use it, but pip has a bug in its processing of the data so warehouse can't fix their bug. And even if pip fixes our bug, warehouse can't fix theirs as older pips will break in a way that means they can't upgrade. #12078 fixes the pip bug, but it also changes the name of the relevant field. Warehouse has a corresponding fix, but the name change needs a PEP which is currently in progress. |
|
Thanks a lot for your responses and efforts as always! |
cc @cosmicexplorer via #11478 (comment) |
|
I did my best guess to resolve the merge conflict (the result is at https://github.com/pypa/pip/compare/main...cosmicexplorer:pip:metadata_only_resolve_no_whl?expand=1) and at first glance I can confirm that it avoids downloading that wheel, which is absolutely what I was hoping to achieve with #11111: (ttt) cosmicexplorer@lightning-strike-teravolts:~/tools/pip$ pip install --no-cache-dir --ignore-installed --dry-run --no-deps --report out.json botocore
Collecting botocore
Obtaining dependency information for botocore from https://files.pythonhosted.org/packages/29/22/d3313243e0e09ff421edb249011dbb8093089de76b84b5ec3438f4c868eb/botocore-1.31.12-py3-none-any.whl.metadata
Downloading botocore-1.31.12-py3-none-any.whl.metadata (5.9 kB)
Downloading botocore-1.31.12-py3-none-any.whl (11.0 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 11.0/11.0 MB 15.3 MB/s eta 0:00:00
Would install botocore-1.31.12
(ttt) cosmicexplorer@lightning-strike-teravolts:~/tools/pip$ PYTHONPATH="$(pwd)/dist/pip-23.3.dev0-py3-none-any.whl" python -m pip install --no-cache-dir --ignore-installed --dry-run --no-deps --report out.json botocore
Collecting botocore
Obtaining dependency information for botocore from https://files.pythonhosted.org/packages/29/22/d3313243e0e09ff421edb249011dbb8093089de76b84b5ec3438f4c868eb/botocore-1.31.12-py3-none-any.whl.metadata
Downloading botocore-1.31.12-py3-none-any.whl.metadata (5.9 kB)
Would install botocore-1.31.12 |
There was a problem hiding this comment.
@jjhelmus it is always extremely helpful when fixing a bug to figure out how to add a test case to demonstrate that your code solves the stated problem, or if that's infeasible, to just describe what makes it difficult to do so (I am not a pip maintainer and do not speak for them). I am going to try to make such a test case now because I hope to extend this idea to short-circuit other parts of the resolve with local caching (see #12184). If I can figure out a good one, I'll probably put up my own PR.
| if dry_run: | ||
| self.factory.preparer.prepare_download_info(reqs) | ||
| else: | ||
| self.factory.preparer.prepare_linked_requirements_more(reqs) |
There was a problem hiding this comment.
I tried this to resolve the merge conflict, but I need to look into the change that introduced this:
| self.factory.preparer.prepare_linked_requirements_more(reqs) | |
| self.factory.preparer.prepare_linked_requirements_more(reqs) | |
| for req in reqs: | |
| req.prepared = True | |
| req.needs_more_preparation = False |
|
Closing in favour of #12186, although if I'm being realistic, neither PR is likely to land anytime soon due to limited maintainer time. Sorry. |
When performing
install --dry-runand PEP 658 .metadata files are available to guide the resolve, do not download the associated wheels.Rather use the distribution information directly from the .metadata files when reporting the results on the CLI and in the --report file.