-
-
Notifications
You must be signed in to change notification settings - Fork 271
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
make use of pip install --report JSON output #2210
Comments
Over slack (https://pantsbuild.slack.com/archives/C087V4P1T/p1691343949034449), @jsirois urged me to look at the prior investigation by @thejcannon, with discussion at https://pantsbuild.slack.com/archives/C087V4P1T/p1688051841183419 and #2044 (comment). In particular, @jsirois raised the possibility of making use of |
In particular, I was advised by pip maintainers (see pypa/pip#12184 (comment)) to approach the metadata lookup caching sketched out in pypa/pip#12184 as a plugin to |
@thejcannon's prior branch testing this is at https://github.com/thejcannon/pex/tree/jcannon/pip-report. |
In my testing, the only large red flag was that VCS reqs in PEX are hashed via their downloaded zip. |
After quite a long saga (pypa/pip#53), pip has the
--report=<out.json>
option topip install
(see pypa/pip#10771). This can be combined with--ignore-installed
and--dry-run
to produce a resolve report specifically for the uses of tools like pex. There are some further changes in flight to make this metadata-only resolve significantly faster by avoiding any downloads at all (see pypa/pip#12186), and plans to get it down to almost instantaneous by caching metadata lookups (pypa/pip#12184). With the--use-feature=fast-deps
option, these improvements also apply to resolves against wheels in a--find-links
index or a pypi-like index that hasn't yet implemented PEP 658 (pypi itself has only just now enabled it).One use case where this shines is lock file creation. A prototype I made incorporating a few of the mentioned in-progress changes exposes a function
pex.resolver.resolve_new()
to executepip install --report
, but with otherwise the same arguments asresolve()
: https://github.com/pantsbuild/pex/compare/main...cosmicexplorer:pip-json-resolve?expand=1. Without any of the work from pypa/pip#12184, this halves the time pex spends within pip when creating a lockfile:Executing pex with sufficient verbosity confirms that >15 seconds of that pex process is spent within pip. In the uncached case, we still do better, at 26s for
resolve_new()
in the prototype branch vs 43s forpex lock create
on main.While looking to incorporate these changes, I found that
pex3 lock create
currently scans the output ofpip download
to extract hashes and download locations, which are contained in the current--report
json output. I didn't want to spend the time replacing that yet, but I suspect leaning on the metadata-only resolve json will make the implementation ofpex3 lock
easier to follow.Remaining tasks (for the prototype branch at https://github.com/cosmicexplorer/pex/tree/pip-json-resolve):
PipVersionValue
to select pip versions that support--report
.resolve_new()
to something likemetadata_only_resolve()
.lock {create,update}
consumemetadata_only_resolve()
.PipVersionValue
to keep up to speed with performance improvements, otherwise defaulting to the current implementation which scans output logs when the latest pip version does not support--report
.The text was updated successfully, but these errors were encountered: