Performance Tracking: generate-lockfiles
#21223
Comments
This is widely misunderstood. Pex does incremental updates of lock files, but it does this using a full Pip resolve which is exactly as fast as a normal lock create. This bug tracks doing an incremental resolve: pex-tool/pex#2044 These are two orthogonal things.
See here, but I tried this and found its still slow: pex-tool/pex#2375 (comment) |
I was just writing a comment on that ticket asking whether what we have today vs what the ticket refers to were comparable. So "incremental update" vs "incremental resolve" would be a better distinction on the pex side.
Ah, that sucks, but yeah - that was just an example I grabbed from one of the many slack/discussions/issues about this. |
|
Tracker of several pending pip performance improvements: pypa/pip#12184 |
|
Sorry, the correct tracker is now: pypa/pip#12921 |
The first version of pip to support Python 3.13 was 24.1, but I don't think there is any merit to holding back when 24.2 is already out. (Pants itself uses `latest`.) The first version of Pex to support 24.2 is v2.13.0. (Pip 24.1 and 24.2 also both contain meaningful performance improvements to dependency resolution --> pantsbuild#21223) Release notes for where pex added support: https://github.com/pex-tool/pex/releases/tag/v2.13.0 ref pantsbuild#20852
The first version of pip to support Python 3.13 was 24.1, but I don't think there is any merit to holding back when 24.2 is already out. (Pants itself uses `latest`.) The first version of Pex to support 24.2 is v2.13.0. (Pip 24.1 and 24.2 also both contain meaningful performance improvements to dependency resolution --> #21223) Release notes for where pex added support: https://github.com/pex-tool/pex/releases/tag/v2.13.0 ref #20852
|
Crosslinking some other pip items of possible relevance: |
|
It would be very helpful if you can report resolutions that are slow or problamatic on either https://github.com/notatallshaw/Pip-Resolution-Scenarios-and-Benchmarks or https://github.com/pypa/pip. Probably you would be best switching to uv (which I do also report lots of resolution issues to), but I do aim to keep improving pip's resolver. |
There is a common mention/complaint that calling
pants generate-lockfiles ::is slow. Creating this ticket to capture associated discussions, tickets, and even better, sample repos.pants generate-lockfilesto useuvfor dependency resolution #20679From my brief experiments, using pip 24.1 improves cold lockfile generation by about 50% on a small set of requirements, but does not affect incremental checking.
Some ideas that have been floated around:
uvto some degreefast-depsif we're not currently doing so (though, whether that will continue to be a standalone option: perform 1-3 HTTP requests for each wheel using fast-deps pypa/pip#12208)The text was updated successfully, but these errors were encountered: