Introduce `--pre-resolved-dists` resolver. #2512

jsirois · 2024-08-20T00:03:44Z

When building a PEX or creating a venv via pex3 venv create you can
now tell Pex to use a set of pre-resolved distributions. This is similar
to using --no-pypi --find-links ... except that:

Its roughly 3x faster since Pip is not asked to do any resolution.
It requires all the distributions specified form an already complete
resolve.

One way to obtain distributions that meet criteria 2 is to use
pip download -d ... or pip wheel -w ... to pre-resolve the
distributions you need.

Closes #1907

When building a PEX or creating a venv via `pex3 venv create` you can now tell Pex to use a set of pre-resolved distributions. This is similar to using `--no-pypi --find-links ...` except that: 1. Its roughly 3x faster since Pip is not asked to do any resolution. 2. It requires all the distributions specified form an already complete resolve. One way to obtain distributions that meet criteria 2 is to use `pip download -d ...` or `pip wheel -w ...` to pre-resolve the distributions you need. Closes pex-tool#1907

jsirois · 2024-09-13T19:18:02Z

On the perf side, I ran 2 tests. One for torch (huge), one for devpi-server (small). Each of the tests tested 3 styles of resolve:

--no-pypy --find-links transitive resolve using 1 root requirement.
--no-pypy --find-links intransitive resolve using the local downloaded dists themselves as the requirements.
The new --pre-resolved-dists resolver using 1 root requirement.

torch

:; hyperfine -w2 -n 'intransitive wheels' -n 'pre-resolved-dists' -n 'find-links' 'python3.11 -mpex --pip-version latest --no-pypi -f ~/support/pex/issues/1907/wheels ~/support/pex/issues/1907/wheels/* --intransitive -otorch-pip-whls.pex --no-pre-install-wheels --layout packed' 'python3.11 -mpex --pre-resolved-dists ~/support/pex/issues/1907/wheels torch -otorch-pre-resolved-dists.pex --no-pre-install-wheels --layout packed' 'python3.11 -mpex --pip-version latest --no-pypi -f ~/support/pex/issues/1907/wheels torch -otorch-fl.pex --no-pre-install-wheels --layout packed'
Benchmark 1: intransitive wheels
  Time (mean ± σ):      7.330 s ±  0.122 s    [User: 5.415 s, System: 1.911 s]
  Range (min … max):    7.214 s …  7.603 s    10 runs

Benchmark 2: pre-resolved-dists
  Time (mean ± σ):      2.414 s ±  0.066 s    [User: 3.350 s, System: 0.561 s]
  Range (min … max):    2.360 s …  2.574 s    10 runs

Benchmark 3: find-links
  Time (mean ± σ):      7.364 s ±  0.114 s    [User: 5.333 s, System: 1.965 s]
  Range (min … max):    7.203 s …  7.560 s    10 runs

Summary
  pre-resolved-dists ran
    4.04 ± 0.10 times faster than intransitive wheels
    3.05 ± 0.10 times faster than find-links

:; du -sh torch*.pex
3.7G    torch-fl.pex
2.7G    torch-pip-whls.pex
2.7G    torch-pre-resolved-dists.pex

:; ls -1sh --sort size ~/support/pex/issues/1907/wheels/
total 2.7G
761M torch-2.4.1-cp311-cp311-manylinux1_x86_64.whl
634M nvidia_cudnn_cu12-9.1.0.70-py3-none-manylinux2014_x86_64.whl
392M nvidia_cublas_cu12-12.1.3.1-py3-none-manylinux1_x86_64.whl
200M triton-3.0.0-1-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.whl
187M nvidia_cusparse_cu12-12.1.0.106-py3-none-manylinux1_x86_64.whl
169M nvidia_nccl_cu12-2.20.5-py3-none-manylinux2014_x86_64.whl
119M nvidia_cusolver_cu12-11.4.5.107-py3-none-manylinux1_x86_64.whl
117M nvidia_cufft_cu12-11.0.2.54-py3-none-manylinux1_x86_64.whl
 54M nvidia_curand_cu12-10.3.2.106-py3-none-manylinux1_x86_64.whl
 23M nvidia_cuda_nvrtc_cu12-12.1.105-py3-none-manylinux1_x86_64.whl
 19M nvidia_nvjitlink_cu12-12.6.68-py3-none-manylinux2014_x86_64.whl
 14M nvidia_cuda_cupti_cu12-12.1.105-py3-none-manylinux1_x86_64.whl
6.0M sympy-1.13.2-py3-none-any.whl
2.3M setuptools-71.1.0-py3-none-any.whl
1.8M pip-24.2-py3-none-any.whl
1.7M networkx-3.3-py3-none-any.whl
808K nvidia_cuda_runtime_cu12-12.1.105-py3-none-manylinux1_x86_64.whl
524K mpmath-1.3.0-py3-none-any.whl
176K fsspec-2024.9.0-py3-none-any.whl
132K jinja2-3.1.4-py3-none-any.whl
100K nvidia_nvtx_cu12-12.1.105-py3-none-manylinux1_x86_64.whl
 68K wheel-0.43.0-py3-none-any.whl
 40K typing_extensions-4.12.2-py3-none-any.whl
 28K MarkupSafe-2.1.5-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
 16K filelock-3.16.0-py3-none-any.whl

devpi-server

:; hyperfine -w2 -n 'intransitive wheels' -n 'pre-resolved-dists' -n 'find-links' 'python3.11 -mpex --pip-version latest --no-pypi -f ~/support/pex/issues/1907/wheels-devpi-server $(find ~/support/pex/issues/1907/wheels-devpi-server/ -name "*.whl" | grep -v -E "/(wheel|pip)-") --intransitive -odevpi-server-pip-whls.pex --no-pre-install-wheels' 'python3.11 -mpex --pre-resolved-dists ~/support/pex/issues/1907/wheels-devpi-server devpi-server -odevpi-server-pre-resolved-dists.pex --no-pre-install-wheels' 'python3.11 -mpex --pip-version latest --no-pypi -f ~/support/pex/issues/1907/wheels-devpi-server devpi-server -odevpi-server-fl.pex --no-pre-install-wheels'
Benchmark 1: intransitive wheels
  Time (mean ± σ):     995.8 ms ±   5.6 ms    [User: 897.1 ms, System: 70.9 ms]
  Range (min … max):   986.1 ms … 1004.1 ms    10 runs

Benchmark 2: pre-resolved-dists
  Time (mean ± σ):     385.0 ms ±   4.6 ms    [User: 431.9 ms, System: 73.8 ms]
  Range (min … max):   378.2 ms … 392.8 ms    10 runs

Benchmark 3: find-links
  Time (mean ± σ):      1.023 s ±  0.010 s    [User: 0.931 s, System: 0.070 s]
  Range (min … max):    1.008 s …  1.046 s    10 runs

Summary
  pre-resolved-dists ran
    2.59 ± 0.03 times faster than intransitive wheels
    2.66 ± 0.04 times faster than find-links

:; ls -1sh devpi-server*.pex
7.3M devpi-server-fl.pex
7.3M devpi-server-pip-whls.pex
7.3M devpi-server-pre-resolved-dists.pex

:; ls -1sh --sort size ~/support/pex/issues/1907/wheels-devpi-server/
total 8.5M
2.3M setuptools-71.1.0-py3-none-any.whl
1.8M pip-24.2-py3-none-any.whl
532K ruamel.yaml.clib-0.2.8-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.manylinux_2_24_x86_64.whl
516K passlib-1.7.4-py2.py3-none-any.whl
460K cffi-1.17.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
272K devpi_server-6.12.1-py3-none-any.whl
256K zope.interface-7.0.3-cp311-cp311-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl
244K pyramid-2.0.2-py3-none-any.whl
228K python_dateutil-2.9.0.post0-py2.py3-none-any.whl
164K certifi-2024.8.30-py3-none-any.whl
140K charset_normalizer-3.3.2-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
124K urllib3-2.2.3-py3-none-any.whl
124K strictyaml-1.7.3-py3-none-any.whl
116K ruamel.yaml-0.18.6-py3-none-any.whl
116K pycparser-2.22-py3-none-any.whl
116K WebOb-1.8.8-py2.py3-none-any.whl
100K py-1.11.0-py2.py3-none-any.whl
 88K anyio-4.4.0-py3-none-any.whl
 88K argon2_cffi_bindings-21.2.0-cp36-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
 80K httpcore-1.0.5-py3-none-any.whl
 76K httpx-0.27.2-py3-none-any.whl
 68K idna-3.8-py3-none-any.whl
 68K wheel-0.43.0-py3-none-any.whl
 64K requests-2.32.3-py3-none-any.whl
 64K attrs-24.2.0-py3-none-any.whl
 60K h11-0.14.0-py3-none-any.whl
 56K waitress-3.0.0-py3-none-any.whl
 56K packaging-24.1-py3-none-any.whl
 28K defusedxml-0.7.1-py2.py3-none-any.whl
 24K hupper-1.12.1-py3-none-any.whl
 24K pluggy-1.5.0-py3-none-any.whl
 20K platformdirs-4.3.2-py3-none-any.whl
 20K PasteDeploy-3.1.0-py3-none-any.whl
 16K itsdangerous-2.2.0-py3-none-any.whl
 16K devpi_common-4.0.4-py3-none-any.whl
 16K argon2_cffi-23.1.0-py3-none-any.whl
 16K translationstring-1.4-py2.py3-none-any.whl
 16K venusian-3.1.0-py3-none-any.whl
 12K plaster-1.1.2-py2.py3-none-any.whl
 12K six-1.16.0-py2.py3-none-any.whl
 12K repoze.lru-0.7-py3-none-any.whl
 12K sniffio-1.3.1-py3-none-any.whl
 12K zope.deprecation-5.0-py3-none-any.whl
 12K packaging_legacy-23.0.post0-py3-none-any.whl
8.0K plaster_pastedeploy-1.0.1-py2.py3-none-any.whl
8.0K lazy-1.6-py2.py3-none-any.whl

jsirois · 2024-09-13T19:21:40Z

@hrfuller I think this solves the last discrepancy in #1907. Although you can't add dists via the Pex API, you can do so using this new --pre-resolved-dists. I know you're now separated from your original use case; so can't test the speed difference, but it should be ~same. I included some perf test results above at any rate. One advantage over adding dists via the API is that this supports sub-setting; i.e. point at one big consistent resolve pool of dists with --pre-resolved-dists ... and you can resolve the whole thing or any subset by just changing the root requirement list you hand Pex.

jsirois · 2024-09-13T19:25:26Z

Pants folks, I added you since this may be one way to get uv in the loop, since its a way to have Pex step out of the resolve loop. Although neither uv pip download nor uv pip wheel exist today, they don't sound completely opposed. They just need someone to do the work.

jsirois · 2024-09-13T19:26:58Z

@zmanji this might be a way for you to get offline Pex support, re: #2201 (comment)

jsirois · 2024-09-13T21:28:25Z

N.B.: The bulk of the delta lines is a test lockfile at ~1800 lines.

zmanji · 2024-09-14T02:28:49Z

@zmanji this might be a way for you to get offline Pex support, re: #2201 (comment)

Yes with this flag pex can work in a docker container without internet access or with nix or similar.

I am assuming the workflow if you have a lockfile will be:

Export the lock to requirements.txt, creating a subset if needed
Use pip to download all of the artifacts.
Use --pre-resolved-dists with the requirements.txt to build the resulting pex.

jsirois · 2024-09-14T02:56:17Z

@zmanji yes, exactly. The tests do this here: https://github.com/pex-tool/pex/pull/2512/files#diff-3aad399daa6b77a0c3e83cdca2225461ef07f4aebe306abdb80ed4b8bfd2fb19R33-R66

You do have to ensure that either the download or the original lock, though, is 100% wheels of course.
If your host platform happens to match the container, easy. If not, you can try pip wheel ... instead of pip download ..., which will build any sdist-onlies on the host. If those turn out py3-none-any wheels or the like, you get lucky. In Pex, this is the yolo mode that was introduced in #2075 via #2073.

... or you can also pre-resolve the --pip-version ... you use as well as any (transitive) build backend wheels you need for sdists. A test goes through this exercise here: https://github.com/pex-tool/pex/pull/2512/files#diff-3aad399daa6b77a0c3e83cdca2225461ef07f4aebe306abdb80ed4b8bfd2fb19R160-R200

zmanji

Overall the code/tests and idea lgtm!

The addition of wheel tag compatibility to the resolve chech adding in pex-tool#2512 regressed users of abbreviated `--platform` in some cases by failing PEX builds that would otherwise succeed and, later, actually work at runtime. Keep the spirit of pex-tool#2512 by emitting a detailed warning at build time with remediation steps instead of failing the build outright. Fixes pex-tool#2532

The addition of a wheel tag compatibility check to the overall post-resolve check in #2512 regressed users of abbreviated `--platform` in some cases by failing PEX builds that would otherwise succeed and, later, actually work at runtime. Keep the spirit of #2512 by emitting a detailed warning at build time with remediation steps instead of failing the build outright. Fixes #2532

jsirois force-pushed the issues/1907 branch from 625b7cb to d638a95 Compare September 12, 2024 03:17

jsirois mentioned this pull request Sep 13, 2024

proposal: pex3 cache introspection/gc command #2201

Closed

jsirois force-pushed the issues/1907 branch from 8c0cd2b to e063da3 Compare September 13, 2024 19:06

jsirois changed the title ~~Support --add-dist(s) to separate resolve from Pex.~~ Support --pre-resolved-dists resolver. Sep 13, 2024

jsirois marked this pull request as ready for review September 13, 2024 19:18

jsirois requested review from zmanji, benjyw and sureshjoshi September 13, 2024 19:22

jsirois mentioned this pull request Sep 13, 2024

Consider using uv as an optional alternate resolver. #2371

Open

jsirois added 5 commits September 13, 2024 12:52

Fix Python 2.7 pickle issue.

8cbf5ab

Prepare the 2.19.0 release.

08276e3

Fix fmt.

797e61b

Re-lock for devpi-server 6.12.1 Python>=3.10,<3.14.

f8ed96a

Plumb Pip configuration & test full offline for sources.

4f86f5a

Fix 1/2 sort.

1d72629

jsirois changed the title ~~Support --pre-resolved-dists resolver.~~ Introduce --pre-resolved-dists resolver. Sep 14, 2024

zmanji approved these changes Sep 14, 2024

View reviewed changes

jsirois merged commit 5e689f9 into pex-tool:main Sep 15, 2024
26 checks passed

jsirois deleted the issues/1907 branch September 15, 2024 04:49

jsirois mentioned this pull request Sep 16, 2024

Warn when --platform resolves fail tag checks. #2533

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Introduce `--pre-resolved-dists` resolver. #2512

Introduce `--pre-resolved-dists` resolver. #2512

jsirois commented Aug 20, 2024 •

edited

Loading

jsirois commented Sep 13, 2024

jsirois commented Sep 13, 2024

jsirois commented Sep 13, 2024

jsirois commented Sep 13, 2024

jsirois commented Sep 13, 2024

zmanji commented Sep 14, 2024

jsirois commented Sep 14, 2024 •

edited

Loading

zmanji left a comment

Introduce --pre-resolved-dists resolver. #2512

Introduce --pre-resolved-dists resolver. #2512

Conversation

jsirois commented Aug 20, 2024 • edited Loading

jsirois commented Sep 13, 2024

torch

devpi-server

jsirois commented Sep 13, 2024

jsirois commented Sep 13, 2024

jsirois commented Sep 13, 2024

jsirois commented Sep 13, 2024

zmanji commented Sep 14, 2024

jsirois commented Sep 14, 2024 • edited Loading

zmanji left a comment

Choose a reason for hiding this comment

Introduce `--pre-resolved-dists` resolver. #2512

Introduce `--pre-resolved-dists` resolver. #2512

jsirois commented Aug 20, 2024 •

edited

Loading

jsirois commented Sep 14, 2024 •

edited

Loading