Unify resolve and runtime wheel caches. #820
Assignees
Labels
Comments
jsirois
added a commit
to jsirois/pex
that referenced
this issue
Dec 4, 2019
Previously these caches were seperate. Downloaded wheels and sdists were cached in `~/.pex/build` and wheels unzipped from zipped pexes at runtime were cached to `~/.pex/install`. Now the caches are unified by the resolver such that any wheel installs performed by it can be seen by zipped PEXes on the same machine when they go to potentially unzip wheel distributions stored within at PEX boot time. N.B.: The cache is not unified in the other direction. If a zipped PEX is executed on the same machine a PEX build resolve later happens on, any intersecting wheels will be re-downloaded, built and installed by the resolve finally unifying the caches from that point forward. Fixes pex-tool#820
jsirois
added a commit
to jsirois/pex
that referenced
this issue
Dec 4, 2019
Previously these caches were seperate. Downloaded wheels and sdists were cached in `~/.pex/build` and wheels unzipped from zipped pexes at runtime were cached to `~/.pex/install`. Now the caches are unified by the resolver such that any wheel installs performed by it can be seen by zipped PEXes on the same machine when they go to potentially unzip wheel distributions stored within at PEX boot time. N.B.: The cache is not unified in the other direction. If a zipped PEX is executed on the same machine a PEX build resolve later happens on, any intersecting wheels will be re-downloaded, built and installed by the resolve finally unifying the caches from that point forward. Fixes pex-tool#820
jsirois
added a commit
to jsirois/pex
that referenced
this issue
Dec 4, 2019
Previously these caches were seperate. Downloaded wheels and sdists were cached in `~/.pex/build` and wheels unzipped from zipped pexes at runtime were cached to `~/.pex/install`. Now the caches are unified by the resolver such that any wheel installs performed by it can be seen by zipped PEXes on the same machine when they go to potentially unzip wheel distributions stored within at PEX boot time. N.B.: The cache is not unified in the other direction. If a zipped PEX is executed on the same machine a PEX build resolve later happens on, any intersecting wheels will be re-downloaded, built and installed by the resolve finally unifying the caches from that point forward. Fixes pex-tool#820
jsirois
added a commit
to jsirois/pex
that referenced
this issue
Dec 4, 2019
Previously these caches were seperate. Downloaded wheels and sdists were cached in `~/.pex/build` and wheels unzipped from zipped pexes at runtime were cached to `~/.pex/install`. Now the caches are unified by the resolver such that any wheel installs performed by it can be seen by zipped PEXes on the same machine when they go to potentially unzip wheel distributions stored within at PEX boot time. N.B.: The cache is not unified in the other direction. If a zipped PEX is executed on the same machine a PEX build resolve later happens on, any intersecting wheels will be re-downloaded, built and installed by the resolve finally unifying the caches from that point forward. Fixes pex-tool#820
jsirois
added a commit
to jsirois/pex
that referenced
this issue
Dec 6, 2019
Previously these caches were seperate. Downloaded wheels and sdists were cached in `~/.pex/build` and wheels unzipped from zipped pexes at runtime were cached to `~/.pex/install`. Now the caches are unified by the resolver such that any wheel installs performed by it can be seen by zipped PEXes on the same machine when they go to potentially unzip wheel distributions stored within at PEX boot time. N.B.: The cache is not unified in the other direction. If a zipped PEX is executed on the same machine a PEX build resolve later happens on, any intersecting wheels will be re-downloaded, built and installed by the resolve finally unifying the caches from that point forward. Fixes pex-tool#820
jsirois
added a commit
to jsirois/pex
that referenced
this issue
Dec 6, 2019
Previously these caches were seperate. Downloaded wheels and sdists were cached in `~/.pex/build` and wheels unzipped from zipped pexes at runtime were cached to `~/.pex/install`. Now the caches are unified by the resolver such that any wheel installs performed by it can be seen by zipped PEXes on the same machine when they go to potentially unzip wheel distributions stored within at PEX boot time. N.B.: The cache is not unified in the other direction. If a zipped PEX is executed on the same machine a PEX build resolve later happens on, any intersecting wheels will be re-downloaded, built and installed by the resolve finally unifying the caches from that point forward. Fixes pex-tool#820
jsirois
added a commit
to jsirois/pex
that referenced
this issue
Dec 6, 2019
Previously these caches were seperate. Downloaded wheels and sdists were cached in `~/.pex/build` and wheels unzipped from zipped pexes at runtime were cached to `~/.pex/install`. Now the caches are unified by the resolver such that any wheel installs performed by it can be seen by zipped PEXes on the same machine when they go to potentially unzip wheel distributions stored within at PEX boot time. N.B.: The cache is not unified in the other direction. If a zipped PEX is executed on the same machine a PEX build resolve later happens on, any intersecting wheels will be re-downloaded, built and installed by the resolve finally unifying the caches from that point forward. Fixes pex-tool#820
jsirois
added a commit
to jsirois/pex
that referenced
this issue
Dec 6, 2019
Previously these caches were seperate. Downloaded wheels and sdists were cached in `~/.pex/build` and wheels unzipped from zipped pexes at runtime were cached to `~/.pex/install`. Now the caches are unified by the resolver such that any wheel installs performed by it can be seen by zipped PEXes on the same machine when they go to potentially unzip wheel distributions stored within at PEX boot time. N.B.: The cache is not unified in the other direction. If a zipped PEX is executed on the same machine a PEX build resolve later happens on, any intersecting wheels will be re-downloaded, built and installed by the resolve finally unifying the caches from that point forward. Fixes pex-tool#820
jsirois
added a commit
to jsirois/pex
that referenced
this issue
Dec 6, 2019
Previously these caches were seperate. Downloaded wheels and sdists were cached in `~/.pex/build` and wheels unzipped from zipped pexes at runtime were cached to `~/.pex/install`. Now the caches are unified by the resolver such that any wheel installs performed by it can be seen by zipped PEXes on the same machine when they go to potentially unzip wheel distributions stored within at PEX boot time. N.B.: The cache is not unified in the other direction. If a zipped PEX is executed on the same machine a PEX build resolve later happens on, any intersecting wheels will be re-downloaded, built and installed by the resolve finally unifying the caches from that point forward. Fixes pex-tool#820
jsirois
added a commit
that referenced
this issue
Dec 6, 2019
Previously these caches were seperate. Downloaded wheels and sdists were cached in `~/.pex/build` and wheels unzipped from zipped pexes at runtime were cached to `~/.pex/install`. Now the caches are unified by the resolver such that any wheel installs performed by it can be seen by zipped PEXes on the same machine when they go to potentially unzip wheel distributions stored within at PEX boot time. N.B.: The cache is not unified in the other direction. If a zipped PEX is executed on the same machine a PEX build resolve later happens on, any intersecting wheels will be re-downloaded, built and installed by the resolve finally unifying the caches from that point forward. Fixes #820 Also add a test demonstrating "ipex". The test demonstrates how to create a dehydrated pex that, upon first execution, produces a hydrated pex that it hands control to from then forward. This is the motivating use case for the cache unification change which prevents dehydrated pexes from performing wheel unzipping twice - once during pex hydration (resolve) and once during hydrated pex run.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
The discussion in #789 and continued in the associated design doc led to some spelunking and the realization that Pex currently has two distinct caches for wheels. One for the buildtime resolve of the wheel and one for the runtime extraction of the wheel from zipped pex files. By unifying these caches, we save disk space on machines where we both build and run wheels.
More interesting, apropos to #789, we gain the ability to populate the runtime wheel cache from a local build operation due to unification. For zipped pexes, this allows those pexes to fully resolve their requirements from the resolve-populated cache and thus not actually ever need any of those requirement wheels embedded in the zipped pex.
The text was updated successfully, but these errors were encountered: