Regression in _populate_deps(?) for --venv PEXes
#2088
Assignees
Labels
Comments
huonw
changed the title
_populate_deps for --venv PEXes_populate_deps(?) for --venv PEXes
|
You bisected the right change. As you may notice, that change purports to be a fix for loose layout PEXes - which you don't use. So likely the Pex misidentifes itself. diff --git a/pex/pex.py b/pex/pex.py
index eebab84..feb50d4 100644
--- a/pex/pex.py
+++ b/pex/pex.py
@@ -172,6 +172,7 @@ class PEX(object): # noqa: T000
# type: () -> Layout.Value
if self._layout is None:
self._layout = Layout.identify(self._pex)
+ print(">>> I'm {} and my layout is: {}".format(self._pex, self._layout), file=sys.stderr)
return self._layout
def pex_info(self, include_env_overrides=True):Shows: I'm ~afk through March 14th and will fix this then if no-one else has. But that's the issue. |
jsirois
added a commit
to jsirois/pex
that referenced
this issue
Mar 9, 2023
The change in pex-tool#2033 that safeguarded loose `--venv` PEXes such that they could be executed, moved and continue to execute properly from their new location. That fix silently broke zipapp and packed layout `--venv` PEX symlinking. Although `--venv` PEXes for those layouts continued to work, they used copying to build their venvs instead of symlinking even when symlinking (the default for `--venv` PEXes) was the intended style. Fixes pex-tool#2088
jsirois
added a commit
that referenced
this issue
Mar 9, 2023
The change in #2033 that safeguarded loose `--venv` PEXes such that they could be executed, moved and continue to execute properly from their new location, silently broke zipapp and packed layout `--venv` PEX symlinking. Although `--venv` PEXes for those layouts continued to work, they used copying to build their venvs instead of symlinking even when symlinking (the default for `--venv` PEXes) was the intended style. Fixes #2088
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
It seems f723302 was a significant performance regression for
--venvPEXes,especially for PEXes with many small files. Potentially it's a correctness fix for which we just have to eat the performance impact. In addition to a regression, this time doesn't seem to be bracketed by any of the tracing spans, and so isn't obvious in logs.The table compares the cold-cache start-up time for two packages that are both fairly large, but with drastically different numbers of files. The table compares 2.1.119 and 2.1.120, but I separately bisected the regression to f723302.
semgrep==1.14.0plotly==5.13.1Reproducer, using
ts -ito record the incremental timings in text (leading number on each line of the output):Output for plotly, with 2.1.119:
Output for plotly with 2.1.120 is broadly the same, except for the last few lines:
In particular there's 4.1s of extra work happening silently (even at
PEX_VERBOSE=9) after the last "Re-writing" line.A
py-spyflame graph of a similar PEX (built with 2.1.125) hints at some possibilities for the location of the regression. Unfortunately, I haven't been able to get this recorded reliably, so it's not the PEX in the reproducer above, nor do I have a trace of 2.1.119 to compare.(As always, thanks for PEX; it is a reliable workhorse in our deploys. 👍 )
The text was updated successfully, but these errors were encountered: