Add more context for Job errors. #2479
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This makes several error paths that manifest when creating locks and PEXes present better error details. Several changes support this: 1. Jobs can now have a context label which is used, when set, to prefix all job error output. This should help root cause the problem; i.e.: is it fundamentally Pex or is it Pip or is it a problem with a bad package? 2. When a Pip download errors, we now show all STDERR output instead of just the last line whenever the last line contains "See above for details", which Pip emits to indicate an error buried deeper in the logs. 3. A new log analyzer is added to all Pip downloads that knows how to spot sdist build errors and surface their details. 4. A fail-safe is added to the Pip download log scraper to just report the full Pip log content when no better analysis was found. Fixes pex-tool#2113
jsirois
requested review from
zmanji,
cburroughs,
kaos,
benjyw,
huonw and
sureshjoshi
jsirois
commented
Jul 28, 2024
| @@ -211,7 +210,7 @@ def bdist(self): | |||
| # type: () -> str | |||
| get_pip( | |||
| interpreter=self._interpreter, | |||
| resolver=ConfiguredResolver(pip_configuration=PipConfiguration()), | |||
| resolver=ConfiguredResolver.default(), | |||
There was a problem hiding this comment.
There are several ConfiguredResolver.default() changes like this in the tests below. This makes those test more robust to combinations not run in CI; I hit a few when testing out this change.
jsirois
commented
Jul 28, 2024
| @@ -137,6 +137,9 @@ commands = | |||
|
|
|||
| [testenv:{typecheck,check}] | |||
| deps = | |||
| # We keep this compatible with Python 2.7 for the 2.7 type check. | |||
There was a problem hiding this comment.
Otherwise unrelated, but helps with dev env setup.
jsirois
commented
Jul 28, 2024
| @@ -1566,45 +1565,6 @@ def __repr__(self): | |||
| ) | |||
|
|
|||
|
|
|||
| def spawn_python_job( | |||
There was a problem hiding this comment.
This was only used in a test - now inlined.
jsirois
commented
Jul 28, 2024
| ResolverVersion.default() is ResolverVersion.PIP_2020, | ||
| reason=( | ||
| "The PIP_2020 resolver triggers download analysis in normal resolves but this test is " | ||
| "concerned with the case when there is no analysis to be performed." |
There was a problem hiding this comment.
There is now always Pip download log analysis; so the test was obsolete.
benjyw
approved these changes
Jul 28, 2024
huonw
approved these changes
Jul 28, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This makes several error paths that manifest when creating locks and
PEXes present better error details.
Several changes support this:
all job error output. This should help root cause the problem; i.e.:
is it fundamentally Pex or is it Pip or is it a problem with a bad
package?
just the last line whenever the last line contains "See above for
details", which Pip emits to indicate an error buried deeper in the
logs.
spot sdist build errors and surface their details.
the full Pip log content when no better analysis was found.
Fixes #2113