This repository has been archived by the owner on Nov 1, 2023. It is now read-only.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Add a setup script-specific timeout of 59 minutes. This is just shorter than the service-side
NODE_EXPIRATION_TIME
which otherwise garbage collects nodes whose setup scripts are stuck or taking too long.Ideally, the timeout would be user-configurable within some range. As-is, we'd only ever want to increase it, and doing so would require dynamically updating the service-side limit (which we'd probably want to revert to something short, after the setup script is done.
With this change, the high-level cause of the timeout is clear, instead of the closest error being something indirect, like "node reimaged during task execution".
Tested by creating a Linux
libfuzzer basic
job with asetup.sh
that invokesyes >/dev/null
. All task VMs got stuck insetting_up
, then failed with an explicit error of "setup script timed out".Closes #1658.