Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Backtrack for network issues as well as missing digests #16667

Open
stuhood opened this issue Aug 26, 2022 · 1 comment
Open

Backtrack for network issues as well as missing digests #16667

stuhood opened this issue Aug 26, 2022 · 1 comment

Comments

@stuhood
Copy link
Member

stuhood commented Aug 26, 2022

#11331 introduced backtracking for missing digests. But with either cache_content_behavior={defer,validate}, a network issue or error can also cause the need to backtrack.

This can exhibit as either:

  1. Store accesses explicitly failing (due to unretryable errors like authentication errors, etc)
  2. Store accesses getting tarpitted in retries, and consequently slowing down local process runs (which cannot fetch their inputs)

In either {defer,validate}, we can't skip or speculate these Store accesses, because we only fall back to accessing the remote if the local store doesn't have the content, meaning that the content only exists remotely. But we should be able to backtrack to the original source of the Digest and produce it locally instead.

@stuhood
Copy link
Member Author

stuhood commented Dec 7, 2022

We see these fairly often in pantsbuild CI, unfortunately. Will disable validate for now.

stuhood added a commit that referenced this issue Dec 7, 2022
Pantsbuild CI experiences #16667 fairly frequently (where a long running job hits an authentication error), and until we have resolved that issue, we don't want to continue subject contributors to flaky failures.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant