test run stuck retrying with eager_fetch=False
#16096
Comments
|
The case in the description appears to be kicked off by a missing test output (for So: why doesn't the test consume the newly produced test result after retry? If it did, the infinite retry would not be an issue afaict, because the new test output would be consumed successfully. |
I have an (unsatisfying) answer to this portion: But that still doesn't actually answer how we're ending with a completely identical output for the second run:
I haven't been able to repro locally, so I think that my next step is to rule out one of the above by adding some additional debug output. |
|
I've posted #16277 to add debug output to differentiate the two cases above. If/when the issue occurs again, folks should post a link to the repro to this issue, and then feel free to set |
|
I've been out - is there any reason to be leaning on "if" and the possibility of red main again? When I last left this was a pretty reliable main CI red. |
This is debug output (and labels itself as such) to assist with differentiating the cases of #16096. [ci skip-build-wheels]
|
No repros of this case yet in |
This is debug output (and labels itself as such) to assist with differentiating the cases of pantsbuild#16096. [ci skip-build-wheels]
|
Ironically (?) but helpfully, I actually got repros in my cherry-pick PR:
The debug output from #16277 doesn't actually trigger, which itself is useful data. Will resume this tomorrow. |
This is debug output (and labels itself as such) to assist with differentiating the cases of pantsbuild#16096. [ci skip-build-wheels]
|
So the relevant aspect of #16277 not triggering in the repro on #16327 is that it means that the
I'm still not able to reproduce this case in an integration test, which makes me suspect that it is related to the stack of Case 2 is relatively easy to confirm (...assuming a repro), so I've pushed #16342 to try and get data on "what" is being invalidated in the relevant cases. As with #16277, if it doesn't repro while landing the PR, I'll likely try cherry-picking it. |
This is debug output (and labels itself as such) to assist with differentiating the cases of pantsbuild#16096. [ci skip-build-wheels]
This is debug output (and labels itself as such) to assist with differentiating the cases of pantsbuild#16096. [ci skip-rust] [ci skip-build-wheels]
…#16398) As reported in #16096, backtracking still has some kinks to work out, and is unlikely to be fully stable before `2.13.0` ships. To gain "most" of the network-usage benefit of deferring fetching cache content while significantly narrowing the window in which backtracking is necessary, this change replaces the `remote_cache_eager_fetch` flag with a `cache_content_behavior` ternary option. As described in the `help` for the new option, the modes are: * `fetch`: Eagerly fetches cache content: the default. Equivalent to `eager_fetch=True`, the previous default. * `validate`: Validates that cache content exists either locally or remotely, without actually fetching files. This cannot be the default (without changing behavior), because it can still require backtracking if cache content expires during the lifetime of `pantsd` or a single run. * `defer`: Does not validate or fetch content: equivalent to `eager_fetch=False`. Hopefully can eventually be made the default once all issues like #16096 are resolved. Because `validate` narrows the window in which we backtrack to cases where content expires during a run, it should be relatively safe for use even within the `2.13.x` release. [ci skip-build-wheels]
…pantsbuild#16398) As reported in pantsbuild#16096, backtracking still has some kinks to work out, and is unlikely to be fully stable before `2.13.0` ships. To gain "most" of the network-usage benefit of deferring fetching cache content while significantly narrowing the window in which backtracking is necessary, this change replaces the `remote_cache_eager_fetch` flag with a `cache_content_behavior` ternary option. As described in the `help` for the new option, the modes are: * `fetch`: Eagerly fetches cache content: the default. Equivalent to `eager_fetch=True`, the previous default. * `validate`: Validates that cache content exists either locally or remotely, without actually fetching files. This cannot be the default (without changing behavior), because it can still require backtracking if cache content expires during the lifetime of `pantsd` or a single run. * `defer`: Does not validate or fetch content: equivalent to `eager_fetch=False`. Hopefully can eventually be made the default once all issues like pantsbuild#16096 are resolved. Because `validate` narrows the window in which we backtrack to cases where content expires during a run, it should be relatively safe for use even within the `2.13.x` release. [ci skip-build-wheels]
… (Cherry-pick of #16398) (#16409) As reported in #16096, backtracking still has some kinks to work out, and is unlikely to be fully stable before `2.13.0` ships. To gain "most" of the network-usage benefit of deferring fetching cache content while significantly narrowing the window in which backtracking is necessary, this change replaces the `remote_cache_eager_fetch` flag with a `cache_content_behavior` ternary option. As described in the `help` for the new option, the modes are: * `fetch`: Eagerly fetches cache content: the default. Equivalent to `eager_fetch=True`, the previous default. * `validate`: Validates that cache content exists either locally or remotely, without actually fetching files. This cannot be the default (without changing behavior), because it can still require backtracking if cache content expires during the lifetime of `pantsd` or a single run. * `defer`: Does not validate or fetch content: equivalent to `eager_fetch=False`. Hopefully can eventually be made the default once all issues like #16096 are resolved. Because `validate` narrows the window in which we backtrack to cases where content expires during a run, it should be relatively safe for use even within the `2.13.x` release. [ci skip-build-wheels]
…pantsbuild#16398) As reported in pantsbuild#16096, backtracking still has some kinks to work out, and is unlikely to be fully stable before `2.13.0` ships. To gain "most" of the network-usage benefit of deferring fetching cache content while significantly narrowing the window in which backtracking is necessary, this change replaces the `remote_cache_eager_fetch` flag with a `cache_content_behavior` ternary option. As described in the `help` for the new option, the modes are: * `fetch`: Eagerly fetches cache content: the default. Equivalent to `eager_fetch=True`, the previous default. * `validate`: Validates that cache content exists either locally or remotely, without actually fetching files. This cannot be the default (without changing behavior), because it can still require backtracking if cache content expires during the lifetime of `pantsd` or a single run. * `defer`: Does not validate or fetch content: equivalent to `eager_fetch=False`. Hopefully can eventually be made the default once all issues like pantsbuild#16096 are resolved. Because `validate` narrows the window in which we backtrack to cases where content expires during a run, it should be relatively safe for use even within the `2.13.x` release. [ci skip-build-wheels]
|
Note that this issue can still occur with |
|
Also very related is: #16667. |
https://github.com/pantsbuild/pants/runs/7244788636?check_suite_focus=true
The text was updated successfully, but these errors were encountered: