Reconnection may not have retry capacity #18764
Labels
help wanted
Someone outside the Bazel team could own this
P2
We'll consider working on this in future. (Assignee optional)
team-Remote-Exec
Issues and PRs for the Execution (Remote) team
type: bug
Comments
|
Confirmed. The SslHandshakeException is continuously generated for a given connection, without the elapsed time to indicate that it would timeout with retries. Additionally, the unwrapping that happens in ReferenceCountedChannel will discard the retriable UNAVAILABLE SRE, leaving a single chance for a server to respond within 10s to a build client on RemoteModule startup under TLS. |
|
@coeuvre ^ |
Pavank1992
added
the
team-Remote-Exec
zhengwei143
added
P2
|
Since you can reliably reproduce this error and it sounds easy to fix. Would you mind send us a PR? |
Happy to. Solution will come in two parts - unwrap fix, then whatever is causing the same error from socket. I've only got the first so far. |
copybara-service bot
pushed a commit
that referenced
this issue
Jul 6, 2023
Unwrapping all StatusRuntimeExceptions in in ReferenceCountedChannel when caused by IOException will discard critical tracing and retriability. The Retrier evaluations may not see an SRE in the causal chain, and presume it is invariably an unretriable exception. In general, IOExceptions as SRE wrappers are unsuitable containers and are routinely misued either for identification (grpc aware status), or capture (handleInitError). Partially addresses #18764 (retries will occur with SSL handshake timeout, but the actual connection will not be retried) Closes #18836. PiperOrigin-RevId: 546037698 Change-Id: I7f6efcb857c557aa97ad3df085fc032c8538eb9a
werkt
added a commit
to werkt/bazel
that referenced
this issue
Jul 17, 2023
Unwrapping all StatusRuntimeExceptions in in ReferenceCountedChannel when caused by IOException will discard critical tracing and retriability. The Retrier evaluations may not see an SRE in the causal chain, and presume it is invariably an unretriable exception. In general, IOExceptions as SRE wrappers are unsuitable containers and are routinely misued either for identification (grpc aware status), or capture (handleInitError). Partially addresses bazelbuild#18764 (retries will occur with SSL handshake timeout, but the actual connection will not be retried) Closes bazelbuild#18836. PiperOrigin-RevId: 546037698 Change-Id: I7f6efcb857c557aa97ad3df085fc032c8538eb9a
werkt
added a commit
to werkt/bazel
that referenced
this issue
Jul 17, 2023
Unwrapping all StatusRuntimeExceptions in in ReferenceCountedChannel when caused by IOException will discard critical tracing and retriability. The Retrier evaluations may not see an SRE in the causal chain, and presume it is invariably an unretriable exception. In general, IOExceptions as SRE wrappers are unsuitable containers and are routinely misued either for identification (grpc aware status), or capture (handleInitError). Partially addresses bazelbuild#18764 (retries will occur with SSL handshake timeout, but the actual connection will not be retried) Closes bazelbuild#18836. PiperOrigin-RevId: 546037698 Change-Id: I7f6efcb857c557aa97ad3df085fc032c8538eb9a
|
The associated #18836 fixes this 'enough' to get the retry in place. The remaining issue is one that looks like grpc-java is doing interesting things after it receives the first handshake timeout - the same exception is thrown repeatedly while no attempt to reconnect occurs, which means that the retries aren't given their fair shake after the first timeout. It will require more investigation, but it is no longer a bug in bazel. Closing. |
werkt
added a commit
to werkt/bazel
that referenced
this issue
Jul 24, 2023
Unwrapping all StatusRuntimeExceptions in in ReferenceCountedChannel when caused by IOException will discard critical tracing and retriability. The Retrier evaluations may not see an SRE in the causal chain, and presume it is invariably an unretriable exception. In general, IOExceptions as SRE wrappers are unsuitable containers and are routinely misued either for identification (grpc aware status), or capture (handleInitError). Partially addresses bazelbuild#18764 (retries will occur with SSL handshake timeout, but the actual connection will not be retried) Closes bazelbuild#18836. PiperOrigin-RevId: 546037698 Change-Id: I7f6efcb857c557aa97ad3df085fc032c8538eb9a
keertk
pushed a commit
that referenced
this issue
Jul 25, 2023
* Include stack trace in all gRPC errors when --verbose_failures is set. Also refactor a couple places where the stack trace was included in an ad-hoc manner, and force Utils.grpcAwareErrorMessage callers to be explicit to avoid future instances. Closes #16086. PiperOrigin-RevId: 502854490 Change-Id: Id2d6a1728630fffea9399b406378c7f173b247bd * Avoid discarding SRE state for IO cause Unwrapping all StatusRuntimeExceptions in in ReferenceCountedChannel when caused by IOException will discard critical tracing and retriability. The Retrier evaluations may not see an SRE in the causal chain, and presume it is invariably an unretriable exception. In general, IOExceptions as SRE wrappers are unsuitable containers and are routinely misued either for identification (grpc aware status), or capture (handleInitError). Partially addresses #18764 (retries will occur with SSL handshake timeout, but the actual connection will not be retried) Closes #18836. PiperOrigin-RevId: 546037698 Change-Id: I7f6efcb857c557aa97ad3df085fc032c8538eb9a * Operation stream termination is not an error According to the GrpcRemoteExecutor when it occurs after a !done operation response. Remove the error from the ExperimentalRemoteGrpcExecutor and reinforce both with tests. Update the FakeExecutionService to generate compatible error responses that appear in the ExecuteResponse, rather than the operation error field, per the REAPI spec. Made required adjustments to ExGRE Test invocations to avoid the ExecutionStatusException interpretation of DEADLINE_EXCEEDED -> FAILED_PRECONDITION in ExecuteResponse. Closes #18785. PiperOrigin-RevId: 546925894 Change-Id: I7a489c8bc936a83cfd94e0138437f3fe6d152da8 * Done operations must be reexecuted Any operation with done == true as reported by the server is not expected to change its result on subsequent waitExecution calls. To properly retry, this action must be reexecuted, if it was truly transient, to achieve a definitive result. Submit a transient status for retry, disallow special behaviors for NOT_FOUND as covered by done observation, and consider method type when handling operation streams. Closes #18943. PiperOrigin-RevId: 548680656 Change-Id: Ib2c9887ead1fbd3de97761db6e8b4077783ad03c --------- Co-authored-by: Tiago Quelhas <tjgq@google.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Description of the bug:
A build failure occurs when GrpcCacheClient cannot reach the remote endpoint in the Ssl handshake timeout. The local exception is wrapped in a grpc UNAVAILABLE StatusRuntimeException with a silent inner request error handler, bounded by downloadBlobs future aggregation.
Evidence for the lack of retry is circumstantial: Unavailability of the server for >10s * 5 would have been unlikely, and if it was caused by a severed connection on time wait, I would have expected a reconnection attempt after the first. I observed this stack trace with a silent accepting remote endpoint from the Capabilities check (which also failed to retry).
What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.
Very little easy here: a server would have to lose connection after startup and be replaced with a silent accepting one in the time period before a download began.
Minimum spec: grpcs (not grpc plaintext), remote_cache (to inspire downloadBlob), hitting the remote cache for an action and downloading results.
Which operating system are you running Bazel on?
No response
What is the output of
bazel info release?6.2.0
If
bazel info releasereturnsdevelopment versionor(@non-git), tell us how you built Bazel.No response
What's the output of
git remote get-url origin; git rev-parse master; git rev-parse HEAD?No response
Is this a regression? If yes, please try to identify the Bazel commit where the bug was introduced.
No response
Have you found anything relevant by searching the web?
No response
Any other information, logs, or outputs that you want to share?
No response
The text was updated successfully, but these errors were encountered: