[v23.1.x] rpc/transport: fix temporary dispatch loop stalls #11184
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Incase RPCs are timing out due to memory pressure (unlikely given we do not set a small limit to the semaphore but just in case). (cherry picked from commit d014caf)
.. to track the sequence of dispatches. (cherry picked from commit 3cbb63b)
Currently the way the code is structured can result in temporary dispatch stalls timing out a bunch of requests. Consider a request queue of sequence numbers [5, 6, 7, 8] Lets assume seq=6 dispatch got delayed. This can happen if timeout is low or there is an unknown delay by the scheduler (eg: debug builds). _last_seq=4, queue = [5, 7, 8] - 5 is dispatched right away _last_seq=5, queue = [7,8] -- out of order. Dispatch of 7, 8 doesn't happen right away due to out of order sequence and we never do a dispatch with seq=6 because we detect it already timed out. Now it takes a new RPC request to clear the stalled queue but they may not happen for a while as most of these are timer based and meanwhile seq=7/8 timeout for no fault of theirs. This patch does two main things. * Consolidates _last_seq tracking. Currently it is not monotonic and can jump all over the place. This is confusing. Now we only update it in a centralized place in dispatch_send(). * ^^ Requires that dispatch_send() is called in all cases which also avoids dispatch loop stalls. For example, a timed out request will clear the stalled queue right away (which is not the case before). (cherry picked from commit 453d340)
(cherry picked from commit 93e8274)
|
issue not observed so far in this branch and the fix is very fundamental, we can revisit this backport once the original fix has more bake time. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Backport of PR #11023