Removed race condition from global_queue_depth_multi_thread test #6936
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
jofas
commented
Oct 25, 2024
| @@ -93,7 +93,7 @@ fn try_block_threaded(rt: &Runtime) -> Result<Vec<mpsc::Sender<()>>, mpsc::RecvT | |||
|
|
|||
| // Spawn a task per runtime worker to block it. | |||
| rt.spawn(async move { | |||
| tx.send(()).unwrap(); | |||
| tx.send(()).ok(); | |||
There was a problem hiding this comment.
I made this fail silently because I triggered a panic here when I ^Ced a test run, which I found needlessly confusing.
mox692
approved these changes
Oct 27, 2024
jofas
force-pushed
the
fix-flaky-injection-queue-test-revisited
branch
from
8235535 to
78f7009
Compare
|
Thanks for the heads up, didn't see that the CI failure was fixed. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
@Darksonn and @mox692 discovered a race condition in the retry logic introduced in #6916 that this PR aims to fix.
In the
global_queue_depth_multi_threadtest, we try to block the two worker threads before spawning more tasks onto the global queue. This allows us to measure theRuntimeMetrics::global_queue_depthdeterministically, as the two worker threads are blocked and therefore unable to pull any tasks from the global queue. Blocking the two worker threads might fail, in which case one of the tasks from the previous try to block the runtime would remain in the global queue. If the task remains in the global queue after the second try to block the runtime succeeds, it would add an unexpected task to theglobal_queue_depth, making the test fail.The solution this PR provides is to just create a new runtime on each try to block it. So if the first try to block the runtime fails, we drop the runtime and create a new one (with an empty global queue) that we then try to block again. We proceed until we succeed in blocking one runtime or run out of tries.