KAFKA-17553 Fix shutdown race condition in StreamThreadTest #17191
+15
−9
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This reverts commit b436499.
mumrah
added
the
do-not-merge
|
Thread dump here https://github.com/apache/kafka/actions/runs/10852637802/artifacts/1932150615 |
|
I managed to reproduce the issue in this run https://github.com/apache/kafka/actions/runs/10893557629/job/30231419064?pr=17191 Notice the 10 second delay in this test. This is coming from the changes I have in this PR: // in DefaultTaskManager
public void shutdown(final Duration duration) {
for (final TaskExecutor t: taskExecutors) {
t.requestShutdown();
}
signalTaskExecutors();
try {
for (final TaskExecutor t: taskExecutors) {
t.awaitShutdown(Duration.ofSeconds(10));
}
} catch (final Exception e) {
signalTaskExecutors();
for (final TaskExecutor t: taskExecutors) {
t.awaitShutdown(duration);
}
}
}This is kind of a hacky solution, but it does avoid the deadlock. |
chia7712
reviewed
Sep 17, 2024
There was a problem hiding this comment.
@mumrah thanks for this quick fix. the approach is good but it may not honor the timeout. Maybe we can add a Supplier<Boolean> to awaitProcessableTasks to return without await? the check is holding the lock, so it can fix the race condition. for example:
awaitProcessableTasks(Supplier<Boolean> needToAwait)taskManager.awaitProcessableTasks(() -> !shutdownRequested.get());
...ams/src/main/java/org/apache/kafka/streams/processor/internals/tasks/DefaultTaskManager.java
Outdated
Show resolved
Hide resolved
...ams/src/main/java/org/apache/kafka/streams/processor/internals/tasks/DefaultTaskManager.java
Outdated
Show resolved
Hide resolved
chia7712
changed the title
Closed
3 tasks
mumrah
changed the title
|
One of the Gradle workers hit an OOM Not sure why, but looks like Mockito was generating a large string as part of a test failure. |
|
@chia7712 I've updated the code with the supplier approach. I was wondering about something like this, but hesitant to change too much code I'm not familiar with 😅. Turns out not to be too bad. |
mumrah
requested review from
guozhangwang,
ableegoldman and
vvcephei
and removed request for
lucasbru and
vvcephei
mumrah
added
streams
and removed
do-not-merge
...src/test/java/org/apache/kafka/streams/processor/internals/tasks/DefaultTaskManagerTest.java
Outdated
Show resolved
Hide resolved
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Reverting #17180 in order to generate a thread dump of the stalled test.