Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

HDDS-8683. Container balancer thread interrupt may not work #6179

Merged
merged 4 commits into from
Feb 28, 2024
Merged

HDDS-8683. Container balancer thread interrupt may not work #6179

Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
800b237
HDDS-8683. Container balancer thread interrupt may not work
Feb 6, 2024
3bcfb20
Remove duplicate line
Feb 6, 2024
60fa8dd
Check isBalancerRunning before sleep for delayStart
Feb 9, 2024
025a9c5
Revert "Check isBalancerRunning before sleep for delayStart"
Feb 26, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 5 additions & 2 deletions ...er-scm/src/main/java/org/apache/hadoop/hdds/scm/container/balancer/ContainerBalancer.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -342,10 +342,13 @@ private static void blockTillTaskStop(Thread balancingThread) {
// NOTE: join should be called outside the lock in hierarchy
// to avoid locking others waiting
// wait for balancingThread to die with interrupt
balancingThread.interrupt();
LOG.info("Container Balancer waiting for {} to stop", balancingThread);
try {
balancingThread.join();
while (balancingThread.isAlive()) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When you interrupt the thread, as soon as it hits the Thread.sleep call, it should throw an interrupted exception and exit, so this retry logic should not be needed. I suspect there is some other bug in the interrupt handling that is causing this problem and we should try to get to the bottom of that.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@sodonnel
yes, thread will get interrupted as as soon it goes for sleep. Tested.
Here, issue is,

  • before thread actually schedule to run, and if there is interrupt, that gets ignored.

@Tejaskriya we can revert loop, but can add a check in run() before going for sleep for the delay, isBalancerRunning(). That will be case where it can ignore interrupt and go for sleep for 5 min as startup delay.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for the reviews. I have added a check in run() to check if balancer is running. Could you please review it again?

// retry interrupt every 5ms to avoid waiting when thread is sleeping
balancingThread.interrupt();
balancingThread.join(5);
}
} catch (InterruptedException exception) {
Thread.currentThread().interrupt();
}
Expand Down