Make the broker round robin selection in task execution configurable #2223
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
efeg
reviewed
Nov 25, 2024
...se-control/src/main/java/com/linkedin/kafka/cruisecontrol/executor/ExecutionTaskManager.java
Outdated
Show resolved
Hide resolved
CCisGG
requested changes
Nov 26, 2024
There was a problem hiding this comment.
Thanks for the change @allenxwang ! Let's get this tested before merging it.
...se-control/src/main/java/com/linkedin/kafka/cruisecontrol/executor/ExecutionTaskPlanner.java
Show resolved
Hide resolved
...se-control/src/main/java/com/linkedin/kafka/cruisecontrol/executor/ExecutionTaskPlanner.java
Show resolved
Hide resolved
| @@ -830,6 +831,22 @@ public synchronized void executeProposals(Collection<ExecutionProposal> proposal | |||
| requestedIntraBrokerPartitionMovementConcurrency, requestedClusterLeadershipMovementConcurrency, | |||
| requestedBrokerLeadershipMovementConcurrency, requestedExecutionProgressCheckIntervalMs, replicaMovementStrategy, | |||
| isTriggeredByUserRequest, loadMonitor); | |||
| if (removedBrokers != null && !removedBrokers.isEmpty()) { | |||
| int count = 0; | |||
There was a problem hiding this comment.
nit: maybe extract this snippet as a separate function.
| } | ||
| } | ||
| } | ||
| LOG.info("User task {}: {} of partition move proposals are related to removed brokers.", uuid, ((float) count) / totalCount); |
There was a problem hiding this comment.
You may want to actually log count and total count here, to observe whether this only happens for large scale rebalance or not.
cruise-control/src/main/java/com/linkedin/kafka/cruisecontrol/executor/Executor.java
Show resolved
Hide resolved
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
We found that the rebalance operation significantly slows down towards the latter half of the execution. It happens when most of the tasks in the proposal involve a single broker. The current behavior is that the ExecutionTaskPlanner would sort the brokers according to the ReplicaMoveStrategy, continuously pick a task from the first broker until it reaches the broker's max concurrency. However, it would skip the broker in the same round if the broker is already involved in a picked task. The effect is that almost all tasks towards the end of the execution are related to the broker that has the most tasks, thus the execution is bottlenecked by the single broker's max concurrency.
We will make this round robin behavior configurable and true by default. When it is set to false, the ExecutionTaskPlanner can keep picking the tasks from same broker that has the highest priority when it is asked to find the next set of tasks to execute. The ExecutionTaskPlanner would compare the broker to pick in the following order:
Expected Behavior
…
Actual Behavior
…
Steps to Reproduce
Known Workarounds
Additional evidence
Categorization
This PR resolves # if any.