KAFKA-12495: Exponential backoff retry to prevent rebalance storms when worker joins after revoking rebalance#12561

Merged

C0urante merged 27 commits intoapache:trunkfrom

vamossagar12:KAFKA-12495

Nov 15, 2022

Commits on Nov 15, 2022

KAFKA-12495: Initial commit to get a balanced assignment and allow consecutive rebalance

Sagar Rao
authored and
Sagar Rao
committed
Some local changes

Sagar Rao
authored and
Sagar Rao
committed
KAFKA-12495: Adding exponential backoff delay when worker joins after a revoking rebalance before a revoking rebalance can be triggered

Sagar Rao
authored and
Sagar Rao
committed
Adding revokedInPrevious state to exponential backoff delay only when there are successive revoking rebalabces

Sagar Rao
authored and
Sagar Rao
committed
Removing time.sleep during successive rebalances and instead using the existing scheduled rebalanceDelay mechanism to introduce exponential backoffs during successive revoking rebalances

Sagar Rao
authored and
Sagar Rao
committed
Update connect/runtime/src/main/java/org/apache/kafka/connect/runtime/distributed/IncrementalCooperativeAssignor.java

authored and
Sagar Rao
committed
Review changes

Sagar Rao
authored and
Sagar Rao
committed
Setting initial interval to 1 and lowering multiplier. Also always setting delay to 0 to maxDelay is 0.

Sagar Rao
authored and
Sagar Rao
committed
Removing needless check while computing exponential backoff delay

Sagar Rao
authored and
Sagar Rao
committed
Reenabling flaky test testMultipleWorkersRejoining and addressing review comments

Sagar Rao
authored and
Sagar Rao
committed
Removing unwanted rebalance check in testSuccessiveRevocationsWhenMaxDelayIsEqualToExpBackOffInitialInterval

Sagar Rao
authored and
Sagar Rao
committed
Changing arguements of performTaskRevocation back to the old ones

Sagar Rao
authored and
Sagar Rao
committed
Fixing failing testcase

Sagar Rao
authored and
Sagar Rao
committed
Removing unwanted file

Sagar Rao
authored and
Sagar Rao
committed
Increasing exp backoff multiplier to 40

Sagar Rao
authored and
Sagar Rao
committed
Removing resetting numSuccessiveRevokingRebalances to 0

Sagar Rao
authored and
Sagar Rao
committed
Resetting revokedInPrevious to false when there's no rebalance + few tests moved around/fixed

Sagar Rao
authored and
Sagar Rao
committed
Removing unwanted sysout

Sagar Rao
authored and
Sagar Rao
committed
Reverting test

Sagar Rao
authored and
Sagar Rao
committed
Reenabling RebalanceSourceConnectorsIntegrationTest#testDeleteConnector test

Sagar Rao
authored and
Sagar Rao
committed
KAFKA-12495: Improve rebalance allocation algorithm
C0urante
committed
Address review comments, remove unused code
C0urante
committed
Address review comments
C0urante
committed
Merge pull request #1 from C0urante/kafka-12495-followup
vamossagar12
authored
Fixing checkstyle

Sagar Rao
authored and
Sagar Rao
committed
Revert "Fixing checkstyle"

Sagar Rao
authored and
Sagar Rao
committed
fixing checkstyle again

Sagar Rao
authored and
Sagar Rao
committed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

KAFKA-12495: Exponential backoff retry to prevent rebalance storms when worker joins after revoking rebalance#12561

KAFKA-12495: Exponential backoff retry to prevent rebalance storms when worker joins after revoking rebalance#12561
C0urante merged 27 commits intoapache:trunkfrom
vamossagar12:KAFKA-12495

Commits on Nov 15, 2022

KAFKA-12495: Initial commit to get a balanced assignment and allow consecutive rebalance

Some local changes

KAFKA-12495: Adding exponential backoff delay when worker joins after a revoking rebalance before a revoking rebalance can be triggered

Adding revokedInPrevious state to exponential backoff delay only when there are successive revoking rebalabces

Removing time.sleep during successive rebalances and instead using the existing scheduled rebalanceDelay mechanism to introduce exponential backoffs during successive revoking rebalances

Update connect/runtime/src/main/java/org/apache/kafka/connect/runtime/distributed/IncrementalCooperativeAssignor.java

Review changes

Setting initial interval to 1 and lowering multiplier. Also always setting delay to 0 to maxDelay is 0.

Removing needless check while computing exponential backoff delay

Reenabling flaky test testMultipleWorkersRejoining and addressing review comments

Removing unwanted rebalance check in testSuccessiveRevocationsWhenMaxDelayIsEqualToExpBackOffInitialInterval

Changing arguements of performTaskRevocation back to the old ones

Fixing failing testcase

Removing unwanted file

Increasing exp backoff multiplier to 40

Removing resetting numSuccessiveRevokingRebalances to 0

Resetting revokedInPrevious to false when there's no rebalance + few tests moved around/fixed

Removing unwanted sysout

Reverting test

Reenabling RebalanceSourceConnectorsIntegrationTest#testDeleteConnector test

KAFKA-12495: Improve rebalance allocation algorithm

Address review comments, remove unused code

Address review comments

Merge pull request #1 from C0urante/kafka-12495-followup

Fixing checkstyle

Revert "Fixing checkstyle"

fixing checkstyle again