KAFKA-17109: Move lock backoff retry to streams TaskManager #17209

aliehsaeedii · 2024-09-16T12:44:17Z

This PR aims at resolving the issue made by #17116

mumrah · 2024-09-16T13:45:59Z

@aliehsaeedii please update the PR title to have a description of the patch. Thanks!

cadonna

Thanks for the PR, @aliehsaeedii !

Here my feedback.

I am missing unit tests.

cadonna · 2024-09-16T14:26:33Z

streams/src/main/java/org/apache/kafka/streams/processor/internals/TaskManager.java

+ }
+
+ public boolean canAttempt(final long nowMs) {
+ return nowMs - lastAttemptMs >= EXPONENTIAL_BACKOFF.backoff(attempts);


nit:

Suggested change

return nowMs - lastAttemptMs >= EXPONENTIAL_BACKOFF.backoff(attempts);

return nowMs - lastAttemptMs >= EXPONENTIAL_BACKOFF.backoff(attempts);

cadonna · 2024-09-16T14:31:22Z

streams/src/main/java/org/apache/kafka/streams/processor/internals/TaskManager.java

+ stateUpdater.add(task);
+ taskIdToBackoffRecord.remove(task.id());


Minor:
I would swap those two lines. Once the task is initialized, the backoff can be removed.

cadonna · 2024-09-16T15:04:25Z

streams/src/main/java/org/apache/kafka/streams/processor/internals/TaskManager.java

+ taskIdToBackoffRecord.remove(task.id());
+ } else {
+ log.trace("Task {} is still not allowed to retry acquiring the state directory lock", task.id());
+ handleUnsuccessfulLockAcquiring(task, nowMs);


Is this correct?
Every time initialization is attempted before the back-off, the time of the last attempt is updated to the current time. If we assume an attempt every poll interval and the poll interval is less than the back-off time, the task will never be initialized.
Assume the last unsuccessful attempt occurred at time 200 and now the current call to canTryLock() is 100ms later at time 300. Furthermore, assume the current back-off is 250. That is, canTryLock() should return false because 300 - 200 >= 250 is not true. The last attempt is updated to 300 and the backoff is exponentially updated with the increased number of attempt (let's say 500). If you try again in 100ms at 400 canTryLock() will again return false, because 400 - 300 >= 500 is still not true and it will also not be true next time. You should only update the back-off record if you actually have attempted to initialize the task and it was unsuccessful and not when you skipped the attempt due to the back-off.

cadonna · 2024-09-16T15:14:30Z

streams/src/main/java/org/apache/kafka/streams/processor/internals/TaskManager.java

+ public static class BackoffRecord {
+ private long attempts;
+ private long lastAttemptMs;
+ private static final ExponentialBackoff EXPONENTIAL_BACKOFF = new ExponentialBackoff(1, 2, 10000, 0.5);


Should the exponential back-off be specified in terms of poll time? Something like

new ExponentialBackoff(pollTime, 2, 10000, 0.5);

If it is to much trouble getting that config into the task manager, just choose something larger than 1ms. 1 ms sounds really small. The sequence of the back-offs would be 1ms, 2ms, 4ms, 8ms, 16ms, 32ms, 64, 128. At the same time, with default configs, the task initialization is attempted every 100ms. So, it seems there will not be much improvement to the current situation because the first 7 poll iterations you attempt to initialize the task.

aliehsaeedii · 2024-09-16T18:30:13Z

Thanks for the PR, @aliehsaeedii !

Here my feedback.

I am missing unit tests.

Thanks @cadonna. Utest is added + review is addressed

cadonna · 2024-09-16T19:00:30Z

streams/src/main/java/org/apache/kafka/streams/processor/internals/TaskManager.java

@@ -2116,4 +2132,37 @@ boolean needsInitializationOrRestoration() {
 void addTask(final Task task) {
 tasks.addTask(task);
 }
+
+ private boolean canTryLock(final TaskId taskId, final long nowMs) {


Sorry, I forgot to add this comment before in my review. Could you please rename this method to canTryInitializeTask()? I think that makes more sense.

@cadonna makes sense!

cadonna

Thanks for the updates!

Here my comments.

cadonna · 2024-09-18T22:44:29Z

streams/src/main/java/org/apache/kafka/streams/processor/internals/TaskManager.java

@@ -1006,14 +1014,22 @@ private void addTasksToStateUpdater() {
 }

 private void addTaskToStateUpdater(final Task task) {
+ final long nowMs = System.currentTimeMillis();


Here, you need to use

Suggested change

final long nowMs = System.currentTimeMillis();

final long nowMs = time.milliseconds();

We inject the time object at creation, so that we can control time for example in tests.

cadonna · 2024-09-18T22:46:16Z

streams/src/main/java/org/apache/kafka/streams/processor/internals/TaskManager.java

+ /* For testing */
+ void setTaskIdToBackoffRecord(final Map<TaskId, BackoffRecord> taskIdToBackoffRecord) {
+ this.taskIdToBackoffRecord = taskIdToBackoffRecord;
+ }


I do not think you need this method if you can control time as I describe on line 1017.

cadonna · 2024-09-18T22:46:53Z

streams/src/test/java/org/apache/kafka/streams/processor/internals/TaskManagerTest.java

+ .inState(State.RUNNING).build();
+ final TasksRegistry tasks = mock(TasksRegistry.class);
+ when(tasks.drainPendingTasksToInit()).thenReturn(mkSet(task00, task01));
+ final TaskManager.BackoffRecord backoffRecord = mock(TaskManager.BackoffRecord.class);


You do not need this mock. You can advance time with the time object.

aliehsaeedii added 2 commits September 16, 2024 13:31

kafka17109: lock backoff-retry

50aa532

remove runtime exception

cacfa1d

aliehsaeedii mentioned this pull request Sep 16, 2024

KAFKA-17109: implement exponential backoff for state directory lock #17116

Merged

aliehsaeedii changed the title ~~KAFKA-17109~~ KAFKA-17109: Move lock backoff retry to streams TaskManager Sep 16, 2024

cadonna reviewed Sep 16, 2024

View reviewed changes

add utest + address reviews

36af73b

cadonna reviewed Sep 16, 2024

View reviewed changes

aliehsaeedii added 2 commits September 17, 2024 13:58

rename method

6999dee

fix failing test

fa1c872

cadonna reviewed Sep 18, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

KAFKA-17109: Move lock backoff retry to streams TaskManager #17209

KAFKA-17109: Move lock backoff retry to streams TaskManager #17209

aliehsaeedii commented Sep 16, 2024

mumrah commented Sep 16, 2024

cadonna left a comment

cadonna Sep 16, 2024

cadonna Sep 16, 2024

cadonna Sep 16, 2024

cadonna Sep 16, 2024

aliehsaeedii commented Sep 16, 2024 •

edited

Loading

cadonna Sep 16, 2024

aliehsaeedii Sep 17, 2024

cadonna left a comment

cadonna Sep 18, 2024

cadonna Sep 18, 2024

cadonna Sep 18, 2024

	return nowMs - lastAttemptMs >= EXPONENTIAL_BACKOFF.backoff(attempts);
	return nowMs - lastAttemptMs >= EXPONENTIAL_BACKOFF.backoff(attempts);

		stateUpdater.add(task);
		taskIdToBackoffRecord.remove(task.id());

	final long nowMs = System.currentTimeMillis();
	final long nowMs = time.milliseconds();

KAFKA-17109: Move lock backoff retry to streams TaskManager #17209

Are you sure you want to change the base?

KAFKA-17109: Move lock backoff retry to streams TaskManager #17209

Conversation

aliehsaeedii commented Sep 16, 2024

mumrah commented Sep 16, 2024

cadonna left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aliehsaeedii commented Sep 16, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cadonna left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aliehsaeedii commented Sep 16, 2024 •

edited

Loading