Refactor Stopping Tasks On Assignment Change of Tasks #868
Conversation
...connector/src/main/java/com/linkedin/datastream/connectors/kafka/AbstractKafkaConnector.java
Outdated
Show resolved
Hide resolved
...connector/src/main/java/com/linkedin/datastream/connectors/kafka/AbstractKafkaConnector.java
Outdated
Show resolved
Hide resolved
...connector/src/main/java/com/linkedin/datastream/connectors/kafka/AbstractKafkaConnector.java
Show resolved
Hide resolved
There was a problem hiding this comment.
So the main objective of the change is to not call stop one more time while a current stop attempt is in progress. Correct?
Is yes I think that is still possible since in "restartDeadTask" we still call "stopUnassignedTasks" which calls "scheduleTasksToStop" with "_tasksPendingStop"
...connector/src/main/java/com/linkedin/datastream/connectors/kafka/AbstractKafkaConnector.java
Show resolved
Hide resolved
@surajkn Actually, the main objective of this change is to separate the two triggers of stopping the tasks. We want to avoid stopping the long-pending tasks whenever onAssignmentChange is invoked. We only want to trigger stop for the running tasks which are not part of the new assignment when the onAssignmentChange is called. Those long-pending tasks could still be re-triggered for stopping periodically from the restartDeadTask function, which should be the expected behavior I think. |
shrinandthakkar
force-pushed
the
refactorStoppingTasksOnAssignmentChange
branch
from
8ffb739 to
e2938f0
Compare
Only trigger stopping for the tasks that need to be canceled per assignment change in the
onAssignmentChangefunction. This change prevents calling stops on tasks recurrently in case of frequent assignment change triggers.Important: DO NOT REPORT SECURITY ISSUES DIRECTLY ON GITHUB.
For reporting security issues and contributing security fixes,
please, email security@linkedin.com instead, as described in
the contribution guidelines.
Please, take a minute to review the contribution guidelines at:
https://github.com/linkedin/Brooklin/blob/master/CONTRIBUTING.md