Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[FLINK-11843] Bind lifespan of Dispatcher to leader session #9832

Closed
wants to merge 47 commits into from
Closed

[FLINK-11843] Bind lifespan of Dispatcher to leader session #9832

wants to merge 47 commits into from

Commits on Oct 25, 2019

  1. [hotfix] Enrich DispatcherRunnerFactory to pass in additional services

    @tillrohrmann
    tillrohrmann committed Oct 25, 2019
    Configuration menu
    Copy the full SHA
    7621d63 View commit details
    Browse the repository at this point in the history

  2. [hotfix] Introduce JobManagerMetricGroupFactory and use for Dispatche… 

    …rServices creation

The JobManagerMetricGroupFactory allows to create a fresh JobManagerMetricGroup whenever
we create a new Dispatcher instance.
    @tillrohrmann
    tillrohrmann committed Oct 25, 2019
    Configuration menu
    Copy the full SHA
    314e9e1 View commit details
    Browse the repository at this point in the history

  3. [hotfix] Stop DispatcherResourceManagerComponent if Dispatcher comple… 

    …tes shutDownFuture in MiniCluster
    @tillrohrmann
    tillrohrmann committed Oct 25, 2019
    Configuration menu
    Copy the full SHA
    76124a2 View commit details
    Browse the repository at this point in the history

  4. [hotfix] Rework ZooKeeperLeaderElectionITCase to not rely on Dispatch… 

    …erResourceManagerComponent internals
    @tillrohrmann
    tillrohrmann committed Oct 25, 2019
    Configuration menu
    Copy the full SHA
    0617180 View commit details
    Browse the repository at this point in the history

  5. [hotfix] Remove unused methods from MiniCluster

    @tillrohrmann
    tillrohrmann committed Oct 25, 2019
    Configuration menu
    Copy the full SHA
    69eca66 View commit details
    Browse the repository at this point in the history

  6. [hotfix] Improve logging in DispatcherResourceManagerComponent

    @tillrohrmann
    tillrohrmann committed Oct 25, 2019
    Configuration menu
    Copy the full SHA
    18e0abf View commit details
    Browse the repository at this point in the history

  7. [FLINK-11665] Add DispatcherRunnerImplTest for FLINK-11665

    @tillrohrmann
    tillrohrmann committed Oct 25, 2019
    Configuration menu
    Copy the full SHA
    dd8264f View commit details
    Browse the repository at this point in the history

  8. [FLINK-11843] Add test case for FLINK-11843 

    DispatcherRunnerImplTest#testJobRecoveryUnderLeaderChange fails only
occasionally.
    @tillrohrmann
    tillrohrmann committed Oct 25, 2019
    Configuration menu
    Copy the full SHA
    5618e9b View commit details
    Browse the repository at this point in the history

  9. [FLINK-11843] Add DispatcherRunnerImplNG with DispatcherLeaderProcess… 

    … abstraction
    @tillrohrmann
    tillrohrmann committed Oct 25, 2019
    Configuration menu
    Copy the full SHA
    acd3a57 View commit details
    Browse the repository at this point in the history

  10. [FLINK-11843] Add DispatcherLeaderProcessImpl

    @tillrohrmann
    tillrohrmann committed Oct 25, 2019
    Configuration menu
    Copy the full SHA
    297de32 View commit details
    Browse the repository at this point in the history

  11. [FLINK-11843] Add DispatcherRunnerImplNGFactory implementation

    @tillrohrmann
    tillrohrmann committed Oct 25, 2019
    Configuration menu
    Copy the full SHA
    d65968b View commit details
    Browse the repository at this point in the history

  12. [FLINK-11843] Allow to pass JobGraphStore into Dispatcher via Dispatc… 

    …herServices

This commit introduces DispatcherFactoryServices which is an extension of
PartialDispatcherFactoryServices. The extension allows to set a specific
JobGraphStore which is being forwarded via the DispatcherServices to the
Dispatcher.
    @tillrohrmann
    tillrohrmann committed Oct 25, 2019
    Configuration menu
    Copy the full SHA
    b26d966 View commit details
    Browse the repository at this point in the history

  13. [FLINK-11843] Remove JobGraphListener from Dispatcher

    @tillrohrmann
    tillrohrmann committed Oct 25, 2019
    Configuration menu
    Copy the full SHA
    6ea3ebe View commit details
    Browse the repository at this point in the history

  14. [FLINK-11843] Allow passing collection of recovered jobs to Dispatcher

    @tillrohrmann
    tillrohrmann committed Oct 25, 2019
    Configuration menu
    Copy the full SHA
    4ada43b View commit details
    Browse the repository at this point in the history

  15. [FLINK-11843] Remove job recovery from Dispatcher

    @tillrohrmann
    tillrohrmann committed Oct 25, 2019
    Configuration menu
    Copy the full SHA
    32df798 View commit details
    Browse the repository at this point in the history

  16. [FLINK-11843] Restrict Dispatcher to only use JobGraphWriter instead … 

    …of JobGraphStore
    @tillrohrmann
    tillrohrmann committed Oct 25, 2019
    Configuration menu
    Copy the full SHA
    e6a2a5b View commit details
    Browse the repository at this point in the history

  17. [FLINK-11843] Make Dispatcher a PermanentlyFencedRpcEndpoint

    @tillrohrmann
    tillrohrmann committed Oct 25, 2019
    Configuration menu
    Copy the full SHA
    16191bd View commit details
    Browse the repository at this point in the history

  18. [FLINK-11843] Enable DispatcherRunnerImplNG

    @tillrohrmann
    tillrohrmann committed Oct 25, 2019
    Configuration menu
    Copy the full SHA
    2bc6bab View commit details
    Browse the repository at this point in the history

  19. [FLINK-11665] Port ZooKeeperDispatcherRunnerImplTest to use Dispatche… 

    …rRunnerImplNG
    @tillrohrmann
    tillrohrmann committed Oct 25, 2019
    Configuration menu
    Copy the full SHA
    da464ad View commit details
    Browse the repository at this point in the history

  20. [FLINK-11843] Enable DispatcherRunnerImplTest for DispatcherRunnerImplNG

    @tillrohrmann
    tillrohrmann committed Oct 25, 2019
    Configuration menu
    Copy the full SHA
    405f16b View commit details
    Browse the repository at this point in the history

  21. [FLINK-11843] Port MiniDispatcherTest to not do leader election for t… 

    …he MiniDispatcher
    @tillrohrmann
    tillrohrmann committed Oct 25, 2019
    Configuration menu
    Copy the full SHA
    4642712 View commit details
    Browse the repository at this point in the history

  22. [FLINK-11843] Create AbstractDispatcherLeaderProcess

    @tillrohrmann
    tillrohrmann committed Oct 25, 2019
    Configuration menu
    Copy the full SHA
    d9f527b View commit details
    Browse the repository at this point in the history

  23. [FLINK-11843] Add proper per-job mode support

    @tillrohrmann
    tillrohrmann committed Oct 25, 2019
    Configuration menu
    Copy the full SHA
    1bbeeed View commit details
    Browse the repository at this point in the history

  24. [hotfix] Remove explicit JobGraphWriter from Dispatcher constructor

    @tillrohrmann
    tillrohrmann committed Oct 25, 2019
    Configuration menu
    Copy the full SHA
    e94773c View commit details
    Browse the repository at this point in the history

  25. [FLINK-11843] Port and remove DispatcherTest 

    [FLINK-11843] Remove DispatcherTest#testLeaderElection

DispatcherTest#testLeaderElection is superseded by
DispatcherRunnerImplNGTest#grantLeadership_validLeader_confirmsLeaderSession.

[FLINK-11843] Port DispatcherTest#testJobSuspensionWhenDispatcherLosesLeadership

Changed DispatcherTest#testJobSuspensionWhenDispatcherLosesLeadership into
DispatcherRunnerImplNGTest#revokeLeadership_withExistingLeader_stopsLeaderProcess to
cover test logic of #testJobSuspensionWhenDispatcherLosesLeadership.

[FLINK-11843] Port DispatcherTest#testFatalErrorAfterJobRecoveryFailure

Port DipsatcherTest#testFatalErrorAfterJobRecoveryFailure to
DispatcherLeaderProcessImplTest#recoverJobs_withRecoveryFailure_failsFatally

[FLINK-11843] Port DispatcherTest#testFatalErrorAfterJobIdRecoveryFailure

Port DispatcherTest#testFatalErrorAfterJobIdRecoveryFailure to
DispatcherLeaderProcessImplTest#recoverJobs_withRecoveryFailure_failsFatally.

[FLINK-11843] Port DispatcherTest#testJobSubmissionErrorAfterJobRecovery

Port DispatcherTest#testJobSubmissionErrorAfterJobRecovery to

[FLINK-11843] Port DispatcherTest#testJobRecovery

Replace DispatcherTest#testJobRecovery with
DispatcherRunnerImplNGITCase#leaderChange_afterJobSubmission_recoversSubmittedJob.

[FLINK-11843] Clean up DispatcherTest
    @tillrohrmann
    tillrohrmann committed Oct 25, 2019
    Configuration menu
    Copy the full SHA
    3b78568 View commit details
    Browse the repository at this point in the history

  26. [FLINK-11843] Port DispatcherResourceCleanupTest 

    Ports the DispatcherResourceCleanupTest to work with the new Dispatcher contract
    @tillrohrmann
    tillrohrmann committed Oct 25, 2019
    Configuration menu
    Copy the full SHA
    6045dc1 View commit details
    Browse the repository at this point in the history

  27. [hotfix] Pass in DispatcherServices into TestingDispatcher

    @tillrohrmann
    tillrohrmann committed Oct 25, 2019
    Configuration menu
    Copy the full SHA
    457920a View commit details
    Browse the repository at this point in the history

  28. [FLINK-11843] Pass in explicit JobGraphWriter in DispatcherTests

    @tillrohrmann
    tillrohrmann committed Oct 25, 2019
    Configuration menu
    Copy the full SHA
    4b4008a View commit details
    Browse the repository at this point in the history

  29. [FLINK-11843] Port and remove ZooKeeperHADispatcherTest#testStandbyDi… 

    …spatcherJobExecution

ZooKeeperHADispatcherTest#testStandbyDispatcherJobExecution has been replaced with
DispatcherLeaderProcessImplTest#onAddedJobGraph_ifNotRunning_isBeingIgnored

[FLINK-11843] Remove ZooKeeperHADispatcherTest#testJobGraphRelease

ZooKeeperHADispatcherTest#testJobGraphRelease is already covered by
DispatcherLeaderProcessImplTest#closeAsync_stopsJobGraphStoreAndDispatcher.

[FLINK-11843] Delete ZooKeeperHADispatcherTest#testStandbyDispatcherJobRecovery

Delete ZooKeeperHADispatcherTest#testStandbyDispatcherJobRecovery because it should no longer
add test coverage. The test is superseded by
DispatcherLeaderProcessImplTest#start_triggersJobGraphRecoveryAndDispatcherServiceCreation,
DispatcherRunnerImplNGTest#grantLeadership_validLeader_confirmsLeaderSession and
DispatcherTest#testPersistedJobGraphWhenDispatcherIsShutDown
    @tillrohrmann
    tillrohrmann committed Oct 25, 2019
    Configuration menu
    Copy the full SHA
    bca0a05 View commit details
    Browse the repository at this point in the history

  30. [FLINK-11843] Port and remove DispatcherHATest 

    Remove DispatcherTest#testFailingRecoveryIsFatalError

DispatcherHATest#testFailingRecoveryIsFatalError has been superseded by
DispatcherLeaderProcessImplTest#recoverJobs_withRecoveryFailure_failsFatally.

[FLINK-11843] Port DispatcherHATest#testRevokeLeadershipTerminatesJobManagerRunners

DispatcherHATest#testRevokeLeadershipTerminatesJobManagerRunners has been replaced by
DispatcherResourceCleanupTest#testDispatcherTerminationTerminatesRunningJobMasters.

[FLINK-11843] Remove DispatcherHATest#testJobRecoveryWhenChangingLeadership

DispatcherHATest#testJobRecoveryWhenChangingLeadership has been replaced with
DispatcherRunnerImplNGITCase#leaderChange_afterJobSubmission_recoversSubmittedJob

[FLINK-11843] Remove DispatcherHATest#testGrantingRevokingLeadership

DispatcherHATest#testGrantingRevokingLeadership has been replaced with
DispatcherLeaderProcessImplTest#closeAsync_duringJobRecovery_preventsDispatcherServiceCreation.

[FLINK-11843] Delete DispatcherHATest
    @tillrohrmann
    tillrohrmann committed Oct 25, 2019
    Configuration menu
    Copy the full SHA
    a6037d8 View commit details
    Browse the repository at this point in the history

  31. [FLINK-11843] Ignore duplicate job submission due to false positive o… 

    …nAddedJobGraph callbacks
    @tillrohrmann
    tillrohrmann committed Oct 25, 2019
    Configuration menu
    Copy the full SHA
    ef12c5c View commit details
    Browse the repository at this point in the history

  32. [FLINK-11843] Properly remove job graphs which have been removed from… 

    … the JobGraphStore

Instead of cancelling job graphs, which have been removed from the JobGraphStore, this commit
simply stops and removes jobs. This has the advantage that the job does not go into a globally
terminal state.
    @tillrohrmann
    tillrohrmann committed Oct 25, 2019
    Configuration menu
    Copy the full SHA
    71b8a66 View commit details
    Browse the repository at this point in the history

  33. [FLINK-11843] Forward shut down future from Dispatcher through to the… 

    … DispatcherRunnerImplNG
    @tillrohrmann
    tillrohrmann committed Oct 25, 2019
    Configuration menu
    Copy the full SHA
    fa320bf View commit details
    Browse the repository at this point in the history

  34. [FLINK-11843] Move DispatcherService into AbstractDispatcherLeaderPro… 

    …cess
    @tillrohrmann
    tillrohrmann committed Oct 25, 2019
    Configuration menu
    Copy the full SHA
    17151ab View commit details
    Browse the repository at this point in the history

  35. [FLINK-11843] Remove old DispatcherRunnerImpl and DispatcherRunnerFac… 

    …toryImpl
    @tillrohrmann
    tillrohrmann committed Oct 25, 2019
    Configuration menu
    Copy the full SHA
    39d8fd3 View commit details
    Browse the repository at this point in the history

  36. [FLINK-11843] Rename various newly introduced classes 

    Rename XImplNG implementations into DefaultX. Rename DispatcherLeaderProcessImpl into
SessionDispatcherLeaderProcess.
    @tillrohrmann
    tillrohrmann committed Oct 25, 2019
    Configuration menu
    Copy the full SHA
    e9e1b58 View commit details
    Browse the repository at this point in the history

  37. [FLINK-11843] Move DispatcherRunnerImplTest#testJobRecoveryUnderLeade… 

    …rChange to DefaultDispatcherRunnerITCase

The DispatcherRunnerImplTest#testJobRecoveryUnderLeaderChange has been moved to the
DefaultDispatcherRunnerITCase#leaderChange_withBlockingJobManagerTermination_doesNotAffectNewLeader.
    @tillrohrmann
    tillrohrmann committed Oct 25, 2019
    Configuration menu
    Copy the full SHA
    f688b80 View commit details
    Browse the repository at this point in the history

  38. [FLINK-11843] Various cleanups for DefaultDispatcherRunner and tests

    @tillrohrmann
    tillrohrmann committed Oct 25, 2019
    Configuration menu
    Copy the full SHA
    2d9c657 View commit details
    Browse the repository at this point in the history

  39. [FLINK-11843] Introduce DispatcherRunnerLeaderElectionLifecycleManager 

    The DispatcherRunnerLeaderElectionLifecycleManager is responsible for stopping the
LeaderElectionService.
    @tillrohrmann
    tillrohrmann committed Oct 25, 2019
    Configuration menu
    Copy the full SHA
    ef99208 View commit details
    Browse the repository at this point in the history

  40. [FLINK-11665] Various ZooKeeperDefaultDispatcherRunnerTest clean ups

    @tillrohrmann
    tillrohrmann committed Oct 25, 2019
    Configuration menu
    Copy the full SHA
    f481c27 View commit details
    Browse the repository at this point in the history

  41. [FLINK-11843] Remove unused dispatcher runner accessor from Dispatche… 

    …rResourceManagerComponent
    @tillrohrmann
    tillrohrmann committed Oct 25, 2019
    Configuration menu
    Copy the full SHA
    aa1d355 View commit details
    Browse the repository at this point in the history

  42. [FLINK-11843] Remove unnecessary getDispatcherGateway method from Dis… 

    …patcherRunner
    @tillrohrmann
    tillrohrmann committed Oct 25, 2019
    Configuration menu
    Copy the full SHA
    d90eb6d View commit details
    Browse the repository at this point in the history

  43. [FLINK-11843] Rename DispatcherLeaderProcess#getConfirmLeaderSessionF… 

    …uture into getLeaderAddressFuture
    @tillrohrmann
    tillrohrmann committed Oct 25, 2019
    Configuration menu
    Copy the full SHA
    1cd59ff View commit details
    Browse the repository at this point in the history

  44. [FLINK-11843] Various SessionDispatcherLeaderProcessTest clean ups

    @tillrohrmann
    tillrohrmann committed Oct 25, 2019
    Configuration menu
    Copy the full SHA
    7abafbc View commit details
    Browse the repository at this point in the history

  45. [hotfix] Remove Nonnull annotations from DispatcherFactory and sub cl… 

    …asses
    @tillrohrmann
    tillrohrmann committed Oct 25, 2019
    Configuration menu
    Copy the full SHA
    4f102c2 View commit details
    Browse the repository at this point in the history

  46. [FLINK-11843] Rename DispatcherService into DispatcherGatewayService

    @tillrohrmann
    tillrohrmann committed Oct 25, 2019
    Configuration menu
    Copy the full SHA
    2987666 View commit details
    Browse the repository at this point in the history

  47. [hotfix] Remove exception suppression from Dispatcher#stopDispatcherS… 

    …ervices
    @tillrohrmann
    tillrohrmann committed Oct 25, 2019
    Configuration menu
    Copy the full SHA
    7d041b2 View commit details
    Browse the repository at this point in the history