@DisallowConcurrentExecution jobs are running at same time on different nodes when cluster de-synced

[pavelkokush]

Use case: Quartz cluster with node A and B. All jobs are marked as @DisallowConcurrentExecution. Node A detect that node B "failed", because node B did not checkin for some period of time. But node B is ok (just failed to check-in for some reason) and there is one running job on node B.

Actual: node A will change status from STATE_BLOCKED to STATE_WAITING for all jobs that was started on node B. So jobs will start running on node A even if same jobs are still running on node B. It is NOT EXPECTED, because jobs are marked as @DisallowConcurrentExecution!!!

Expected: Probably add some config like "org.quartz.jobStore.pauseOnNodeFail: true". Then pause jobs those have status STATE_BLOCKED. So user that is monitoring cluster will verify cluster state manually and decide is ok to un-pause job, then un-pause it manually.

[pavelkokush]

It is in JobStoreSupport.clusterRecover()

// free up stateful job's triggers
if (ftRec.isJobDisallowsConcurrentExecution()) {
    getDelegate()
            .updateTriggerStatesForJobFromOtherState(
                    conn, jKey,
                    STATE_WAITING, STATE_BLOCKED);
    getDelegate()
            .updateTriggerStatesForJobFromOtherState(
                    conn, jKey,
                    STATE_PAUSED, STATE_PAUSED_BLOCKED);
}

[zemian]

Hi, sorry for the long delay in respond to this issue. Your issue seems to be duplication of #107 ! Please try the solution provided over there. Check carefully about your cluster setting in scheduler config and your DB isolation level. If your problem is exists, try the acquireTriggersWithinLock=true config as discussed over there.