When concurrency is forbidding and dkron exiting before it's done - job will never run again

[zacharya19]

Quick reproduce:

1. docker-compose up -d.
2. create job with this json:

{
    "name": "test_job_1",
    "command": "sleep 50",
    "schedule": "@every 10s",
    "concurrency": "forbid"
}

3. docker-compose restart dkron.

Now you can see dkron thinks the job is running, but it's not and forever we will see scheduler: Skipping execution.

EDIT: it's happening also when restarting dkron service (not only SIGKILL).

[koolay]

I have the same problem.

This code has a concurrency scene bug, should be with a distributed lock.

func (j *Job) isRunnable() bool {
	status := j.Status()

	if status == Running {
		if j.Concurrency == ConcurrencyAllow {
			return true
		} else if j.Concurrency == ConcurrencyForbid {
			log.WithFields(logrus.Fields{
				"job":         j.Name,
				"concurrency": j.Concurrency,
				"job_status":  status,
			}).Debug("scheduler: Skipping execution")
			return false
		}
	}

	return true
}

[zacharya19]

I believe each execution should save the process id so dkron would be able to check the status.

[vcastellm]

Fixed in #359

[zacharya19]

@Victorcoder I think the issue still exists.
I added GetStatus to the log and I see this:
concurrency=forbid get_status_function=failed job=test_job_1 job_status=running node=39d4b2198b72

I think it's because the check for:
if j.Status == StatusNotSet {

[vcastellm]

Should be fixed in #383