Fix scheduler concurrency with multiple clusters

[maerteijn]

This pull request fixes two issues:

• Prevents that the Sentinel is running the Scheduler on startup (even when we added scheduler: False to the configuration).

• Lock the selected rows with select_for_update when creating tasks from Scheduler items. This makes sure that tasks are created only once (fixes scheduler creating duplicate tasks in multiple cluster environment #231). I tested this with PostgreSQL and MySQL.

[codecov-io]

Codecov Report

Merging #347 into master will not change coverage.
The diff coverage is 94%.

@@          Coverage Diff           @@
##           master    #347   +/-   ##
======================================
  Coverage    90.4%   90.4%           
======================================
  Files          43      43           
  Lines        2846    2846           
======================================
  Hits         2573    2573           
  Misses        273     273

Impacted Files	Coverage Δ
django_q/cluster.py	91.7% <94%> (ø)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 8a6c9d2...958c79e. Read the comment docs.

[randlet]

@maerteijn Thanks for prepping this! I think it will make it possible for me to run in e.g. an AWS Auto Scaling group where I have 2-N servers all running qcluster and not have any tasks duplicated is that right? Are you running this in production now?

[maerteijn]

@maerteijn Thanks for prepping this! I think it will make it possible for me to run in e.g. an AWS Auto Scaling group where I have 2-N servers all running qcluster and not have any tasks duplicated is that right? Are you running this in production now?

Yes and yes 😃

[randlet]

@maerteijn Thanks! I experimented a bit with your branch and so far no duplicate tasks while running clusters on 4 separate servers :)

[Koed00]

Broke this PR by fixing the Arrow issue. If you could change the 'replace' to 'shift' in your code, we should be good.

[maerteijn]

Ok, I’ll try to fix this soon(ish)

[maerteijn]

I updated the pull request