Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[AIRFLOW-56] Airflow's scheduler can "lose" queued tasks #1378

Merged
merged 2 commits into from
May 9, 2016
Merged

[AIRFLOW-56] Airflow's scheduler can "lose" queued tasks #1378

merged 2 commits into from
May 9, 2016

Commits on May 9, 2016

  1. Handle queued tasks from multiple jobs/executors 

    When Scheduler is run with `—num-runs`, there can be multiple
Schedulers and Executors all trying to run tasks. For queued tasks,
Scheduler was previously only trying to run tasks that it itself had
queued — but that doesn’t work if the Scheduler is restarting. This PR
reverts that behavior and adds two types of “best effort” executions —
before running a TI, executors check if it is already running, and
before ending executors call sync() one last time
    @jlowin
    jlowin committed May 9, 2016
    Configuration menu
    Copy the full SHA
    43bdd7a View commit details
    Browse the repository at this point in the history

  2. Add logic to lock DB and avoid race condition 

    The scheduler can encounter a queued task twice before the
task actually starts to run -- this locks the task and avoids
that condition.
    @jlowin
    jlowin committed May 9, 2016
    Configuration menu
    Copy the full SHA
    c1aa93f View commit details
    Browse the repository at this point in the history