Disabling backfill functionality?

[martingrayson]

If I have a process that needs to run daily regardless of input parameters, for example a process that bulk loads a table and doesn't depend on delta data, is there a way of preventing backfills running a dag multiple times?

If I have a dag with for arguments sake, one task, can I make a backfill with a weeks' date range only run the task once rather than 7 times?

[mistercrunch]

Right, this is a common use case, look for an upcoming only_run_latest flag in BaseOperator, implemented in a similar way as depends_on_past.

It the meantime, you may want to run an individual backfill on that task with a --mark_success flag on start_date to end_date-1, then run this task individually for end_date, and then proceed to run your backfill, which will skip all the previously succeeded tasks.

only_run_latest will move all that complexity to BackfillJob and to the scheduler who will be able to skip runs when possible.

[mistercrunch]

I captured the essence of this in: #59, closing this one