Skip to content

Conversation

@uranusjr
Copy link
Member

@uranusjr uranusjr commented Jan 17, 2025

See AIP-83 amendment to support classic Airflow authoring style and Option 2 clarification doc WIP (Question 3) for context.

This is one of the less controversial approaches to solve ordering and run ID generation for runs with a null logical date.

Depending on the run type, the value is set to:

  • BACKFILL_JOB: DagRunInfo.run_after
  • SCHEDULED: DagModel.next_run_create_after (this is the same as backfill, only calculated earlier in the scheduler loop and stored in db)
  • MANUAL: data_interval.end
  • ASSET_TRIGGERED: Creation time of the last triggering asset event

@boring-cyborg boring-cyborg bot added area:API Airflow's REST/HTTP API area:CLI area:db-migrations PRs with DB migration area:Scheduler including HA (high availability) scheduler kind:documentation labels Jan 17, 2025
@uranusjr uranusjr added the legacy api Whether legacy API changes should be allowed in PR label Jan 17, 2025
@uranusjr uranusjr force-pushed the dagrun-run-after branch 10 times, most recently from cb4b87a to 8e0022f Compare January 20, 2025 05:49
@uranusjr
Copy link
Member Author

Hell yeah this works. We shouldn’t merge this before we resolve the discussion on this, but feel free to review the changes and get an idea what will change (not a lot).

@uranusjr uranusjr changed the title [WIP] Add run_after column to DagRun model Add run_after column to DagRun model Jan 20, 2025
@Lee-W Lee-W self-requested a review January 20, 2025 07:46
Copy link
Member

@Lee-W Lee-W left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

one nitpick: everything else looks good

Depending on the run type, the value is set to:

* BACKFILL_JOB: DagRunInfo.run_after
* SCHEDULED: DagModel.next_run_create_after (this is the same as the
  backfill variant, only calculated pre-emptively and stored in db)
* MANUAL: data_interval.end
* ASSET_TRIGGERED: Creation time of the last triggering asset event
@uranusjr uranusjr force-pushed the dagrun-run-after branch 2 times, most recently from 475e1d6 to 6db9e07 Compare February 3, 2025 07:40
@uranusjr uranusjr merged commit 880b067 into apache:main Feb 3, 2025
87 of 89 checks passed
@uranusjr uranusjr deleted the dagrun-run-after branch February 3, 2025 10:44
dabla pushed a commit to dabla/airflow that referenced this pull request Feb 3, 2025
* Add run_after column to DagRun model

* Set run_after when creating DAG run

    Depending on the run type, the value is set to:

    * BACKFILL_JOB: DagRunInfo.run_after
    * SCHEDULED: DagModel.next_run_create_after (this is the same as the
      backfill variant, only calculated pre-emptively and stored in db)
    * MANUAL: data_interval.end
    * ASSET_TRIGGERED: Creation time of the last triggering asset event

* Pass run_after to create_dagrun in tests

* Pass run_after to DagRun in tests

* Set default run_after to now
ambika-garg pushed a commit to ambika-garg/airflow that referenced this pull request Feb 4, 2025
* Add run_after column to DagRun model

* Set run_after when creating DAG run

    Depending on the run type, the value is set to:

    * BACKFILL_JOB: DagRunInfo.run_after
    * SCHEDULED: DagModel.next_run_create_after (this is the same as the
      backfill variant, only calculated pre-emptively and stored in db)
    * MANUAL: data_interval.end
    * ASSET_TRIGGERED: Creation time of the last triggering asset event

* Pass run_after to create_dagrun in tests

* Pass run_after to DagRun in tests

* Set default run_after to now
niklasr22 pushed a commit to niklasr22/airflow that referenced this pull request Feb 8, 2025
* Add run_after column to DagRun model

* Set run_after when creating DAG run

    Depending on the run type, the value is set to:

    * BACKFILL_JOB: DagRunInfo.run_after
    * SCHEDULED: DagModel.next_run_create_after (this is the same as the
      backfill variant, only calculated pre-emptively and stored in db)
    * MANUAL: data_interval.end
    * ASSET_TRIGGERED: Creation time of the last triggering asset event

* Pass run_after to create_dagrun in tests

* Pass run_after to DagRun in tests

* Set default run_after to now
ambika-garg pushed a commit to ambika-garg/airflow that referenced this pull request Feb 17, 2025
* Add run_after column to DagRun model

* Set run_after when creating DAG run

    Depending on the run type, the value is set to:

    * BACKFILL_JOB: DagRunInfo.run_after
    * SCHEDULED: DagModel.next_run_create_after (this is the same as the
      backfill variant, only calculated pre-emptively and stored in db)
    * MANUAL: data_interval.end
    * ASSET_TRIGGERED: Creation time of the last triggering asset event

* Pass run_after to create_dagrun in tests

* Pass run_after to DagRun in tests

* Set default run_after to now
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:API Airflow's REST/HTTP API area:CLI area:db-migrations PRs with DB migration area:Scheduler including HA (high availability) scheduler kind:documentation legacy api Whether legacy API changes should be allowed in PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants