Skip to content

Conversation

@Lee-W
Copy link
Member

@Lee-W Lee-W commented Oct 21, 2025

Why

Closes: #56763

how to reproduce (through breeze on e24cfb0)

  1. Add the following lines to files/airflow-breeze-config/environment_variables.env
AIRFLOW__SCHEDULER__TASK_INSTANCE_HEARTBEAT_TIMEOUT=1
AIRFLOW__SCHEDULER__TASK_INSTANCE_HEARTBEAT_TIMEOUT_DETECTION_INTERVAL=5
  1. Add the following dag to files/dags
Dag
from __future__ import annotations

import os
import time

import pendulum

from airflow.sdk import Asset, Metadata, dag, task

asset = Asset("test")


@dag(schedule=None, start_date=pendulum.datetime(2025, 1, 1), catchup=False)
def producer():
    @task(outlets=[asset])
    def producer_task():
        yield Metadata(asset=asset, extra={"test": 1})

    producer_task()


@dag(schedule=asset, start_date=pendulum.datetime(2025, 1, 1), catchup=False)
def consumer():
    @task()
    def consumer_task(ds_str):
        print(ds_str)
        time.sleep(5)
        os._exit(1)  # immediate exit, no cleanup, no heartbeat

    consumer_task("{{ (triggering_asset_events.values() | first | last).extra }}")


producer()
consumer()
  1. breeze start-airflow --db-reset --backend postgres
  2. Turn on both produer and conumser Dags
  3. Continously trigger producer (10 ~ 20 times) really quick

What

Kaxil implemented a _eager_load_dag_run_for_validation method to ensure that dag_run exists even when a ti becomes a zombie task


^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.

@Lee-W Lee-W requested review from XD-DENG and ashb as code owners October 21, 2025 01:13
@boring-cyborg boring-cyborg bot added the area:Scheduler including HA (high availability) scheduler label Oct 21, 2025
@Lee-W Lee-W requested review from kaxil and removed request for XD-DENG and ashb October 21, 2025 01:13
@Lee-W Lee-W force-pushed the fix-asset-DetachedInstanceError branch from 6f35b7f to d896184 Compare October 21, 2025 01:17
Copy link
Member

@kaxil kaxil left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Were you able to reproduce the error & does this fix it. (It should but worth verifying)

@Lee-W
Copy link
Member Author

Lee-W commented Oct 21, 2025

Were you able to reproduce the error & does this fix it. (It should but worth verifying)

yep, I'm now simplifying the sample code I used and will add to the desciprion

@Lee-W
Copy link
Member Author

Lee-W commented Oct 21, 2025

Updated the Dag and steps I used in detail

@kaxil kaxil added this to the Airflow 3.1.1 milestone Oct 21, 2025
@Lee-W Lee-W merged commit 8ffb7b1 into apache:main Oct 21, 2025
60 checks passed
@Lee-W Lee-W deleted the fix-asset-DetachedInstanceError branch October 21, 2025 02:07
@Lee-W Lee-W added the backport-to-v3-1-test Mark PR with this label to backport to v3-1-test branch label Oct 21, 2025
github-actions bot pushed a commit to aws-mwaa/upstream-to-airflow that referenced this pull request Oct 21, 2025
…alidation (apache#56916)

(cherry picked from commit 8ffb7b1)

Co-authored-by: Wei Lee <weilee.rx@gmail.com>
potiuk added a commit to potiuk/airflow that referenced this pull request Oct 21, 2025
@potiuk
Copy link
Member

potiuk commented Oct 21, 2025

Hey @Lee-W -> we had to revert it as it fails for Python 3.13. Can you please re-do it (and set all versions label on PR to trigger the tests for 3.13.

See https://github.com/apache/airflow/actions/runs/18676364216/job/53249204014

potiuk added a commit that referenced this pull request Oct 21, 2025
Jonpaco23 pushed a commit to Jonpaco23/airflow that referenced this pull request Oct 21, 2025
potiuk added a commit to potiuk/airflow that referenced this pull request Oct 21, 2025
…on API validation (apache#56916)" (apache#56943)

This reverts commit 8ffb7b1.
(cherry picked from commit 209fb2e)

Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
potiuk added a commit that referenced this pull request Oct 21, 2025
…on API validation (#56916)" (#56943) (#56956)

This reverts commit 8ffb7b1.
(cherry picked from commit 209fb2e)
Lee-W added a commit to astronomer/airflow that referenced this pull request Oct 22, 2025
potiuk pushed a commit that referenced this pull request Oct 22, 2025
…57010)

* Reapply "fix(scheduler): load dag_run eagering for execution API validation (#56916)" (#56943)

This reverts commit 209fb2e.

* fix(scheduler): joinedload TI.dag_run instead of selectinload
github-actions bot pushed a commit that referenced this pull request Oct 22, 2025
…alidation (#57010)

* Reapply "fix(scheduler): load dag_run eagering for execution API validation (#56916)" (#56943)

This reverts commit 209fb2e.

* fix(scheduler): joinedload TI.dag_run instead of selectinload
(cherry picked from commit 833717a)

Co-authored-by: Wei Lee <weilee.rx@gmail.com>
Lee-W added a commit that referenced this pull request Oct 22, 2025
…alidation (#57010) (#57045)

Co-authored-by: Wei Lee <weilee.rx@gmail.com>
fix(scheduler): load dag_run eagering for execution API validation (#56916)" (#56943)
kaxil pushed a commit to astronomer/airflow that referenced this pull request Oct 22, 2025
…alidation (apache#57010) (apache#57045)

Co-authored-by: Wei Lee <weilee.rx@gmail.com>
fix(scheduler): load dag_run eagering for execution API validation (apache#56916)" (apache#56943)
pierrejeambrun added a commit that referenced this pull request Oct 22, 2025
…57062)

* [v3-1-test] Fix Outlet Event Extra Data is Empty in Task Instance Success Listener (#54568) (#57031)

Co-authored-by: Kevin Yang <85313829+sjyangkevin@users.noreply.github.com>

* [v3-1-test] Fix Extra Links translation key. (#57030) (#57040)

(cherry picked from commit 8d71c07)

Co-authored-by: Karthikeyan Singaravelan <tir.karthi@gmail.com>

* [v3-1-test] Fix dags list not rendering (#57037) (#57046)

(cherry picked from commit 8668fe5)

Co-authored-by: Pierre Jeambrun <pierrejbrun@gmail.com>

* [v3-1-test] fix(scheduler): load dag_run eagering for execution API validation (#57010) (#57045)

Co-authored-by: Wei Lee <weilee.rx@gmail.com>
fix(scheduler): load dag_run eagering for execution API validation (#56916)" (#56943)

* [v3-1-test] Fix dag list cache invalidation on favouriting a dag (#57039) (#57047)

Fix dag list cache invalidation on favouriting a dag
(cherry picked from commit 4e46212)

Co-authored-by: Pierre Jeambrun <pierrejbrun@gmail.com>

* [v3-1-test] Add new PL translations. (#56359) (#57055)

(cherry picked from commit 6583b6d)

* Add last missing translation for 3.1.1 (#57057)

* [v3-1-test] Close German translation gaps for full UI translation 2025-10-21 (#56980) (#57048)

* [v3-1-test] Fix: Adjust PanelButtons spacing and alignment (#56570)

* fix: adjust PanelButtons spacing and alignment

* Fix PanelButtons layout and button styling

* Fix PanelButtons: use bg.subtle and remove unnecessary _hover
(cherry picked from commit 4c35cd2)

Co-authored-by: anshuksi282-ksolves <anshu.singh@ksolves.com>

---------

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Kevin Yang <85313829+sjyangkevin@users.noreply.github.com>
Co-authored-by: Karthikeyan Singaravelan <tir.karthi@gmail.com>
Co-authored-by: Pierre Jeambrun <pierrejbrun@gmail.com>
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
Co-authored-by: anshuksi282-ksolves <anshu.singh@ksolves.com>
kaxil pushed a commit that referenced this pull request Oct 23, 2025
…57062)

* [v3-1-test] Fix Outlet Event Extra Data is Empty in Task Instance Success Listener (#54568) (#57031)

Co-authored-by: Kevin Yang <85313829+sjyangkevin@users.noreply.github.com>

* [v3-1-test] Fix Extra Links translation key. (#57030) (#57040)

(cherry picked from commit 8d71c07)

Co-authored-by: Karthikeyan Singaravelan <tir.karthi@gmail.com>

* [v3-1-test] Fix dags list not rendering (#57037) (#57046)

(cherry picked from commit 8668fe5)

Co-authored-by: Pierre Jeambrun <pierrejbrun@gmail.com>

* [v3-1-test] fix(scheduler): load dag_run eagering for execution API validation (#57010) (#57045)

Co-authored-by: Wei Lee <weilee.rx@gmail.com>
fix(scheduler): load dag_run eagering for execution API validation (#56916)" (#56943)

* [v3-1-test] Fix dag list cache invalidation on favouriting a dag (#57039) (#57047)

Fix dag list cache invalidation on favouriting a dag
(cherry picked from commit 4e46212)

Co-authored-by: Pierre Jeambrun <pierrejbrun@gmail.com>

* [v3-1-test] Add new PL translations. (#56359) (#57055)

(cherry picked from commit 6583b6d)

* Add last missing translation for 3.1.1 (#57057)

* [v3-1-test] Close German translation gaps for full UI translation 2025-10-21 (#56980) (#57048)

* [v3-1-test] Fix: Adjust PanelButtons spacing and alignment (#56570)

* fix: adjust PanelButtons spacing and alignment

* Fix PanelButtons layout and button styling

* Fix PanelButtons: use bg.subtle and remove unnecessary _hover
(cherry picked from commit 4c35cd2)

Co-authored-by: anshuksi282-ksolves <anshu.singh@ksolves.com>

---------

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Kevin Yang <85313829+sjyangkevin@users.noreply.github.com>
Co-authored-by: Karthikeyan Singaravelan <tir.karthi@gmail.com>
Co-authored-by: Pierre Jeambrun <pierrejbrun@gmail.com>
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
Co-authored-by: anshuksi282-ksolves <anshu.singh@ksolves.com>
potiuk added a commit that referenced this pull request Oct 25, 2025
… en (#57060) (#57064)

* [v3-1-test] Fix Outlet Event Extra Data is Empty in Task Instance Success Listener (#54568) (#57031)

Co-authored-by: Kevin Yang <85313829+sjyangkevin@users.noreply.github.com>

* [v3-1-test] Fix: Adjust PanelButtons spacing and alignment (#56570) (#57062)

* [v3-1-test] Fix Outlet Event Extra Data is Empty in Task Instance Success Listener (#54568) (#57031)

Co-authored-by: Kevin Yang <85313829+sjyangkevin@users.noreply.github.com>

* [v3-1-test] Fix Extra Links translation key. (#57030) (#57040)

(cherry picked from commit 8d71c07)

Co-authored-by: Karthikeyan Singaravelan <tir.karthi@gmail.com>

* [v3-1-test] Fix dags list not rendering (#57037) (#57046)

(cherry picked from commit 8668fe5)

Co-authored-by: Pierre Jeambrun <pierrejbrun@gmail.com>

* [v3-1-test] fix(scheduler): load dag_run eagering for execution API validation (#57010) (#57045)

Co-authored-by: Wei Lee <weilee.rx@gmail.com>
fix(scheduler): load dag_run eagering for execution API validation (#56916)" (#56943)

* [v3-1-test] Fix dag list cache invalidation on favouriting a dag (#57039) (#57047)

Fix dag list cache invalidation on favouriting a dag
(cherry picked from commit 4e46212)

Co-authored-by: Pierre Jeambrun <pierrejbrun@gmail.com>

* [v3-1-test] Add new PL translations. (#56359) (#57055)

(cherry picked from commit 6583b6d)

* Add last missing translation for 3.1.1 (#57057)

* [v3-1-test] Close German translation gaps for full UI translation 2025-10-21 (#56980) (#57048)

* [v3-1-test] Fix: Adjust PanelButtons spacing and alignment (#56570)

* fix: adjust PanelButtons spacing and alignment

* Fix PanelButtons layout and button styling

* Fix PanelButtons: use bg.subtle and remove unnecessary _hover
(cherry picked from commit 4c35cd2)

Co-authored-by: anshuksi282-ksolves <anshu.singh@ksolves.com>

---------

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Kevin Yang <85313829+sjyangkevin@users.noreply.github.com>
Co-authored-by: Karthikeyan Singaravelan <tir.karthi@gmail.com>
Co-authored-by: Pierre Jeambrun <pierrejbrun@gmail.com>
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
Co-authored-by: anshuksi282-ksolves <anshu.singh@ksolves.com>

* [v3-1-test] Add possibility of removing extra translations missing in en (#57060)

Sometimes we rename/move translation - we did not have a tool to
remove such dangling translations. Now we have.
(cherry picked from commit 2314099)

Co-authored-by: Jarek Potiuk <jarek@potiuk.com>

---------

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Kevin Yang <85313829+sjyangkevin@users.noreply.github.com>
Co-authored-by: Karthikeyan Singaravelan <tir.karthi@gmail.com>
Co-authored-by: Pierre Jeambrun <pierrejbrun@gmail.com>
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
Co-authored-by: anshuksi282-ksolves <anshu.singh@ksolves.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:Scheduler including HA (high availability) scheduler backport-to-v3-1-test Mark PR with this label to backport to v3-1-test branch

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Scheduler is crashing when using Assets + the presence of Zombie tasks

3 participants