`ExternalTaskSensor` deferrable doesn't fail with `failed_states`

[mjohansenwork]

Apache Airflow version

2.9.3

If "Other Airflow 2 version" selected, which one?

No response

What happened?

For ExternalTaskSensor, when I specify deferrable=True and failed_states=["failed"], the operator hangs in deferred mode and repeatedly pokes the upstream DAG status. This works fine if I don't use deferrable.

What you think should happen instead?

I would expect this to fail immediately if the upstream DAG status is failed.

How to reproduce

Create 2 DAGs, upstream and downstream. Make the upstream DAG fail intentionally. Make the downstream DAG depend on the upstream DAG using ExternalTaskSensor, setting deferrable=True and failed_states=["failed"]. Observe that the downstream ExternalTaskSensor loops and pokes continuously, even though the upstream DAG has failed.

Operating System

MacOS

Versions of Apache Airflow Providers

No response

Deployment

Other Docker-based deployment

Deployment details

I'm using the Astronomer runtime 11.7.0 docker image.

Anything else?

I noticed that in deferrable mode, the failed_states don't appear to be forwarded to the WorkflowTrigger here, which seems like a bug.
Possibly related:
#16204 and #27190 #34207 and #34204

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

[mjohansenwork]

Initial stab at PR #41141

[shahar1]

Initial stab at PR #41141

Seems like a good catch!
I'll be happy to review it when you're done.