ExternalTaskSensor does not fail when failed_states is set along with a execution_date_fn

[eitanme]

Apache Airflow version: 2.x including main

What happened:

I am using an execution_date_fn in an ExternalTaskSensor that also sets allowed_states=['success'] and failed_states=['failed']. When one of the N upstream tasks fails, the sensor will hang forever in the poke method because there is a bug in checking for failed_states.

What you expected to happen:

I would expect the ExternalTaskSensor to fail.

I think this is due to a bug in the poke method where it should check if count_failed > 0 as opposed to checking count_failed == len(dttm_filter). I've created a fix locally that works for my case and have submitted a PR #16205 for it as reference.

How to reproduce it:

Create any ExternalTaskSensor that checks for failed_states and have one of the external DAGs tasks fail while others succeed.

E.g.

ExternalTaskSensor(
        task_id='check_external_dag',                                                     
        external_dag_id='external_dag',                                                               
        external_task_id=None,                              
        execution_date_fn=dependent_date_fn,
        allowed_states=['success'],
        failed_states=['failed'],                                                                     
        check_existence=True)

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template!

[o-nikolas]

Looks like this bug is still present and the proposed PR was marked as stale due to inactivity.
@eitanme are you willing to open another PR, to try get this one pushed through?

[eitanme]

@o-nikolas it's been a bit, but I can try to open a new PR (or re-open the old one) if you're up for reviewing and helping me get it through? The last one didn't see much in the way of activity or responses...

[o-nikolas]

Hey @eitanme,

I do apologize about that, it wasn't a great experience for your first contribution! But we really appreciate your time, I'll definitely have a look at the PR if you re-raise it, feel free to ping me directly on it once it's up.

[eitanme]

@o-nikolas , OK, I've re-opened the PR after merging in main as #27190.

We can pick up discussion over there, hopefully it's pretty straightforward.