Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allowing task_group Sensor in ExternalTaskSensor #24902

Merged
merged 14 commits into from
Aug 22, 2022

Conversation

pateash
Copy link
Contributor

@pateash pateash commented Jul 7, 2022

closes: #14563
related: #14640

Add meaningful description above
Hi All,

This PR proposes to allow users to provide external_task_group_id in ExternalTaskSensor to make tasks on a DAG wait for another task_group on a different DAG for a specific execution_date.
we can also provide options to set if the Task on a remote DAG succeeded or failed via allowed_states and failed_states parameters.

@boring-cyborg boring-cyborg bot added area:core-operators Operators, Sensors and hooks within Core Airflow kind:documentation labels Jul 7, 2022
@pateash pateash changed the title allowing task_group Sensor in ExternalTaskSensor Allowing task_group Sensor in ExternalTaskSensor Jul 7, 2022
@pateash
Copy link
Contributor Author

pateash commented Jul 7, 2022

cases

  1. group1_sensor -> wait for task_group from external dag which exists and succeeded ( as per allowed_states )
  2. group2_sensor- > wait for task_group from external dag which doesn't exists ( wait till timeout )
  3. group2_sensor_check_enabled -> wait for task_group from external dag which doesn't exists and with check_enabled=True ( fail with error )

image

@pateash pateash force-pushed the airflow-14563 branch 2 times, most recently from 1425c95 to eb29793 Compare July 9, 2022 22:20
airflow/models/dag.py Outdated Show resolved Hide resolved
airflow/models/dag.py Outdated Show resolved Hide resolved
airflow/sensors/external_task.py Show resolved Hide resolved
@pateash
Copy link
Contributor Author

pateash commented Jul 23, 2022

Hi @eladkal, could you please take a look into this PR.

@uranusjr
Copy link
Member

Since we are already allowing multiple task IDs, I wonder if we should just go directly with external_group_ids instead of just one single group. Someone’s going to ask for that sooner or later.

@pateash
Copy link
Contributor Author

pateash commented Jul 28, 2022

@uranusjr, we already allow multiple taskIds which I personally don't think a good idea, because it is not very transparent.

Although For TaskGroup, I think if someone want to depend on multiple of them, they must create different task sensors.

tests/sensors/test_external_task_sensor.py Outdated Show resolved Hide resolved
tests/sensors/test_external_task_sensor.py Outdated Show resolved Hide resolved
@eladkal eladkal added this to the Airflow 2.4.0 milestone Aug 3, 2022
Copy link
Contributor

@eladkal eladkal left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@eladkal eladkal merged commit bc04c5f into apache:main Aug 22, 2022
@pateash pateash deleted the airflow-14563 branch August 23, 2022 07:42
@ephraimbuddy ephraimbuddy added the type:new-feature Changelog: New Features label Sep 13, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area:core-operators Operators, Sensors and hooks within Core Airflow kind:documentation type:new-feature Changelog: New Features
Projects
None yet
Development

Successfully merging this pull request may close these issues.

TaskGroup Sensor
4 participants