Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

set "has_outlet_datasets" to true if "dataset alias" exists #41091

Conversation

Lee-W
Copy link
Member

@Lee-W Lee-W commented Jul 29, 2024

Why

After #40478, we're able to create a dataset event and thus potentially trigger a DAG run through setting outlets=DatasetAlias(...). However, the dataset events created in this way do not appear in the task instance detail.

image

What

This PR changes the set has_outlet_datasets to true if DatasetAlias (which could potentially trigger a DAG run) exists in outlets.


^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

@boring-cyborg boring-cyborg bot added the area:webserver Webserver related Issues label Jul 29, 2024
@Lee-W
Copy link
Member Author

Lee-W commented Jul 29, 2024

It's also possible a DAG with DatasetAlias doesn't create a dataset event. The UI will show

Screenshot 2024-07-29 at 5 52 12 PM

We could also check whether a DatasetAlias can be resolved into datasets, but I feel it might downgrade the performance without gaining much.

I would like to know what @bbovenzi thinks. Thanks!

@bbovenzi bbovenzi merged commit 36b9234 into apache:main Jul 29, 2024
48 checks passed
@bbovenzi bbovenzi deleted the add-missing-dataset-events-in-gird-view-when-dataset-alias-is-used branch July 29, 2024 16:28
@utkarsharma2 utkarsharma2 added the type:improvement Changelog: Improvements label Jul 31, 2024
@utkarsharma2 utkarsharma2 added this to the Airflow 2.10.0 milestone Jul 31, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area:webserver Webserver related Issues type:improvement Changelog: Improvements
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants