-
Notifications
You must be signed in to change notification settings - Fork 16.3k
Support templated extra in outlets assets #54885
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Support templated extra in outlets assets #54885
Conversation
c83fb13 to
220785a
Compare
Lee-W
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
one question, one nit. no major issue on my end
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
|
Not an issue, but a question: What's the intended use case here? |
The asset can be used to connect two DAGs for two different teams. The owner of the dataset (the team who owns the producer task) is responsible for providing some metadata to the consumer DAG to explain what to process and how. To simplify the answer, I can take a part of AIP-76 as an example. With this PR, the partition can be defined at runtime, since we can have backfills and runs created by external events, and the partition is not always known at parse time. SomeOperator(
task_id="write_asset",
...,
outlets=Asset(
"asset_example",
extra={
"partition": "{{ logical_date.strftime('%Y_%m_%d/%H') }}",
}
),
)Also, when the team owner of the dataset wants to change the partition pattern, there’s no need to notify the consumer side, as the dataset schema still respects the data contract. |
This reverts commit 438b76c.
This reverts commit 438b76c.
…ache#55199) This reverts commit 438b76c.
…ache#55199) This reverts commit 438b76c.
…ache#55199) This reverts commit 438b76c.
…ache#55199) This reverts commit 438b76c.
…ache#55199) This reverts commit 438b76c.
…ache#55199) This reverts commit 438b76c.
…ache#55199) This reverts commit 438b76c.
…ache#55199) This reverts commit 438b76c.
…ache#55199) This reverts commit 438b76c.
…ache#55199) This reverts commit 438b76c.
…ache#55199) This reverts commit 438b76c.
…ache#55199) This reverts commit 438b76c.
…ache#55199) This reverts commit 438b76c.
…ache#55199) This reverts commit 438b76c.
…ache#55199) This reverts commit 438b76c.
…ache#55199) This reverts commit 438b76c.
…ache#55199) This reverts commit 438b76c.
…ache#55199) This reverts commit 438b76c.
This PR adds support for Jinja templates in outlet assets, which will be rendered in the emitted events.
^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named
{pr_number}.significant.rstor{issue_number}.significant.rst, in airflow-core/newsfragments.