add root parent information to OpenLineage events #49237
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
mobuchowski
requested review from
ashb,
eladkal,
gopidesupavan,
josh-fell,
mik-laj,
o-nikolas and
potiuk
as code owners
boring-cyborg
bot
added
area:dev-tools
area:providers
kind:documentation
provider:amazon
mobuchowski
force-pushed
the
handle-root-parent-information
branch
7 times, most recently
from
f7d7ccc to
b800438
Compare
Contributor
|
@mobuchowski test failures on this one. Is it important before today's provider wave? |
mobuchowski
force-pushed
the
handle-root-parent-information
branch
from
b800438 to
fe603ef
Compare
Contributor
Author
|
@amoghrajesh looking at this right now. |
Contributor
Author
|
@amoghrajesh rebase fixed those issues. |
kacpermuda
reviewed
Apr 18, 2025
providers/dbt/cloud/src/airflow/providers/dbt/cloud/utils/openlineage.py
Show resolved
Hide resolved
providers/openlineage/src/airflow/providers/openlineage/utils/utils.py
Outdated
Show resolved
Hide resolved
providers/openlineage/src/airflow/providers/openlineage/utils/utils.py
Outdated
Show resolved
Hide resolved
providers/openlineage/src/airflow/providers/openlineage/plugins/listener.py
Outdated
Show resolved
Hide resolved
providers/snowflake/src/airflow/providers/snowflake/utils/openlineage.py
Show resolved
Hide resolved
providers/openlineage/src/airflow/providers/openlineage/plugins/macros.py
Show resolved
Hide resolved
mobuchowski
force-pushed
the
handle-root-parent-information
branch
from
fe603ef to
39966e0
Compare
mobuchowski
force-pushed
the
handle-root-parent-information
branch
5 times, most recently
from
1c8d095 to
def2bf1
Compare
kacpermuda
approved these changes
Apr 24, 2025
Contributor
kacpermuda
left a comment
kacpermuda
left a comment
There was a problem hiding this comment.
Thanks for all the changes, LGTM.
Contributor
Author
|
@amoghrajesh want to take another look at that one? @kacpermuda reviewed it and approved. |
Member
|
needs conflict resolving |
mobuchowski
force-pushed
the
handle-root-parent-information
branch
2 times, most recently
from
c3511a3 to
7a5a871
Compare
Contributor
Author
|
@potiuk done. |
potiuk
reviewed
May 5, 2025
providers/dbt/cloud/src/airflow/providers/dbt/cloud/utils/openlineage.py
Show resolved
Hide resolved
potiuk
reviewed
May 5, 2025
potiuk
approved these changes
May 5, 2025
Member
potiuk
left a comment
potiuk
left a comment
There was a problem hiding this comment.
One NIT - with version consistency
Signed-off-by: Maciej Obuchowski <maciej.obuchowski@datadoghq.com>
mobuchowski
force-pushed
the
handle-root-parent-information
branch
from
7a5a871 to
9760a17
Compare
potiuk
approved these changes
May 5, 2025
mvfc
pushed a commit
to mvfc/airflow
that referenced
this pull request
May 6, 2025
Signed-off-by: Maciej Obuchowski <maciej.obuchowski@datadoghq.com>
potiuk
added a commit
to potiuk/airflow
that referenced
this pull request
May 6, 2025
The apache#49237 did not bump openlineage provider. This PR fixes it.
potiuk
added a commit
that referenced
this pull request
May 6, 2025
The #49237 did not bump openlineage provider. This PR fixes it.
31 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR extends OpenLineage integration in Airflow to support root parent lineage metadata via the newly added
rootfield in the ParentRunFacet.Previously, consumers had to recursively traverse parent references to determine root job in lineage hierarchy. With this update, consumers can immediately identify the DAG-level root job and run for any depth of children task.
More on the underlying OpenLineage feature: https://docs.google.com/document/d/1JypxrJc0MkrarzYfMopK-TF8QiPpjPnk2je4mgq9e4k
This updates both
openlineageprovider as few others that generate OL events by themselves.