feat: Send dedicated OpenLineage events for each Snowflake query_id #47736
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
boring-cyborg
bot
added
area:providers
provider:snowflake
kacpermuda
force-pushed
the
feat-ol-snowflake-child-events
branch
3 times, most recently
from
fd96163 to
fb983ee
Compare
mobuchowski
reviewed
Mar 17, 2025
providers/snowflake/src/airflow/providers/snowflake/utils/openlineage.py
Outdated
Show resolved
Hide resolved
Contributor
Author
|
Let's wait for #47909 to be merged, so that I can adjust tests. |
kacpermuda
force-pushed
the
feat-ol-snowflake-child-events
branch
from
fb983ee to
20ea002
Compare
kacpermuda
force-pushed
the
feat-ol-snowflake-child-events
branch
3 times, most recently
from
94094b2 to
05c6058
Compare
kacpermuda
force-pushed
the
feat-ol-snowflake-child-events
branch
from
05c6058 to
1623b48
Compare
mobuchowski
reviewed
Mar 28, 2025
providers/snowflake/src/airflow/providers/snowflake/utils/openlineage.py
Show resolved
Hide resolved
providers/snowflake/src/airflow/providers/snowflake/utils/openlineage.py
Show resolved
Hide resolved
kacpermuda
force-pushed
the
feat-ol-snowflake-child-events
branch
from
1623b48 to
179605e
Compare
mobuchowski
approved these changes
Apr 1, 2025
nailo2c
pushed a commit
to nailo2c/airflow
that referenced
this pull request
Apr 4, 2025
diogotrodrigues
pushed a commit
to diogotrodrigues/airflow
that referenced
this pull request
Apr 6, 2025
simonprydden
pushed a commit
to simonprydden/airflow
that referenced
this pull request
Apr 8, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
closes: #47488
Prevent loss of information when using OpenLineage with Snowflake that is currently happening. When multiple query ids are encountered, pair of dedicated OL events will be sent for each query_id (with execution metadata whenever available).
Not sure how we should name OpenLineage job for each of this Snowflake query. For job namespace we can use the same OpenLineage namespace as Airflow is using, but for job name ? For now I wroteWe agreed on OpenLineage committer sync to go with job_name = {dag_id}.{task_id}.query.{counter} and this is how it's implemented in this PR.snowflake_query:{query_id}- but I have no idea if it makes sense.cc @mobuchowski
^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named
{pr_number}.significant.rstor{issue_number}.significant.rst, in newsfragments.