Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add OpenLineage support for BigQueryToBigQueryOperator #44214

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

kacpermuda
Copy link
Contributor

@kacpermuda kacpermuda commented Nov 20, 2024

This PR adds OpenLineage support for BigQueryToBigQueryOperator.

Within the operator itself, i removed the additional call to BQ API that got the job configuration as it's already returned by method that's submitting job - I adjusted the code to take advantage of that. The configuration returned is also saved as instance attribute for later use of OpenLineage method.

In the same time, I'm modifying two internal OpenLineage utils function:

  • get_facets_from_bq_table now do not return facets instead of returning empty facets when there is no schema or description for bq table
  • get_identity_column_lineage_facet is now checking if the source columns included in column lineage facet are actually in the schema of source datasets. It's now possible to generate this facet when source tables contain subset of columns of a destination table, which can be a case f.e. in BQ to BQ copy.

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

@boring-cyborg boring-cyborg bot added area:providers provider:google Google (including GCP) related issues labels Nov 20, 2024
Signed-off-by: Kacper Muda <mudakacper@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area:providers provider:google Google (including GCP) related issues
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant