Skip to content

Comments

feat: Add OpenLineage methods to DatabricksHook#62179

Merged
mobuchowski merged 1 commit intoapache:mainfrom
kacpermuda:feat-ol-databricks-hook-hll
Feb 23, 2026
Merged

feat: Add OpenLineage methods to DatabricksHook#62179
mobuchowski merged 1 commit intoapache:mainfrom
kacpermuda:feat-ol-databricks-hook-hll

Conversation

@kacpermuda
Copy link
Contributor

Adding OL methods to DatabricksHook, so that we can compute databricks OL namespace when gathering hook level lineage #62171 . Ol namespace is required when creating ExternalQueryRunFacet, that contains query_id of executed query.

So far we've only added those methods to DbApiHook based classes, but we can also add them to other hooks - they will not be used in BaseSQLOperator children anyway, as there is a check for DbApiHook inheritance there. So we can safely add OL db methods to different hooks, so that our sql parser can take advantage of it.


Was generative AI tooling used to co-author this PR?
  • Yes (please specify the tool below)

  • Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
  • For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
  • When adding dependency, check compliance with the ASF 3rd Party License Policy.
  • For significant user-facing changes create newsfragment: {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.

@kacpermuda
Copy link
Contributor Author

@mobuchowski ptal

@kacpermuda kacpermuda force-pushed the feat-ol-databricks-hook-hll branch from 7e47624 to 6df0216 Compare February 20, 2026 07:50
@kacpermuda kacpermuda force-pushed the feat-ol-databricks-hook-hll branch from 6df0216 to e575a9c Compare February 20, 2026 09:25
Copy link
Contributor

@mobuchowski mobuchowski left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we need to standarize databricks lineage too in OpenLineage's dataset naming

@mobuchowski mobuchowski merged commit e75e443 into apache:main Feb 23, 2026
90 checks passed
@kacpermuda kacpermuda deleted the feat-ol-databricks-hook-hll branch February 23, 2026 13:54
xvega pushed a commit to xvega/airflow that referenced this pull request Feb 24, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants