Skip to content

Comments

feat: Consume SQL hook lineage in OpenLineage#62171

Open
kacpermuda wants to merge 1 commit intoapache:mainfrom
kacpermuda:feat-ol-sql-hll
Open

feat: Consume SQL hook lineage in OpenLineage#62171
kacpermuda wants to merge 1 commit intoapache:mainfrom
kacpermuda:feat-ol-sql-hll

Conversation

@kacpermuda
Copy link
Contributor

Followup to #61535, this PR updates the OpenLineage listener to include SQL lineage reported by hooks.

For each SQL query, it parses inputs/outputs via the existing SQLParser, enriches facets (SQL job, external query), and emits dedicated START/COMPLETE (or FAIL) child OpenLineage events. When exactly one query was executed, the parsed metadata is also merged into the parent task's lineage event, for simplicity (but the separate events are always emitted).

Updated documentation template and Sphinx extension to better describe hook-level lineage capabilities and discover these new sql hook lineage hooks.


Was generative AI tooling used to co-author this PR?
  • Yes (please specify the tool below)

Generated-by: Cursor following the guidelines


  • Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
  • For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
  • When adding dependency, check compliance with the ASF 3rd Party License Policy.
  • For significant user-facing changes create newsfragment: {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant