Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

AIP-62: add hook lineage collection support for S3Hook #40819

Merged
merged 1 commit into from
Jul 22, 2024
Merged

Conversation

mobuchowski
Copy link
Contributor

@mobuchowski mobuchowski commented Jul 16, 2024

This PR implements #40819.

This adds support for automatic lineage collection from S3Hook (non-async methods) and adds support for S3 dataset.

@mobuchowski mobuchowski added the AIP-62 Tasks tracking implementation of AIP-62 Getting Lineage from Hook Instrumentation label Jul 16, 2024
@mobuchowski mobuchowski force-pushed the aip-62/s3hook branch 4 times, most recently from 6e2cfb5 to d34b054 Compare July 16, 2024 22:22
@mobuchowski mobuchowski force-pushed the aip-62/s3hook branch 2 times, most recently from 3418daa to b6508e4 Compare July 16, 2024 23:45
@mobuchowski mobuchowski force-pushed the aip-62/s3hook branch 2 times, most recently from dbca2ae to c27ffcd Compare July 17, 2024 09:43
Copy link
Member

@potiuk potiuk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved with one nit about compat dependency.

Signed-off-by: Maciej Obuchowski <obuchowski.maciej@gmail.com>
@mobuchowski mobuchowski merged commit 0212f67 into main Jul 22, 2024
106 checks passed
@ephraimbuddy ephraimbuddy added the changelog:skip Changes that should be skipped from the changelog (CI, tests, etc..) label Jul 22, 2024
@ephraimbuddy ephraimbuddy added this to the Airflow 2.10.0 milestone Jul 23, 2024
romsharon98 pushed a commit to romsharon98/airflow that referenced this pull request Jul 26, 2024
Signed-off-by: Maciej Obuchowski <obuchowski.maciej@gmail.com>
@mobuchowski mobuchowski deleted the aip-62/s3hook branch August 2, 2024 13:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
AIP-62 Tasks tracking implementation of AIP-62 Getting Lineage from Hook Instrumentation area:dev-tools area:providers changelog:skip Changes that should be skipped from the changelog (CI, tests, etc..) provider:amazon-aws AWS/Amazon - related issues
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants