Skip to content

Conversation

@mobuchowski
Copy link
Contributor

When receiving malformed file assets with non-absolute paths, converted assets will contain file name in namespace. This does not match OL naming scheme and can cause explosion in namespace table size for OpenLineage users.

Fixes OL issue: OpenLineage/OpenLineage#3557

Signed-off-by: Maciej Obuchowski <maciej.obuchowski@datadoghq.com>
@mobuchowski mobuchowski merged commit 79e3700 into apache:main Mar 17, 2025
62 checks passed
@vincbeck
Copy link
Contributor

This causes an error in main:

FAILED providers/amazon/tests/unit/amazon/aws/hooks/test_s3.py::TestAwsS3Hook::test_download_file_with_preserve_name_exposes_lineage - AssertionError: assert 0 == 1
 +  where 0 = len([])
 +    where [] = HookLineage(inputs=[AssetLineageInfo(asset=Asset(name='s3://test_bucket/test_key/test.log', uri='s3://test_bucket/test..., watchers=[]), count=1, context=<airflow.providers.amazon.aws.hooks.s3.S3Hook object at 0x7f4419c711c0>)], outputs=[]).outputs
 +      where HookLineage(inputs=[AssetLineageInfo(asset=Asset(name='s3://test_bucket/test_key/test.log', uri='s3://test_bucket/test..., watchers=[]), count=1, context=<airflow.providers.amazon.aws.hooks.s3.S3Hook object at 0x7f4419c711c0>)], outputs=[]) = <airflow.lineage.hook.HookLineageCollector object at 0x7f441fc276a0>.collected_assets

Looking into it

@vincbeck
Copy link
Contributor

Fix here: #47880

agupta01 pushed a commit to agupta01/airflow that referenced this pull request Mar 21, 2025
…#47818)

Signed-off-by: Maciej Obuchowski <maciej.obuchowski@datadoghq.com>
nailo2c pushed a commit to nailo2c/airflow that referenced this pull request Apr 4, 2025
…#47818)

Signed-off-by: Maciej Obuchowski <maciej.obuchowski@datadoghq.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants