Skip to content

Comments

Return list of destination URIs from GCSToGCSOperator#61320

Merged
shahar1 merged 3 commits intoapache:mainfrom
yuseok89:fix/11323-gcs-to-gcs-return-destination-uris
Feb 10, 2026
Merged

Return list of destination URIs from GCSToGCSOperator#61320
shahar1 merged 3 commits intoapache:mainfrom
yuseok89:fix/11323-gcs-to-gcs-return-destination-uris

Conversation

@yuseok89
Copy link
Contributor

@yuseok89 yuseok89 commented Feb 1, 2026

Related: #11323

  • Return value: execute() now returns a list[str] of destination URIs for copied objects (single and multiple files). This operator did not return a value before, so unwrap_single was not added.

  • Deduplication for multi-file copy: When multiple source paths map to the same destination (e.g. source_objects=["src/foo.png", "src/data/"] with foo.png under data/), the same URI can appear more than once. The result list is deduplicated while preserving order via dict.fromkeys.

  • Directory-like URIs excluded from return: If GCS list returns keys ending with / (e.g. prefix/), they are still copied, but such URIs are filtered out of the returned list so only file URIs are included.


Test screenshot

image
Was generative AI tooling used to co-author this PR?
  • Yes (please specify the tool below)
    • Cursor

  • Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
  • For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
  • When adding dependency, check compliance with the ASF 3rd Party License Policy.
  • For significant user-facing changes create newsfragment: {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.

@yuseok89 yuseok89 requested a review from shahar1 as a code owner February 1, 2026 15:08
@boring-cyborg boring-cyborg bot added area:providers provider:google Google (including GCP) related issues labels Feb 1, 2026
Copy link
Contributor

@shahar1 shahar1 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Excellent work! LGTM.
Failing CI job seems unrelated - hopefully it should be fixed soon by another PR.
Please let me handle the rebase and merging, feel free to take on other issues.
I'll wait couple of days to give the chance for Google team to review.

CC: @VladaZakharova @MaksYermak

@shahar1 shahar1 force-pushed the fix/11323-gcs-to-gcs-return-destination-uris branch from 7a33df9 to 3eaf7b0 Compare February 2, 2026 12:00
@shahar1 shahar1 merged commit 4e006ad into apache:main Feb 10, 2026
90 checks passed
Ratasa143 pushed a commit to Ratasa143/airflow that referenced this pull request Feb 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:providers provider:google Google (including GCP) related issues

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants