-
Notifications
You must be signed in to change notification settings - Fork 16.5k
Description
Description
Currently some storage operators return files destination uris list for example GoogleSheetsToGCSOperator. It would be a good idea to make all of such operators to return this list.
Use case / motivation
- Having list of destination uris for files transferred with operator is useful for debugging and troubleshooting
- Inconsistent operator behaviour might be confusing for users.
Edit (by @shahar1 on Jan. 31, 2026)
For operators that currently return a single entity, add the flag unwrap_single which controls whether the returned value is either list[obj] or obj (usually obj is str but could also be a dict). Until we apply consistency for all operators (unwrap_single=False), which will be a breaking change - set for now the default to return the single entity (unwrap_single=True).
Tracking (added by @shahar1 on Jan. 24, 2026)
All operators should return list[str] of destination URIs (gs://bucket/object).
To Do (Generated by AI, please double check if current situation is as described)
-
ADLSToGCSOperator (
transfers/adls_to_gcs.py)- Current:
list[str](ADLS source paths) → Need: destination URIs - Assignee: @Abhishekmishra2808 | PR: Return destination GCS URIs from ADLSToGCSOperator #61463
- Current:
-
AzureFileShareToGCSOperator (
transfers/azure_fileshare_to_gcs.py)- Current:
list[str](source filenames) → Need: destination URIs - Assignee: @nailo2c | PR: Return list of GCS URIs from Azure*ToGCS operators #61048
- Current:
-
AzureBlobStorageToGCSOperator (
transfers/azure_blob_to_gcs.py)- Current:
str→ Need:list[str](wrap in list) - Assignee: @nailo2c | PR: Return list of GCS URIs from Azure*ToGCS operators #61048
- Current:
-
GCSToGCSOperator (
transfers/gcs_to_gcs.py)- Current:
None→ Need: destination URIs - Assignee: @yuseok89 | PR: Return list of destination URIs from GCSToGCSOperator #61320
- Current:
-
LocalFilesystemToGCSOperator (
transfers/local_to_gcs.py)- Current:
None→ Need: destination URIs - Assignee: @yuseok89 | PR: Return list of destination URIs from LocalFilesystemToGCSOperator #61355
- Current:
-
GoogleDriveToGCSOperator (
transfers/gdrive_to_gcs.py)- Current: Inconsistent → Need: destination URIs
- Assignee: @Ajay9704 | PR: Return files destination uris in
GoogleDriverToGCSOperatorandSheetsToGCSOperator#61347
-
HttpToGCSOperator (
transfers/http_to_gcs.py)- Current:
None→ Need: destination URIs - Assignee: @Srabasti | PR: Return list of destination URIs from HttpToGCSOperator #61306
- Current:
-
FacebookAdsReportToGcsOperator (
transfers/facebook_ads_to_gcs.py)- Current:
None→ Need: destination URIs - Assignee: @Abhishekmishra2808 | PR: : Return list of GCS URIs from FacebookAdsReportToGcsOperator #61349
- Current:
-
GoogleCalendarToGCSOperator (
transfers/calendar_to_gcs.py)- Current: object name (not full URI) → Need: full
gs://URIs - Assignee: @Ajay9704 | PR: GoogleCalendarToGCSOperator: add unwrap_single flag and return full GCS URIs #61284
- Current: object name (not full URI) → Need: full
-
GoogleSheetsToGCSOperator (
transfers/sheets_to_gcs.py)- Current:
list[str](object names only) → Need: fullgs://URIs - Assignee: @Ajay9704 | PR: Return files destination uris in
GoogleDriverToGCSOperatorandSheetsToGCSOperator#61347
- Current:
-
SalesforceToGcsOperator (
transfers/salesforce_to_gcs.py)- Current:
str→ Need:list[str](wrap in list) - Assignee: @Prab-27 | PR: Return list of GCS URIs from SalesforceToGcsOperator #61659
- Current:
-
GoogleAdsToGcsOperator (
ads/transfers/ads_to_gcs.py)- Current:
None→ Need: destination URIs - Assignee: @Abhishekmishra2808 | PR: Return GCS URIs from GoogleAdsToGcsOperator for Issue #11323 #61334
- Current:
-
GoogleDisplayVideo360SDFtoGCSOperator (
marketing_platform/operators/display_video.py)- Current: non-list / non-URI → Need: full
gs://URIs - Assignee: @Ajay9704 | PR: GoogleDisplayVideo360SDFtoGCSOperator: add unwrap_single flag #61709
- Current: non-list / non-URI → Need: full
-
S3ToGcsOperator (
cloud/transfers/s3_to_gcs.py)- Need: adjustments to deferrable mode.
- Assignee: @AlexisBRENON | PR: In S3ToGcsOperator return the list of copied files even in deferrable mode #49768