Skip to content

BigQueryToGCSOperator should return destination GCS URI(s) #59344

@AymaneElmahi

Description

@AymaneElmahi

Description

BigQueryToGCSOperator currently returns None from execute(), so no XCom is produced.
This makes it difficult for downstream tasks to know where BigQuery exported the data.

I propose that the operator returns the value of destination_cloud_storage_uris from execute() (and from execute_complete() when deferrable mode is used).

This change is:

  • backward-compatible (the operator previously returned None)
  • consistent with Airflow’s design where operators return their outputs via XCom
  • aligned with other transfer operators that expose output paths
  • helpful for downstream consumers of exported data

If maintainers agree, I am happy to submit a PR implementing this.

Use case/motivation

Many BigQuery export workflows need to consume the exported files in downstream Airflow tasks (e.g., loading into GCS, Firestore, BigQuery Load jobs, or custom processing).
However, BigQueryToGCSOperator currently returns None, so no XCom is available to downstream tasks.

Returning the destination URI list would:

  • make DAGs easier to author and understand
  • avoid redundant GCS listing operations
  • provide consistency with other operators that expose their outputs
  • improve interoperability between BigQuery → GCS → next-step workflows
  • enable dynamic downstream tasks that depend on the export result

This is especially useful for templated URIs or partitioned tables where downstream tasks must know the exact location of exported files.

Related issues

No response

Are you willing to submit a PR?

  • Yes I am willing to submit a PR!

Code of Conduct

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions