FR: GCP Component for Arbitrary BQ Op without export job

[dhodun]

FR for a more generic BQ Op component that doesn't try to run an export job every time. Use cases include intermediary steps in BQ. Also, AutoML Tables pipelines, where AutoML tables expects entire tables in BQ and does not support a select statement. It would be nice to provide AutoML tables with the temp table or create a separate table vs. pull data out to GCS.

Current component:
https://github.com/kubeflow/pipelines/blob/master/components/gcp/bigquery/query/README.md

[parthmishra]

I think once #2606 is fixed then you should be able to not specify a GCS path and thus no extract job is run.

[dhodun]

Makes sense, I did try that. The component doc should then be updated to indicate the component isn't solely for BQ query + extract job use case.

[parthmishra]

More generally speaking, once #2616 is released, customizing the query job should be doable and support the use cases you described. That being said, I do wonder if there's a balance between having a component being able to everything versus components that do a limited set of actions.

[dhodun]

In general I think it's more clear / readable to have a bigquery_query_op(...) and bigquery_query_to_gcs_op(...) or biqquery_query_to_export_op(...).

It is likely that BQ to GCS is the most common use case anyway, so there is an argument for leaving as is. There are more options that users might use in the future that could overload a single op, such as export to various formats like JSON.

[Ark-kun]

In general I think it's more clear / readable to have a bigquery_query_op(...) and bigquery_query_to_gcs_op(...) or biqquery_query_to_export_op(...).

+1
I favor making separate components when inputs/outputs are different.

Also biqquery_query_to_table.

[NikeNano]

I will look in to this.

/assign

[NikeNano]

@Ark-kun as I understand it we would like to have the following three components:

• bigquery_query_to_gcs_op -- The query is run and the data is extracted to a separate table which is then extracted to GCS
• bigquery_query_op -- A query is executed, the results are not written to a new table.
• biqquery_query_to_table -- The Query is run and the data is extracted to a separate table.

is this correct?

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale]

This issue has been automatically closed because it has not had recent activity. Please comment "/reopen" to reopen it.