use insert_job to execute gcs_to_bq_operator

[TY-chang]

Description

• I met LoadJob bug in BigQueryInsertJobOperator and tried to use GCSToBigQueryOperator instead when update from Airflow 2.2.5 to 2.3.2.

Use case/motivation

• most BigQuery-related operator recommend that we should use insert_job(), but gcs_to_bq_operator still use run_load ().
• GCSToBigQueryOperator still uses lots of parameter to build config. Not like BigQueryInsertJobOperator which uses a clean configuration. The configuration is much straightforward and easy to follow.
• I would like to know whether to keep GCSToBigQueryOperator and replace run_load with insert_job, Or have BigQueryInsertJobOperator. If we could replace run_load with insert_job in GCSToBigQueryOperator , all BigQuery-related operator could be more consistent.

Related issues

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template!

[TY-chang]

Here is error I met when we updated to version 2.3.2:

Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.9/site-packages/airflow/providers/google/cloud/operators/bigquery.py", line 2170, in execute
    table = job.to_api_repr()["configuration"]["query"]["destinationTable"]
KeyError: 'query'

[eladkal]

duplicate: #23826
Please update your google provider.