[ADAP-934] [Bug] Dataproc serverless option submission in dbt fails due to timeout issue

[alimasri]

Is this a new bug in dbt-bigquery?

• I believe this is a new bug in dbt-bigquery
• I have searched the existing issues, and I could not find an existing issue for this bug

Current Behavior

When using the dataproc serverless option submission in dbt, my dataproc job takes around 30 minutes to complete. However, my dbt runs keep failing with the error message "Operation did not complete within the designated timeout of 900 seconds." The dbt run stops but the dataproc serverless jobs continues running and finishes with no issues.

I have gone through the code: https://github.com/dbt-labs/dbt-bigquery/blob/main/dbt/adapters/bigquery/python_submissions.py#L51 and noticed that there is a possibility to change the timeout value.

I have tried defining the timeout value in my profile, yaml config, and in the model dbt config object in the python model itself, but with no success. I have also changed my job_execution_timeout_seconds value to 6000, but that did not work either. I am looking for a way to override the default timeout value in this scenario.

Expected Behavior

The dbt run should not fail due to a timeout issue and should complete successfully as the dataproc serverless job does.

Steps To Reproduce

1. Use the dataproc serverless option submission in dbt.
2. Set your timeout to anything greater than 30 minutes
3. Run a dataproc job that takes around 30 minutes to complete.
4. Observe that the dbt run fails with a timeout error, while the dataproc serverless job continues running and finishes successfully.

Relevant log output

Operation did not complete within the designated timeout of 900 seconds.

Environment

- OS: Ubuntu 20.04
- Python: 3.10.11
- dbt-core: 1.5.6
- dbt-bigquery: 1.5.6

Additional Context

No response

[dbeatty10]

It sounds like you were able to accomplish your goal after all @alimasri ?

[alimasri]

No I found a similar issue #577. It was closed by #766, but still I do not know where should I reference the timeout variable in my dbt code.

[dbeatty10]

The bugfix in #766 wasn't released until dbt-bigquery 1.6.0 - July 31, 2023.

Could you try something like this with dbt-bigquery 1.6+ and see if it works for you or not?

my-profile:
  target: dev
  outputs:
    dev:
      type: bigquery
      method: oauth
      project: abc-123
      dataset: my_dataset
      job_execution_timeout_seconds: 1800  # 30 minutes

[alimasri]

Thank you @dbeatty10 for the information. I upgraded to 1.6.0 and the timeout parameter worked as expected.