Timeout BQ queries #902
Merged
Timeout BQ queries #902
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Thank you for your pull request! We could not find a changelog entry for this change. For details on how to document a change, see the dbt-bigquery contributing guide. |
|
It's on my wishlist for dbt features! |
@github-christophe-oudar don't know if we'll tackle that in this pr as I think this calls for a consistent approach across adapters but I definitely want to tackle this |
mikealfare
reviewed
Aug 29, 2023
mikealfare
reviewed
Aug 29, 2023
mikealfare
reviewed
Aug 29, 2023
mikealfare
reviewed
Aug 29, 2023
mikealfare
approved these changes
Sep 1, 2023
3 tasks
github-actions bot
pushed a commit
that referenced
this pull request
Sep 15, 2023
* use dynamic schema in test_grant_access_to.py * use dynamic schema in test_grant_access_to.py * experiment with query job cancel on timeout * modify unit tests * remove test grants change * starting functional test * update functional test and experiment with polling logic * experiment with async wait_for * modifying connections.py for asyncio logic * swap back to new_event_loop * close loop, now seeing asyncio timeoutError * improve order and update functional test * update unit test * add changie * add max_result back to result call in async path * rescope the dbt_profile_target to being a class fixture * raise DbtRuntimeError instead database * remove exception type check in job timeout --------- Co-authored-by: Matthew McKnight <matthew.mcknight@dbtlabs.com> Co-authored-by: Matthew McKnight <91097623+McKnight-42@users.noreply.github.com> (cherry picked from commit 2eb407d)
mikealfare
pushed a commit
that referenced
this pull request
Sep 15, 2023
* use dynamic schema in test_grant_access_to.py * experiment with query job cancel on timeout * modify unit tests * remove test grants change * update functional test and experiment with polling logic * experiment with async wait_for * modifying connections.py for asyncio logic * swap back to new_event_loop * close loop, now seeing asyncio timeoutError * improve order and update functional test * add max_result back to result call in async path * rescope the dbt_profile_target to being a class fixture * raise DbtRuntimeError instead database * remove exception type check in job timeout --------- Co-authored-by: Matthew McKnight <matthew.mcknight@dbtlabs.com> Co-authored-by: Matthew McKnight <91097623+McKnight-42@users.noreply.github.com> (cherry picked from commit 2eb407d) Co-authored-by: colin-rogers-dbt <111200756+colin-rogers-dbt@users.noreply.github.com>
2 tasks
4 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Time out and cancel a model execution if it exceeds the provided number of seconds.
Currently the bigquery python client does not provide us with a mechanism to specify a timeout on the database side. As a workaround until they do we can do the next best thing: timeout the execution, cancel the query and raise an exception.
Note we will only use this new logic if a user specifies the
job_execution_timeoutparameter in their profile. If it is not set we continue using the same behavior as today.resolves #231
docs dbt-labs/docs.getdbt.com/#
Problem
Solution
Checklist