Add retry mechanism and error handling to DBT Hook #56651
Conversation
* fix issue when single API error fail entire DbtCloudRunJobOperator task * fix formatting and refactor get_job_details function * add hook params to operator and trigger DBT classes * add docstring, fix function annotation * add test for retry in hooks and triggers * fix Hook class param, change method for sync requests * simplify tests, remove account param * change test names * change rerise implementation * add hook_params for others DBT operators * change timeout to optional * add test for timeout param, fix mock warning * reformat using prek * change import order, change log request after retry
* fix issue when single API error fail entire DbtCloudRunJobOperator task * fix formatting and refactor get_job_details function * add hook params to operator and trigger DBT classes * add docstring, fix function annotation * add test for retry in hooks and triggers * fix Hook class param, change method for sync requests * simplify tests, remove account param * change test names * change rerise implementation * add hook_params for others DBT operators * change timeout to optional * add test for timeout param, fix mock warning * reformat using prek * change import order, change log request after retry
* fix issue when single API error fail entire DbtCloudRunJobOperator task * fix formatting and refactor get_job_details function * add hook params to operator and trigger DBT classes * add docstring, fix function annotation * add test for retry in hooks and triggers * fix Hook class param, change method for sync requests * simplify tests, remove account param * change test names * change rerise implementation * add hook_params for others DBT operators * change timeout to optional * add test for timeout param, fix mock warning * reformat using prek * change import order, change log request after retry
* fix issue when single API error fail entire DbtCloudRunJobOperator task * fix formatting and refactor get_job_details function * add hook params to operator and trigger DBT classes * add docstring, fix function annotation * add test for retry in hooks and triggers * fix Hook class param, change method for sync requests * simplify tests, remove account param * change test names * change rerise implementation * add hook_params for others DBT operators * change timeout to optional * add test for timeout param, fix mock warning * reformat using prek * change import order, change log request after retry
|
Looking forward to getting this released. We are getting intermittent 502 errors when monitoring dbt jobs. Hopefull this would solve that. |
This was released a couple week ago as part of v4.4.4 of the provider. Are you seeing this issue on that version still? |
|
Ahh, sorry. I tried looking at the latest tag for the provider and then I did not see this change. But I checked again, and now all is clear. User error. :) I will try out updating our provider. We are using quite an outdated airflow ( |
|
It is not; only supported on Airflow 2.10+. In the provider's changelog I linked to, and on PyPI, you can find the minimum Airflow version requirements that is supported in each provider version. If it helps for context, we have documentation on why/when the minimum Airflow version is bumped for providers too. |
4 participants
Description
This PR enhances error handling and retry mechanisms in the DBT Hook and its integration with DBT operators.
Changes Made
DBT Hook
runmethod withrun_with_advanced_retryfor improved request reliability"reraise": Trueparameter to maintain backward compatibility by raising the original exception after retry threshold is exceeded, rather than a retry-specific errorDBT Operator Integration
hook_paramsparameterBackward Compatibility
The implementation maintains full backward compatibility with existing code through:
reraiseto maintain consistent exception handlingTesting
Added comprehensive test coverage for the new retry functionality.
closes: #51801
^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named
{pr_number}.significant.rstor{issue_number}.significant.rst, in airflow-core/newsfragments.