-
Notifications
You must be signed in to change notification settings - Fork 54
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add retry mechanism for the given error #322
Comments
In dbt we utilise something called artifacts: We use this to store test results into a table called dbt_results , on our project level on-run-end. Which outputs test results to a single table with all test results. We don’t run all jobs in one single DBT job but they are spread out. This leads to multiple DBT jobs finishing, trying to write on on-run-end:
- "{{ store_dbt_results(results) }}" The delta write lock issue then kicks in here since multiple DBT jobs, within the same single cluster, try to write their output to the results table from the actual dbt run and dbt test. So even if we spread out the run of the jobs, collision can happen. So this retry would be good when you have a few collisions, or rare ones, we can try to avoid it as much as possible due to how we run and schedule the DBT jobs, but we still need to be able to avoid collision, when having multiple run, test output tables in this case for each run. |
Describe the feature
Add retry mechanism for the given error, like for example
DELTA_LAKE_BAD_WRITE
.That feature should be configurable, meaning that the user can specify one or more errors on which the query from the model will be retried. It should also include max number of retries and wait/sleep before the next attempt.
Describe alternatives you've considered
No response
Who will benefit?
No response
Are you willing to submit PR?
The text was updated successfully, but these errors were encountered: