feat: Set unique table suffix to allow parallel incremental executions

[pierrebzl]

Description

For some specific cases (eg. backfill very large amount of data), we need to execute parallel multiple dbt build of specific incremental model in which we pass the date (or batch number) as var argument.
For example, we have a model we run every hour using Airflow for which we pass the a date relative to the Airflow scheduler.

If we want to process by batch of N hours in parallel using Airflow concurrency, we need the tmp table create by each of the dbt run to be unique. Else, you are going to end up with N insert attempting to run with the same __dbt_tmp name, creating conflict and ultimately creating failure.

Similar issue open here: Tomme/dbt-athena#62

Models used to test - Optional

Checklist

• You followed contributing section
• You kept your Pull Request small and focused on a single feature or bug fix.
• You added unit testing when necessary
• You added functional testing when necessary

Second (incremental) run model output:

-- /* {"app": "dbt", "dbt_version": "1.8.0", "profile_name": "test", "target_name": "default", "node_id": "model.test.unique_tmp_table_suffix"} */

create table "awsdatacatalog"."test17157222480613300988_test_unique_tmp_table_suffix"."unique_tmp_table_suffix__dbt_tmp_375973fa-ecdd-44dc-9a63-788abaa41832"
    with (
      table_type='hive',
    is_external=true,external_location='s3://XXX/test17157222480613300988_test_unique_tmp_table_suffix/unique_tmp_table_suffix__dbt_tmp_375973fa-ecdd-44dc-9a63-788abaa41832/1058bc6a-9eb1-4b0b-b23a-cce26c28b669',
      format='parquet'
    )
    as
      
select
    random() as rnd,
    cast(from_iso8601_date('2024-01-02') as date) as date_column

[nicor88]

The general idea of using unique tmp table location is good.
Few notes and concerns:

• parallel transactions on the same iceberg table might fail, e.g if you run in parallel backfilling by a date, with a delete/insert operations leads to a failure, due to glue optimistic logic. With some retries might work
• running parallel insert+overwrites with an hive/parquet table should just work fine, because each dbt run will operate on a specific "partition" with its own unique prefix, and tmp table

@pierrebzl I left few comments, please have a look.

[pierrebzl]

Hi @nicor88,
Thanks a lot for your feedback, definitely makes a lot of sense. I've attempted to implement the logic you mentioned.
Couple of points I don't really know how to address:

1. Any idea how in the functional test, assert the table name generate in the second run / CTAS query? Ideally, I would like to check a uuid is present and not the default __dbt_tmp value.
2. Is the logic I put in the incremental.sql at the right place and looks good with you in terms of when this unique id should apply? In my use case, I only need it on insert_overwrite + hive.
   Thanks again for your time!

[nicor88]

@Jrmyy @svdimchenko could you assist @pierrebzl?
I will be away from keyboard till next week. Thanks

[nicor88]

@pierrebzl could you lint your changes? Regarding functional testing, what you proposed seems fine. It's hard to intercept the tmp table name in the test, I'm fine with what you proposed

[pierrebzl]

@pierrebzl could you lint your changes? Regarding functional testing, what you proposed seems fine. It's hard to intercept the tmp table name in the test, I'm fine with what you proposed

Yes sorry for this. Should be done:

(venv) dbt-athena $ pre-commit run mypy --show-diff-on-failure --color=always --all-files
mypy.....................................................................Passed

[pierrebzl]

Hi @nicor88, let me know if anything missing to go ahead with this change. Thanks!

[Jrmyy]

Overall looks good ✅

[nicor88]

@pierrebzl there are some conflicts in the Readme, due to the latest merge, could you please resolve that?

[nicor88]

@pierrebzl could you lint again? please, I left a tiny comment, looks good.

[pierrebzl]

@pierrebzl could you lint again? please, I left a tiny comment, looks good.

Sorry for that.
Done, I must be something wrong but my local pre-commit check always fails on flake8 as it seems to check other files than changed ones.

[pierrebzl]

I had to change the regex expression format: 4a52129#diff-58cf52251bceac0bf774fa9511051801f5c8d6b34e5af08e3af56dbe271c6df8R65-R68 as it got broke by the auto lintting. Should pass tests now.

[nicor88]

@pierrebzl thanks for fixing, I was wondering why the CI was failing, but it was a genuine failure, and I just re-trigger it.

[pierrebzl]

@pierrebzl thanks for fixing, I was wondering why the CI was failing, but it was a genuine failure, and I just re-trigger it.

Yes, I think we are good now 🥳

[nicor88]

@pierrebzl great job 💯

[pierrebzl]

@pierrebzl great job 💯

Thanks for your support!