Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Randomize temp table __dbt_new_data name to allow concurrent dbt-clickhouse jobs #150

Closed
winterrobert opened this issue May 11, 2023 · 1 comment
Labels
enhancement New feature or request

Comments

@winterrobert
Copy link

Is your feature request related to a problem? Please describe.
We are running into an issue with the dbt-clickhouse adapter when using delete+insert. When running multiple dbt-clickhouse jobs at the same time, we often get a 500 complaining that the temporary xxx_dbt_new_data already exists:

09:36:29    :HTTPDriver for clickhouse:8123 returned response code 500)
09:36:29     Code: 57. DB::Exception: Table default.tablenamet__dbt_new_data already exists. (TABLE_ALREADY_EXISTS) (version 23.3.2.37 (official build))

My guess is that it's related to this row here

{% macro clickhouse__incremental_delete_insert(existing_relation, unique_key, incremental_predicates) %}
    {% set new_data_relation = existing_relation.incorporate(path={"identifier": model['name'] + '__dbt_new_data'}) %}
    {{ drop_relation_if_exists(new_data_relation) }}
....
{% endmacro %}

Describe the solution you'd like
Would it be a good PR to add a random suffix to the table so that the temp tables created by dbt-clickhouse doesn't collide with each other? Could somebody at the team fix this (seems like a quick fix) or would you accept a PR?

Describe alternatives you've considered
Change our pipelines to make sure we always just run one dbt-clickhouse job at a time.

@winterrobert winterrobert added the enhancement New feature or request label May 11, 2023
@winterrobert winterrobert changed the title Randomize temp table __dbt_new_data name so allow concurrent dbt-clickhouse jobs Randomize temp table __dbt_new_data name to allow concurrent dbt-clickhouse jobs May 11, 2023
@genzgd
Copy link
Contributor

genzgd commented May 11, 2023

Included in the 1.4.1 patch release

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants