Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

replace partitionOverwriteMode inside merge strategy #117

Merged
merged 5 commits into from
Nov 5, 2020
+6 −4
Merged

replace partitionOverwriteMode inside merge strategy #117

Changes from 3 commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
087a158
replace partitionOverwriteMode inside merge strategy
charlottevdscheun Oct 30, 2020
30b2222
dynamic overwrite when partition || unique key
charlottevdscheun Nov 2, 2020
f240857
Update dbt/include/spark/macros/materializations/incremental.sql
charlottevdscheun Nov 2, 2020
f6d54d7
add documentation about insert_overwrite when no partition
charlottevdscheun Nov 2, 2020
a34527a
Merge branch 'fix/incremental_overwrite' of https://github.com/charlo…
charlottevdscheun Nov 2, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 6 additions & 3 deletions dbt/include/spark/macros/materializations/incremental.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -62,6 +62,7 @@


{% macro spark__get_merge_sql(target, source, unique_key, dest_columns, predicates=none) %}

{# ignore dest_columns - we will just use `*` #}
merge into {{ target }} as DBT_INTERNAL_DEST
using {{ source.include(schema=false) }} as DBT_INTERNAL_SOURCE
Expand Down Expand Up @@ -100,9 +101,11 @@
{% do dbt_spark_validate_merge(file_format) %}
{% endif %}

{% call statement() %}
set spark.sql.sources.partitionOverwriteMode = DYNAMIC
{% endcall %}
{% if config.get('partition_by') %}
{% call statement() %}
set spark.sql.sources.partitionOverwriteMode = DYNAMIC
{% endcall %}
{% endif %}

{% call statement() %}
set spark.sql.hive.convertMetastoreParquet = false
Expand Down