Not dropping table for incremental full refresh with delta

[grindheim]

resolves 286

Description

Since the spark__create_table_as implementation already runs create or replace table if the file format is delta, this PR disables dropping the table first in this case. The advantage being that it will preserve grants on the table, in addition to it simply being unnecessary.

Checklist

• I have signed the CLA
• I have run this code in development and it appears to resolve the stated issue
• This PR includes tests, or tests are not required/relevant for this PR
• I have updated the CHANGELOG.md and added information about my change to the "dbt-spark next" section.

[cla-bot]

Thank you for your pull request and welcome to our community. We could not parse the GitHub identity of the following contributors: Sindre Grindheim.
This is most likely caused by a git client misconfiguration; please make sure to:

1. check if your git client is configured with an email to sign commits git config --list | grep email
2. If not, set it up using git config --global user.email email@example.com
3. Make sure that the git commit email is configured in your GitHub account settings, see https://github.com/settings/emails

[cla-bot]

Thank you for your pull request and welcome to our community. We could not parse the GitHub identity of the following contributors: Sindre Grindheim.
This is most likely caused by a git client misconfiguration; please make sure to:

1. check if your git client is configured with an email to sign commits git config --list | grep email
2. If not, set it up using git config --global user.email email@example.com
3. Make sure that the git commit email is configured in your GitHub account settings, see https://github.com/settings/emails

[cla-bot]

The cla-bot has been summoned, and re-checked this pull request!

[cla-bot]

Thank you for your pull request and welcome to our community. We could not parse the GitHub identity of the following contributors: Sindre Grindheim.
This is most likely caused by a git client misconfiguration; please make sure to:

1. check if your git client is configured with an email to sign commits git config --list | grep email
2. If not, set it up using git config --global user.email email@example.com
3. Make sure that the git commit email is configured in your GitHub account settings, see https://github.com/settings/emails

[cla-bot]

The cla-bot has been summoned, and re-checked this pull request!

[cla-bot]

The cla-bot has been summoned, and re-checked this pull request!

[McKnight-42]

Hi @grindheim sorry for the delay in response, gathering context now , and was wondering if you wouldn't mind updating your code to be in parity with main and re-push up so we can start to get some hanging tests running in the meantime.

[grindheim]

@McKnight-42 Updated the code. Let me know if there's anything else you'd like me to do.

[McKnight-42]

@grindheim for some reason the unit test didn't kick off and looks like we might need to now also update to main to take get access to our newly added black formatter and pre-commit functionality in-order for the code-quality check to pass. i'm hoping doing this will trigger both tests if not I'll keep looking into why the unit test didn't trigger. sorry for any inconvenience.

[jtcohen6]

Sorry for the delay @grindheim! Hoping we can get this merged soon, it'll definitely help out folks who are running into this issue. One small comment from me on the specific code change.

[jtcohen6]

Two quick thoughts here:

• I think this conditional logic can be simplified to avoid the nested if statements
• We need to check both that the table already exists in Delta format, and will be created in Delta format

So further up:

{% set is_delta = (file_format == 'delta' and existing_relation.is_delta) %}

And then:

  {% elif existing_relation.is_view or (full_refresh_mode and not is_delta) %}
    {% do adapter.drop_relation(existing_relation) %}

[grindheim]

Thanks for the thoughts and suggestions, and sorry for my delay. I've implemented and pushed your suggested changes.

[jtcohen6]

@grindheim Noticed one thing in the logic that doesn't look quite right. If you could take a look in the next few days, we can try to get this in for v1.2!

[jtcohen6]

Just going to push up an empty commit commit an update to the changelog, and to trigger CircleCI tests

[jtcohen6]

Thanks so much @grindheim! Glad to finally get this in :)