Set default file format to parquet #422
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
jtcohen6
requested review from
Fleid,
lostmygithubaccount,
a team and
VersusFacit
|
Thank you for your pull request! We could not find a changelog entry for this change. For details on how to document a change, see the dbt-spark contributing guide. |
jtcohen6
force-pushed
the
fix/default-file-format-parquet
branch
from
e860c50 to
62e3c61
Compare
|
Running into more issues on ODBC cluster + endpoint than I'm able to debug right now. I'm going to close this PR for now; someone else could pick up this work and try to carry it over the finish line. |
3 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
resolves #363
Description
We already set this default here, but it's clearly not flowing through properly, because
AdapterConfigdoesn't really seem to be working indbt-core(dbt-labs/dbt-core#5236):dbt-spark/dbt/adapters/spark/impl.py
Line 38 in 8744cf1
Let's set the default explicitly in the macro, so that if the user hasn't provided a
file_format,dbt-sparkuses Parquet. This prevents weird issues with query rewrite in Databricks SQL warehouses, too (#236).Update: If the user is connecting via
ODBC(= Databricks), let's usedeltaas the default file format instead. This helps out Databricks SQL warehouse connections, which can only write Delta-formatted tables.Checklist
CHANGELOG.mdand added information about my change to the "dbt-spark next" section.