Fix persist_docs for columns #180
Conversation
|
I forgot that Databricks SQL endpoints can't create parquet tables. I'll update when I get a chance tomorrow. |
| @@ -81,10 +81,7 @@ | |||
| {%- set agate_table = load_agate_table() -%} | |||
| {%- do store_result('agate_table', response='OK', agate_table=agate_table) -%} | |||
|
|
|||
| {{ run_hooks(pre_hooks, inside_transaction=False) }} | |||
There was a problem hiding this comment.
Why do the run_hooks calls change? Is it related to the persist_docs change or is this related to something else?
There was a problem hiding this comment.
This isn't really related to the specific change in the PR at all, I just noticed because I was editing related lines. We clearly copied these lines from the default seed materialization a few years back. The default materialization, which was originally written for Redshift/Postgres, assumes a transactional database. Some hooks may want to run before the main transaction starts (begin), after it starts, before it ends (commit), or after it ends.
Spark doesn't have transactions, so there's no need to distinguish between inside_transaction = True|False. I figured I would simplify it while here, but I'd also be fine reverting those lines for PR cleanliness.
|
I'll cherry-pick to |
* Fix persist_docs for columns * Disable parquet model on endpoint * Rm parquet model, not worth the fuss * Update changelog [skip ci]
|
Hi, I was wondering why |
|
@binhnefits I think that was an oversight on my part! Would you be able to open a separate issue for that? Also, if you'd be interested in contributing the fix for it: I think it would just look like adding |
follow up to #84, #170
Description
Although #170 implemented
spark__alter_column_comment, we were missing the needed call to thepersist_docsmacro itself in needed materializations, since relation-level docs are handled within thecreate_x_asmacro DDL.So this PR:
spark__persist_docs, to add column descriptions onlypersist_docsto the table + seed materializations. It's already in the snapshot materialization, and views cannot persist columns descriptions.show table extended in ... like '*', so instead of pulling fromcatalog.json, we have to rundescribe extendedfor each table we want to check.Checklist
CHANGELOG.mdand added information about my change to the "dbt next" section.