Add column comments for DLT tables using dbt schema #794
Conversation
Amin-Siddique
requested review from
andrefurlan-db,
benc-db and
rcypher-databricks
as code owners
| {%- if model.columns -%} | ||
| {%- set column_definitions = [] -%} | ||
| {%- for column_name, column in model.columns.items() -%} | ||
| {%- set data_type = column.data_type or 'STRING' -%} |
There was a problem hiding this comment.
Does this actually work? In general we don't know the column data type, and I'm concerned that specifying as string will break when the columns are string type.
There was a problem hiding this comment.
Yes you're right, it breaks! Unfortunately, for this to work it we need to explicitly define data_types in the schema file of the respective model.
There was a problem hiding this comment.
@benc-db I have added a workaround where we initially create a temp view to infer the schema and utilize that to get the data type if undefined. Please let me know your thoughts! 😄 I have tested it in my local setup (1. with schema file and defined data_types 2. with schema file without defining any data types 3. without schema file) and works for both ST/MV
There was a problem hiding this comment.
This is an interesting approach. I want to talk to dbt Labs about this to get their perspective, and I definitely will want to put behind a behavior flag (new tech recently introduced into dbt) because this will add some latency, but might be worth the trade off.
|
Please add or edit the functional tests to demonstrate that this functionality works. I'm fairly suspicious of setting STRING as type when the column type is unknown. |
Resolves #793
Description
Add support for column comments in Delta Live Tables (ST/MV) from dbt model schema descriptions
Checklist
CHANGELOG.mdand added information about my change to the "dbt-databricks next" section.