You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
To make use to Delta's ChangeDataFeed I set 'delta.enableChangeDataFeed': true as one of the table properties. I set is at the project level for all models like this:
All models materialized as table will get this property upon creation. Models materialized as 'materialized_view' however do not. I also tried setting it in the models config directly like this:
Unfortunately, that didn't work either. I think it is important to be able to set this property because it is a requirement to be able to to do an incremental refresh on a materialized view. So if I would want to create an incremental flow of multiple models using MV's this property needs to be set otherwise it will always execute a full compute.
Steps To Reproduce
Set the table property to a model as described above and run the project. MV should not be there as this property can only be set upon creating the mv.
Expected behavior
I expect all property I specify through the tblproperties config that are supported by that type of object to be set.
Screenshots and log output
view tables tab from catalog when created via dbt.
System information
The output of dbt --version:
Core:
- installed: 1.7.6
- latest: 1.7.7 - Update available!
Your version of dbt-core is out of date!
You can find instructions for upgrading here:
https://docs.getdbt.com/docs/installation
Plugins:
- databricks: 1.7.4 - Ahead of latest version!
- spark: 1.7.1 - Up to date!
The operating system you're using:
windows 11
The output of python --version:
Python 3.11.4
Additional context
I can create the view manually with that property and verified that the next mv (the one that selects from it) won't do a full refresh when this property is set.
The text was updated successfully, but these errors were encountered:
@benc-db , with 1.8.0.b1 the properties ARE created. Thanks for picking this one up.
For everybody who stumbles upon this later, it seems you don't need to set the 'delta.enableChangeDataFeed' on an MV (anymore). In the simple demo setup I did, an MV reading from another MV did a full or incremental refresh regardless of this property being set or not.
Describe the bug
To make use to Delta's ChangeDataFeed I set
'delta.enableChangeDataFeed': true
as one of the table properties. I set is at the project level for all models like this:All models materialized as table will get this property upon creation. Models materialized as 'materialized_view' however do not. I also tried setting it in the models config directly like this:
Unfortunately, that didn't work either. I think it is important to be able to set this property because it is a requirement to be able to to do an incremental refresh on a materialized view. So if I would want to create an incremental flow of multiple models using MV's this property needs to be set otherwise it will always execute a full compute.
Steps To Reproduce
Set the table property to a model as described above and run the project. MV should not be there as this property can only be set upon creating the mv.
Expected behavior
I expect all property I specify through the tblproperties config that are supported by that type of object to be set.
Screenshots and log output
view tables tab from catalog when created via dbt.
System information
The output of
dbt --version
:The operating system you're using:
windows 11
The output of
python --version
:Python 3.11.4
Additional context
I can create the view manually with that property and verified that the next mv (the one that selects from it) won't do a full refresh when this property is set.
The text was updated successfully, but these errors were encountered: