Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Creating materialized views do not set table properties #577

Closed
thijs-nijhuis opened this issue Feb 6, 2024 · 3 comments
Closed

Creating materialized views do not set table properties #577

thijs-nijhuis opened this issue Feb 6, 2024 · 3 comments
Labels
bug Something isn't working

Comments

@thijs-nijhuis
Copy link
Contributor

Describe the bug

To make use to Delta's ChangeDataFeed I set 'delta.enableChangeDataFeed': true as one of the table properties. I set is at the project level for all models like this:

dbt_model_name:
  +tblproperties:
    'delta.enableChangeDataFeed': true

All models materialized as table will get this property upon creation. Models materialized as 'materialized_view' however do not. I also tried setting it in the models config directly like this:

{{
  config(
    materialized = 'materialized_view',
    tblproperties={'delta.enableChangeDataFeed': 'true'},
    )
}}

Unfortunately, that didn't work either. I think it is important to be able to set this property because it is a requirement to be able to to do an incremental refresh on a materialized view. So if I would want to create an incremental flow of multiple models using MV's this property needs to be set otherwise it will always execute a full compute.

Steps To Reproduce

Set the table property to a model as described above and run the project. MV should not be there as this property can only be set upon creating the mv.

Expected behavior

I expect all property I specify through the tblproperties config that are supported by that type of object to be set.

Screenshots and log output

view tables tab from catalog when created via dbt.
image

System information

The output of dbt --version:

Core:
  - installed: 1.7.6
  - latest:    1.7.7 - Update available!

  Your version of dbt-core is out of date!
  You can find instructions for upgrading here:
  https://docs.getdbt.com/docs/installation

Plugins:
  - databricks: 1.7.4 - Ahead of latest version!
  - spark:      1.7.1 - Up to date!

The operating system you're using:
windows 11

The output of python --version:
Python 3.11.4

Additional context

I can create the view manually with that property and verified that the next mv (the one that selects from it) won't do a full refresh when this property is set.

@thijs-nijhuis thijs-nijhuis added the bug Something isn't working label Feb 6, 2024
@benc-db
Copy link
Collaborator

benc-db commented Feb 6, 2024

This is coming soon. I plan to have a beta out this week with expanded MV capabilities.

@benc-db
Copy link
Collaborator

benc-db commented Feb 26, 2024

Please give the beta a shot (1.8.0b1) to see if it fixes your issue. #595

@thijs-nijhuis
Copy link
Contributor Author

@benc-db , with 1.8.0.b1 the properties ARE created. Thanks for picking this one up.
For everybody who stumbles upon this later, it seems you don't need to set the 'delta.enableChangeDataFeed' on an MV (anymore). In the simple demo setup I did, an MV reading from another MV did a full or incremental refresh regardless of this property being set or not.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants