Replies: 9 comments 13 replies
-
|
Example diffs are empty |
Beta Was this translation helpful? Give feedback.
-
|
sorry @damian3031 I've just updated it -- does it work now? |
Beta Was this translation helpful? Give feedback.
-
|
yes it works now, thanks |
Beta Was this translation helpful? Give feedback.
-
|
Hi @dataders, what about Materialized View, Materialized Tests in 1.5.0? Would |
Beta Was this translation helpful? Give feedback.
-
|
My bad, these 2 items were bumped to 1.6! Nothing to do for 1.5 :) |
Beta Was this translation helpful? Give feedback.
-
Beta Was this translation helpful? Give feedback.
-
|
Thank you @dataders I have started to work on v.1.5.0 for dbt-oracle and have a couple of questions.
create table dbt_test.my_model
(id int)
as
select 1 as id from dualraises Following works correctly create table dbt_test.my_model
(id primary key not null)
as
select 1 as id from dual |
Beta Was this translation helpful? Give feedback.
-
|
hey @aosingh -- great question. I recommend you start by reading https://docs.getdbt.com/reference/resource-properties/constraints#platform-specific-support. the below should also help (thanks @dbeatty10!)
The model won't even build if it violates this constraint
The platform supports specifying this type of constraint, but a model can still build even if it violates it. It's basically a "fake" constraint only for metadata purposes. This is common for modern cloud data warehouses and less common for legacy databases.
You can't specify this type of constraint for this platform more background reading |
Beta Was this translation helpful? Give feedback.
-
|
Got it @dataders. Thank you for the simple explanation @dbeatty10 . Kudos to the dbt Labs team for implementing model contracts in an elegant manner. I needed to override a few macros to generate the "pre-flight" SQL and the adapter method |
Beta Was this translation helpful? Give feedback.
-
|
Thank you @dataders. I would like to use dbt-core v1.5.0 with dbt-databricks adapter. Any ETA by when that will be available? |
Beta Was this translation helpful? Give feedback.
-
|
any day now! I failed to communicate to the team maintaining dbt-databricks about migration off of the legacy test suite, so the extra work delayed the release. I'd follow https://github.com/databricks/dbt-databricks/ for more information |
Beta Was this translation helpful? Give feedback.
-
|
Thank you @dataders. I will follow the progress. |
Beta Was this translation helpful? Give feedback.
-
|
While working on the upgrade, I noticed that this snippet enforces that contracts can only be applied to Something else that is confusing is that the returned error uses constraints, rather than contracts: IIUC, contracts are supported for |
Beta Was this translation helpful? Give feedback.
-
|
TL;DR: what you ask cannot be done without patching dbt-core. here's my paraphrase of @MichelleArk's response from our internal slack thread
Of the three custom materializations you mention, So your remaining ask is for contracts on |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for clarifying, @dataders! Being able to support any custom materialization would be ideal, but I understand that not many adapters have this requirement (the Athena folks seem to have found a workaround, for example). It's fine to wait until materialized views are in
We'd like to avoid coupling dbt with other tools, since not all |
Beta Was this translation helpful? Give feedback.
-
|
Hello, I am exploring dbt-databricks==1.5.1 (it uses dbt-core 1.5.0). I believe contracts work for materialized tables. Is there something I am missing My model files are as follows
test_version_001.sql test_version_002.sql |
Beta Was this translation helpful? Give feedback.
-
|
@nikitasinghvi this is better suited as an issue on dbt-databricks. Do you mind opening an issue over there? If you look at dbt-databricks v1.5.0 release notes, you'll see that your bug is actually intended behavior. That said, databricks/dbt-databricks#339 tells me that this support has been merged to the main branch and will likely be soon released as If you pip install the package directly from GitHub, you might be able to experiment with this new feature pip install git+https://github.com/databrickslabs/dbt-databricks.git |
Beta Was this translation helpful? Give feedback.
-
|
Thank you for sharing all the links @dataders, I really appreciate it. I just saw merged but didn't realise it is not for 1.5.1 |
Beta Was this translation helpful? Give feedback.
-
|
@dataders Could you clarify the logging TBC? |
Beta Was this translation helpful? Give feedback.
-
|
@dataders I know we're a bit late to the party, but ClickHouse has case sensitive table and database (schema) names, so the current adapter |
Beta Was this translation helpful? Give feedback.
-
|
@genzgd Could you share more what you mean by "ClickHouse has case sensitive table and database (schema) names"? Is it that both of these would create a table with a case-sensitive name even though only the latter has a quoted identifier? Or something else? CREATE TABLE My_Table ...
CREATE TABLE "My_Table" ...We'd be willing to consider adapter-level interfaces to accommodate tricky edge cases (like the case-sensitivity of table names), but we'd need to better understand what the key database behaviors are. |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for the response -- in ClickHouse quoted identifiers change nothing with respect to table names; both of those attempt to create the same table with a mixed case name. (Quoted identifiers are only useful for allowing some special characters). And "My_Table" is a different table than "my_table" or "MY_TABLE", and three tables with all three of those names could exist on the same server (at least on Linux). Similarly "DB1" and "db1" databases (ClickHouse has no "schema" level) could both exist on the same server, and quoting in the CREATE statement again has no relevant effect. We are also having issues with the cache related to the fact that the cache expects both database and schema to be valid strings, but again ClickHouse doesn't actually have schemas, so we reuse the database for that and the relation database property is |
Beta Was this translation helpful? Give feedback.
-
Overview
This discussion is for communicating to adapter maintainers the scope of work needed to make use of the changes in 1.5.0. If you have questions and concerns, please ask them here for posterity.
For the guide to upgrading an adapter to be compatible with dbt-core=1.4.0, please see #6624.
The latest release for 1.5.0,
dbt-core==1.5.0rc1, (PyPI | Github).The following releases are forthcoming with the projected dates:
April 13, 2023: release cut (1.5.0rc1)1.5.0)Changes
Example diffs from dbt Labs-owned adapters
Below gives the changes as of the latest beta release (
1.5.0rc1)dbt-labs/dbt-redshift@v1.4.0...1.5.latest
dbt-labs/dbt-bigquery@v1.4.0...1.5.latest
dbt-labs/dbt-snowflake@v1.4.0...1.5.latest
dbt-labs/dbt-spark@v1.4.0...1.5.latest
Behind the scenes improvements
Support for Py 3.11
Do this if you haven't already! Python 3.11 is the default latest version that is installed without specifying a verson. For example
brew install python.Adapter zone tests
The work is finally finished converting our test suite. This means that, going forward, any adapter test that is useful for more than one adapter will be available to community and partner adapter maintainers. There are many new tests You can find them all in the
tests/adapter/dbt/tests/adapter/directory of the dbt-core repo(link). Quick callout to some new categories of tests:
Logging restructuring
TBC (to-be-communicated)
Upgrade test suite version
Be sure to upgrade the version of
dbt-tests-adapterNew features
as an adapter maintainer, there's three options for new features in dbt Cloud:
Materialized View, Materialized Tests
Bumped to 1.6! Nothing to do for 1.5 :)
Model Contracts
How to support:
data_type_code_to_nameto convert Python connector return types to strings (example: dbt-snowflake PR)create_table_as, add calls to two macros that assert column spec equivalence (get_assert_columns_equivalent) and template out DDL containing column names, types, and constraints (get_table_columns_and_constraints) (example: dbt-snowflake)create_view_as, add a call to just the macro asserting column spec equivalence -get_assert_columns_equivalent(example: dbt-snowflake)constraintsdata structure (docs) by:Adapterclass calledCONSTRAINT_SUPPORT(Snowflake example) that maps constraint types to their support/enforcement values,andAdaptermethodsrender_column_constraintandrender_model_constraintas necessary and to render out the appropriate text for each constraint type.NOT NULL(docs), sodbt-trinowould implement this method and data structure with onlynot_nullconstraints supported.How to stub elegantly:
In both
create_table_asandcreate_view_asmacros, raise an explicit warning if a model is configured with an enforced contract. For example,{% macro <adapter>__create_table_as(...) %} {% set contract_config = config.get('contract') %} {% if contract_config.enforced %} {{exceptions.warn("Model contracts cannot be enforced by <adapter>!")}} {% endif %} {% endmacro %}What if you do nothing:
Users will see new documentation about model contracts and expect to be able to use it!
If your adapter uses the default materializations for table, view, and incremental, and the user configures a model to have an enforced contract, they will see an explicit error:
If, however, your adapter reimplements those materializations, their configuration may do nothing—which is worse! If you have reimplemented those materializations, and your adapter cannot implement support for contracts, we recommend you raise an explicit warning if they configure a model with an enforced contract.
Beta Was this translation helpful? Give feedback.
All reactions