`dbt clone` #7881

aranke · 2023-06-15T04:53:12Z

resolves #7256

Description

Checklist

I have read the contributing guide and understand what's expected of me
I have signed the CLA
I have run this code in development and it appears to resolve the stated issue
This PR includes tests, or tests are not required/relevant for this PR
I have opened an issue to add/update docs, or docs changes are not required/relevant for this PR
I have run changie new to create a changelog entry

github-actions · 2023-06-15T04:53:30Z

Thank you for your pull request! We could not find a changelog entry for this change. For details on how to document a change, see the contributing guide.

core/dbt/include/global_project/macros/adapters/clone.sql

McKnight-42 · 2023-06-16T14:28:43Z

if we are technically creating a materialization do we need to add a new is_type check inBaseRelation?

McKnight-42 · 2023-06-16T15:42:35Z

tests/functional/defer_state/test_defer_state.py

+{%- endmacro %}
+"""
+
+


To circle back to talk of test placement does creating two base classes ex. BaseClonePossible and BaseCloneNotPossible and define like the tests below to look for the correct types view or table so users can pull in one or both options to test? or potentially move both those to their own file to be the adapter zone tests? as I don't believe defer_state is pulled into all adapters

What would you recommend?
I don't know this code very well, so happy to defer to the recommendation of the Adapters team.

will make some mock stuff for snowflake pr we can look at

I would lean towards having some sort of tests in adapter zone and have all adapters use it unless we found some very specific reasons not to do that

@ChenyuLInx currently working towards defining some adapter zone tests here dbt-snowflake since it actually uses dbt_clone will then move them to dbt-core wanting to find a way to properly tests exception throwing if users try to hard throw ability to use dbt_clone on in project by creating a custom macro.

aranke · 2023-06-16T15:54:17Z

if we are technically creating a materialization do we need to add a new is_type check inBaseRelation?

Discussed offline, not required.

jtcohen6

It's happening !!

jtcohen6 · 2023-06-20T13:18:19Z

core/dbt/include/global_project/macros/materializations/models/clone/clone.sql

+      -- TODO: this should probably be illegal
+      -- I'm just doing it out of convenience to reuse the 'view' materialization logic
+      {%- do context.update({
+          'sql': get_clone_target(state_relation),
+          'compiled_code': get_clone_target(state_relation)
+      }) -%}
+
+      -- reuse the view materialization
+      -- TODO: support actual dispatch for materialization macros
+      {% set search_name = "materialization_view_" ~ adapter.type() %}
+      {% if not search_name in context %}
+          {% set search_name = "materialization_view_default" %}
+      {% endif %}
+      {% set materialization_macro = context[search_name] %}
+      {% set relations = materialization_macro() %}
+      {{ return(relations) }}


This is definitely the bit I felt worst about in #7258.

I opened an issue to track real dispatch support for materialization macros: #7799. Until we do that, this will work much of the time, but it won't support adapters that inherit from other adapters. E.g. dbt-postgres → dbt-redshift, ergo Redshift is allowed to use postgres__ implementations.

As far as hard-overwriting the context values for sql + compiled_code: Gross, right? The other potential approach I can think of is, making materialization macros accept kwargs, so that sql/compiled_code can be passed in explicitly rather than set + pulled out of the ~~aether~~ context.

ergo Redshift is allowed to use postgres__ implementations

TIL, wow

As far as hard-overwriting the context values for sql + compiled_code: Gross, right?

Yes, but I've seen worse

making materialization macros accept kwargs, so that sql/compiled_code can be passed in explicitly

This makes sense, we can pass in a dict for now before migrating to a typed object (similar to dbtRunner)

To break down the questions here a bit:

Do we have to reuse the view materialization? Would it be better if we just write the logic here given clone potentially have a more restrictive assumption than the actual view materialization, for example: always create a view that points to the other table no matter what's going on? Looks like there's not as much going on in view as other materialization, sometimes it's better to repeat simple stuff again instead of reuse

If option 1 is not viable, would this implementation bite us somewhere else? what happens if we try to run this code with clone on redshift today?

I'll defer to @dbt-labs/core-adapters on 1, I'm fine with either approach.

@McKnight-42: question for you when you get back.

For 2, I don't know the answer, but can sync with @McKnight-42 and reply here.

Good breakdown of the questions @ChenyuLInx!

Thinking through the logic we'd want from the view materialization:

Handling the cases where the object already exists as a table, and needs to be dropped first

Knowing whether this adapter wants to create or replace view versus create temp + alter/rename/swap

Applying grants, persisting descriptions as comments - I think we still might actually want these things!

So - I would argue that's the right way to go. I'm definitely interested in a world where materializations are more composable than they are currently, so that more-complex build strategies ("clone an object from one env into another") can call into simpler ones ("make a view").

Macros are already pretty composable. Materializations are just macros, but they pull all their inputs from this node's g-d context, rather than receiving them as explicit arguments.

Answers to @ChenyuLInx questions from offline chat with @McKnight-42:

Do we have to reuse the view materialization?

I believe so, since different adapters implement views differently.
I previously believed that all adapters share a single view materialization, but that doesn't seem to be the case.

If option 1 is not viable, would this implementation bite us somewhere else?

I don't know.

What happens if we try to run this code with clone on Redshift today?

In theory, it should do the same thing as Postgres (create views) since neither support zero-copy cloning.
I'll have to verify what happens in practice.

core/dbt/task/clone.py

core/dbt/context/providers.py

ChenyuLInx

Thanks for adding this patch!! Really glad it's happening! As this work is the foundation piece for adapters, I am hoping that we can think through what's strictly needed at this stage to make this function easy to maintain in the long run and make sure all of those are included in this patch.

core/dbt/cli/main.py

ChenyuLInx · 2023-06-20T16:27:18Z

core/dbt/include/global_project/macros/materializations/models/clone/clone.sql

+      -- TODO: this should probably be illegal
+      -- I'm just doing it out of convenience to reuse the 'view' materialization logic
+      {%- do context.update({
+          'sql': get_clone_target(state_relation),
+          'compiled_code': get_clone_target(state_relation)
+      }) -%}
+
+      -- reuse the view materialization
+      -- TODO: support actual dispatch for materialization macros
+      {% set search_name = "materialization_view_" ~ adapter.type() %}
+      {% if not search_name in context %}
+          {% set search_name = "materialization_view_default" %}
+      {% endif %}
+      {% set materialization_macro = context[search_name] %}
+      {% set relations = materialization_macro() %}
+      {{ return(relations) }}


To break down the questions here a bit:

Do we have to reuse the view materialization? Would it be better if we just write the logic here given clone potentially have a more restrictive assumption than the actual view materialization, for example: always create a view that points to the other table no matter what's going on? Looks like there's not as much going on in view as other materialization, sometimes it's better to repeat simple stuff again instead of reuse

If option 1 is not viable, would this implementation bite us somewhere else? what happens if we try to run this code with clone on redshift today?

ChenyuLInx · 2023-06-20T16:31:49Z

tests/functional/defer_state/test_defer_state.py

+{%- endmacro %}
+"""
+
+


I would lean towards having some sort of tests in adapter zone and have all adapters use it unless we found some very specific reasons not to do that

core/dbt/task/clone.py

core/dbt/context/providers.py

tests/unit/test_context.py

jtcohen6

IMO A prerequisite to merging this is having a working implementation in one of the adapters that actually supports create table ... clone (Snowflake, BigQuery, or Databricks), to make sure we have the right constructs in place.

core/dbt/task/clone.py

core/dbt/include/global_project/macros/materializations/models/clone/clone.sql

core/dbt/task/clone.py

ChenyuLInx · 2023-06-26T16:11:12Z

Great!!!!

we need to modify the clone target directory to be different from the previous directory.

My understanding is that having a separate state directory is all you need? Do we have to have --target snowclone --target-path target_clone specified? Am I missing something?

aranke · 2023-06-26T16:21:59Z

Do we have to have --target snowclone --target-path target_clone specified?

I think so, the --target flag for clone specifies the database, schema, and credentials for the destination we're cloning to.

Is there another way to pass this information in via the command line?

aranke · 2023-06-26T17:48:14Z

Confirmed that tables created in Snowflake have zero bytes (see active bytes column)

jtcohen6

I've also removed the ugly context.update hack (thanks @gshank @ChenyuLInx).

Good riddance! Thanks for making it happen, sorry I ever introduced it into this world :)

core/dbt/task/clone.py

…t_clone

aranke · 2023-06-27T17:24:35Z

@jtcohen6 Created a backlog spike to investigate combining merge_from_artifact and add_from_artifact.

I'll merge this PR in EOD today if there are no other comments.
cc @McKnight-42 @gshank @ChenyuLInx.

McKnight-42 · 2023-06-27T17:33:46Z

@aranke only questions I currently have will be around possible ways of universally adding the clean_up methods and some possible stuff around spark implementation. this pr looks good on my end

core/dbt/context/providers.py

sungchun12 · 2023-06-29T00:37:17Z

Love this

* break out a large test suite as a separate execution to avoid memory issues on windows CI runs (#7669) * Remove `DelayedFileHandler` (#7661) * remove DelayedFileHandler * Changelog * set_path to no-op * more no-ops for rpc * Clearer comments * Model Deprecation (#7562) * CT-2461: Work toward model deprecation * CT-2461: Remove unneeded conversions * CT-2461: Fix up unit tests for new fields, correct a couple oversights * CT-2461: Remaining implementation and tests for model/ref deprecation warnings * CT-2461: Changelog entry for deprecation warnings * CT-2461: Refine datetime handling and tests * CT-2461: Fix up unit test data * CT-2461: Fix some more unit test data. * CT-2461: Fix merge issues * CT-2461: Code review items. * CT-2461: Improve version -> str conversion * fix #7502: write run_results.json for run operation (#7655) * Honor `--skip-profile-setup` parameter when inside an existing project (#7609) * Honor `--skip-profile-setup` parameter when inside an existing project * Use project name as the profile name * Use separate file connections for reading and writing * Raise a custom exception when no adapters are installed * Test skipping interactive profile setup when inside a dbt project * Replace `assert_not_called()` since it does not work * Verbose CLI argument for skipping profile setup * Use separate file connections for reading and writing * Check empty list in a Pythonic manner * Include null checks in utils test base (#7672) * Include null checks in utils test base * Add tests for the schema test * Add tests for this macro --------- Co-authored-by: Mila Page <versusfacit@users.noreply.github.com> * Bump ubuntu from 23.04 to 23.10 (#7675) * Bump ubuntu from 23.04 to 23.10 Bumps ubuntu from 23.04 to 23.10. --- updated-dependencies: - dependency-name: ubuntu dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> * Add automated changelog yaml from template for bot PR --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Github Build Bot <buildbot@fishtownanalytics.com> * Adding performance modeling for 1.3.4 to refs/heads/main (#7525) * adding performance baseline for 1.3.4 * Add newline --------- Co-authored-by: Github Build Bot <buildbot@fishtownanalytics.com> Co-authored-by: leahwicz <60146280+leahwicz@users.noreply.github.com> * Adding performance modeling for 1.4.1 to refs/heads/main (#7527) * adding performance baseline for 1.4.1 * Add newline --------- Co-authored-by: Github Build Bot <buildbot@fishtownanalytics.com> Co-authored-by: leahwicz <60146280+leahwicz@users.noreply.github.com> * Adding performance modeling for 1.3.0 to refs/heads/main (#7530) * adding performance baseline for 1.3.0 * Add newline --------- Co-authored-by: Github Build Bot <buildbot@fishtownanalytics.com> Co-authored-by: leahwicz <60146280+leahwicz@users.noreply.github.com> * Adding performance modeling for 1.4.6 to refs/heads/main (#7532) * adding performance baseline for 1.4.6 * Add newline --------- Co-authored-by: Github Build Bot <buildbot@fishtownanalytics.com> Co-authored-by: leahwicz <60146280+leahwicz@users.noreply.github.com> * CT 2516 ensure that paths in Jinja context flags object are strings (#7678) * Relaxed the pyyaml dependency to >=5.3. (#7681) * Updating CODEOWNERS to consolidate (#7693) Consolidating to remove Language and Execution and instead default to the `core-team` * CT 2590 write pub artifact to log (#7686) * Bumping version to 1.6.0b2 and generate changelog * Adding link to 1.5 release notes (#7707) * Add `%` to adapter suite test cases for `persist_docs` (#7699) * Test table/view/column-level comments with `%` symbol * Test docs block with `%` symbol * Changelog entry * Update to reusable workflow for branch testing (#7676) * fix overlooked node12 case with abandonded marketplace action * update slack notification * remove spaces per formatting * replace with cli dispatch * move conditional * add explicit token, temp comment out slack * add checkout * checkout teh right branch * switch to PAT * add back repo checkout * manually check workflow status * fix notification * swap to reusable workflow * fix path * swap permissions * remove trigger * fix secrets * point to main * fix #7300: Enable state for deferral to be separate from state for selectors (#7690) * Target path should be relative to project dir, rather than current working directory (#7706) * Adding performance modeling for 1.2.0 to refs/heads/main (#7560) * adding performance baseline for 1.2.0 * Adding newline --------- Co-authored-by: Github Build Bot <buildbot@fishtownanalytics.com> Co-authored-by: leahwicz <60146280+leahwicz@users.noreply.github.com> * Readd exp_path for config deprecation warnings (#7536) * Dropped support for Python 3.7 (#7623) * Template rendering optimization (#7451) * CT-2478: Template rendering optimization * CT-2478: Fix type annotation, and accomodate non-string unit test cases. * fix StopIteration error when publication not found (#7710) * add ability to select models by access (#7739) * add ability to select models by access * changie * Update core/dbt/graph/selector_methods.py * pass optional sql_header to empty subquery sql rendering (#7734) * Fix doc link in selector.py (#7755) * Fix doc link in selector.py * Ran changie to modify changelog entry * Fix constraint rendering for expressions and foreign key constraint types (#7512) * add project_name to manifest metadata (#7754) * Version 0 for model works for latest_version (#7712) * force dependency between test models (#7767) * remove entire changes folder (#7766) Co-authored-by: Kshitij Aranke <kshitij.aranke@dbtlabs.com> * log PublicationArtifactAvailable even when partially parsing & public models unchanged (#7783) * Fixes #7299: dbt retry (#7763) * Fix not equals comparison to be null-safe for adapters/utils tests (#7776) * Fix names within functional test * Changelog entry * Test for implementation of null-safe equals comparison * Remove duplicated where filter * Fix null-safe equals comparison * Fix tests for `concat` and `hash` by using empty strings () instead of `null` * Remove macro namespace interpolation * Improve warnings for constraints and materialization types (#7696) * first pass * debugging * regen proto types * refactor to use warn_supported flag * PR feedback * Package-namespaced generate_x_name macro resolution (#7509) * generalize BaseModelConstraintsRuntimeEnforcement (#7805) * Fixes #7551: Create `add_from_artifact` to populate `state_relation` field of nodes (#7796) * Validate public models are not materialized as ephemeral (#7794) * fix error message for empty/None: --warn-error-options handling (#7735) * Adap 496/add test connection mode to debug (#7741) * --connection-flag * Standardize the plugin functions used by DebugTask * Cleanup redundant code and help logic along. * Add more output tests to add logic coverage and formatting. * Code review --------- Co-authored-by: Mila Page <versusfacit@users.noreply.github.com> * fix: removing dependency from dependencies.yml (#7743) * ADAP-2: Materialized Views (#7239) * changie * ADAP-387: Stub materialized view as a materialization (#7211) * init attempt at mv and basic forms of helper macros by mixing view and experimental mv sources * init attempt at mv and basic forms of helper macros by mixing view and experimental mv sources * remove unneeded return statement, rename directory * remove unneeded () * responding to some pr feedback * adjusting order of events for mv base work * move up prexisting drop of backup * change relatiion type to view to be consistent * add base test case * fix jinja exeception message expression, basic test passing * response to feedback, removeal of refresh infavor of combined create_as, etc. * swapping to api layer and stratgeies for default implementation (basing off postgres, redshift) * remove stratgey to limit need for now * remove unneeded story level changelog entry * add strategies to condtional in place of old macros * macro name fix * rename refresh macro in api level * align names between postgres and default to same convention * align names between postgres and default to same convention * change a create call to full refresh * pull adapter rename into strategy, add backup_relation as optional arg * minor typo fix, add intermediate relation to refresh strategy and initial attempt at further conditional logic * updating to feature main --------- Co-authored-by: Matthew McKnight <matthew.mcknight@dbtlabs.com> * ADAP-387: reverting db_api implementation (#7322) * changie * init attempt at mv and basic forms of helper macros by mixing view and experimental mv sources * remove unneeded return statement, rename directory * remove unneeded () * responding to some pr feedback * adjusting order of events for mv base work * move up prexisting drop of backup * change relatiion type to view to be consistent * add base test case * fix jinja exeception message expression, basic test passing * response to feedback, removeal of refresh infavor of combined create_as, etc. * swapping to api layer and stratgeies for default implementation (basing off postgres, redshift) * remove stratgey to limit need for now * remove unneeded story level changelog entry * add strategies to condtional in place of old macros * macro name fix * rename refresh macro in api level * align names between postgres and default to same convention * change a create call to full refresh * pull adapter rename into strategy, add backup_relation as optional arg * minor typo fix, add intermediate relation to refresh strategy and initial attempt at further conditional logic * updating to feature main * removing db_api and strategies directories in favor of matching current materialization setups * macro name change * revert to current approach for materializations * added tests * added `is_materialized_view` to `BaseRelation` * updated materialized view stored value to snake case * typo * moved materialized view tests into adapter test framework * add enum to relation for comparison in jinja --------- Co-authored-by: Mike Alfare <mike.alfare@dbtlabs.com> * ADAP-391: Add configuration change option (#7272) * changie * init attempt at mv and basic forms of helper macros by mixing view and experimental mv sources * move up pre-existing drop of backup * change relation type to view to be consistent * add base test case * fix jinja exception message expression, basic test passing * align names between postgres and default to same convention * init set of Enum for config * work on initial Enum class for on_configuration_change base it off ConstraintTypes which is also a str based Enum in core * add on_configuration_change to unit test expected values * make suggested name change to Enum class * add on_configuration_change to some integration tests * add on_configuration_change to expected_manifest to pass functional tests * added `is_materialized_view` to `BaseRelation` * updated materialized view stored value to snake case * moved materialized view tests into adapter test framework * add alter materialized view macro * change class name, and config setup * play with field setup for on_configuration_change * add method for default selection in enum class * renamed get_refresh_data_in_materialized_view_sql to align with experimental package * changed expected values to default string * added in `on_configuration_change` setting * change ignore to skip * updated default option for on_configuration_change on NodeConfig * removed explicit calls to enum values * add test setup for testing fail config option * updated `config_updates` to `configuration_changes` to align with `on_configuration_change` name * setup configuration change framework * skipped tests that are expected to fail without adapter implementation * cleaned up log checks --------- Co-authored-by: Mike Alfare <mike.alfare@dbtlabs.com> * ADAP-388: Stub materialized view as a materialization - postgres (#7244) * move the body of the default macros into the postgres implementation, throw errors if the default is used, indicating that materialized views have not been implemented for that adapter --------- Co-authored-by: Matthew McKnight <matthew.mcknight@dbtlabs.com> * ADAP-402: Add configuration change option - postgres (#7334) * changie * init attempt at mv and basic forms of helper macros by mixing view and experimental mv sources * remove unneeded return statement, rename directory * remove unneeded () * responding to some pr feedback * adjusting order of events for mv base work * move up prexisting drop of backup * change relatiion type to view to be consistent * add base test case * fix jinja exeception message expression, basic test passing * added materialized view stubs and test * response to feedback, removeal of refresh infavor of combined create_as, etc. * updated postgres to use the new macros structure * swapping to api layer and stratgeies for default implementation (basing off postgres, redshift) * remove stratgey to limit need for now * remove unneeded story level changelog entry * add strategies to condtional in place of old macros * macro name fix * rename refresh macro in api level * align names between postgres and default to same convention * change a create call to full refresh * pull adapter rename into strategy, add backup_relation as optional arg * minor typo fix, add intermediate relation to refresh strategy and initial attempt at further conditional logic * init copy of pr 387 to begin 391 implementation * init set of Enum for config * work on initial Enum class for on_configuration_change base it off ConstraintTypes which is also a str based Enum in core * remove postgres-specific materialization in favor of core default materialization * update db_api to use native types (e.g. str) and avoid direct calls to relation or config, which would alter the run order for all db_api dependencies * add clarifying comment as to why we have a single test that's expected to fail at the dbt-core layer * add on_configuration_change to unit test expected values * make suggested name change to Enum class * add on_configuration_change to some integretion tests * add on_configuration_change to expected_manifest to pass functuional tests * removing db_api and strategies directories in favor of matching current materialization setups * macro name change * revert to current approach for materializations * revert to current approach for materializations * added tests * move materialized view logic into the `/materializations` directory in line with `dbt-core` * moved default macros in `dbt-core` into `dbt-postgres` * added `is_materialized_view` to `BaseRelation` * updated materialized view stored value to snake case * moved materialized view tests into adapter test framework * updated materialized view tests to use adapter test framework * add alter materialized view macro * add alter materialized view macro * change class name, and config setup * change class name, and config setup * play with field setup for on_configuration_change * add method for default selection in enum class * renamed get_refresh_data_in_materialized_view_sql to align with experimental package * changed expected values to default string * added in `on_configuration_change` setting * change ignore to skip * added in `on_configuration_change` setting * updated default option for on_configuration_change on NodeConfig * updated default option for on_configuration_change on NodeConfig * fixed list being passed as string bug * removed explicit calls to enum values * removed unneeded test class * fixed on_configuration_change to be picked up appropriately * add test setup for testing fail config option * remove breakpoint, uncomment tests * update skip scenario to use empty strings * update skip scenario to avoid using sql at all, remove extra whitespace in some templates * push up initial addition of indexes for mv macro * push slight change up * reverting alt macro and moving the do create_index call to be more in line with other materializations * Merge branch 'feature/materialized-views/ADAP-2' into feature/materialized-views/ADAP-402 # Conflicts: # core/dbt/contracts/graph/model_config.py # core/dbt/include/global_project/macros/materializations/models/materialized_view/alter_materialized_view.sql # core/dbt/include/global_project/macros/materializations/models/materialized_view/create_materialized_view_as.sql # core/dbt/include/global_project/macros/materializations/models/materialized_view/get_materialized_view_configuration_changes.sql # core/dbt/include/global_project/macros/materializations/models/materialized_view/materialized_view.sql # core/dbt/include/global_project/macros/materializations/models/materialized_view/refresh_materialized_view.sql # core/dbt/include/global_project/macros/materializations/models/materialized_view/replace_materialized_view.sql # plugins/postgres/dbt/include/postgres/macros/materializations/materialized_view.sql # tests/adapter/dbt/tests/adapter/materialized_views/base.py # tests/functional/materializations/test_materialized_view.py * merge feature branch into story branch * merge feature branch into story branch * added indexes into the workflow * fix error in jinja that caused print error * working on test messaging and skipping tests that might not fit quite into current system * add drop and show macros for indexes * add drop and show macros for indexes * add logic to determine the indexes to create or drop * pulled index updates through the workflow properly * convert configuration changes to fixtures, implement index changes into tests * created Model dataclass for readability, added column to swap index columns for testing * fixed typo --------- Co-authored-by: Matthew McKnight <matthew.mcknight@dbtlabs.com> * ADAP-395: Implement native materialized view DDL (#7336) * changie * changie * init attempt at mv and basic forms of helper macros by mixing view and experimental mv sources * init attempt at mv and basic forms of helper macros by mixing view and experimental mv sources * remove unneeded return statement, rename directory * remove unneeded () * responding to some pr feedback * adjusting order of events for mv base work * move up prexisting drop of backup * change relatiion type to view to be consistent * add base test case * fix jinja exeception message expression, basic test passing * added materialized view stubs and test * response to feedback, removeal of refresh infavor of combined create_as, etc. * updated postgres to use the new macros structure * swapping to api layer and stratgeies for default implementation (basing off postgres, redshift) * remove stratgey to limit need for now * remove unneeded story level changelog entry * add strategies to condtional in place of old macros * macro name fix * rename refresh macro in api level * align names between postgres and default to same convention * align names between postgres and default to same convention * change a create call to full refresh * pull adapter rename into strategy, add backup_relation as optional arg * minor typo fix, add intermediate relation to refresh strategy and initial attempt at further conditional logic * init copy of pr 387 to begin 391 implementation * updating to feature main * updating to feature main * init set of Enum for config * work on initial Enum class for on_configuration_change base it off ConstraintTypes which is also a str based Enum in core * remove postgres-specific materialization in favor of core default materialization * update db_api to use native types (e.g. str) and avoid direct calls to relation or config, which would alter the run order for all db_api dependencies * add clarifying comment as to why we have a single test that's expected to fail at the dbt-core layer * add on_configuration_change to unit test expected values * make suggested name change to Enum class * add on_configuration_change to some integretion tests * add on_configuration_change to expected_manifest to pass functuional tests * removing db_api and strategies directories in favor of matching current materialization setups * macro name change * revert to current approach for materializations * revert to current approach for materializations * added tests * move materialized view logic into the `/materializations` directory in line with `dbt-core` * moved default macros in `dbt-core` into `dbt-postgres` * added `is_materialized_view` to `BaseRelation` * updated materialized view stored value to snake case * typo * moved materialized view tests into adapter test framework * updated materialized view tests to use adapter test framework * add alter materialized view macro * add alter materialized view macro * added basic sql to default macros, added postgres-specific sql for alter scenario, stubbed a test case for index update * change class name, and config setup * change class name, and config setup * play with field setup for on_configuration_change * add method for default selection in enum class * renamed get_refresh_data_in_materialized_view_sql to align with experimental package * changed expected values to default string * added in `on_configuration_change` setting * change ignore to skip * added in `on_configuration_change` setting * updated default option for on_configuration_change on NodeConfig * updated default option for on_configuration_change on NodeConfig * fixed list being passed as string bug * fixed list being passed as string bug * removed explicit calls to enum values * removed explicit calls to enum values * removed unneeded test class * fixed on_configuration_change to be picked up appropriately * add test setup for testing fail config option * remove breakpoint, uncomment tests * update skip scenario to use empty strings * update skip scenario to avoid using sql at all, remove extra whitespace in some templates * push up initial addition of indexes for mv macro * push slight change up * reverting alt macro and moving the do create_index call to be more in line with other materializations * Merge branch 'feature/materialized-views/ADAP-2' into feature/materialized-views/ADAP-402 # Conflicts: # core/dbt/contracts/graph/model_config.py # core/dbt/include/global_project/macros/materializations/models/materialized_view/alter_materialized_view.sql # core/dbt/include/global_project/macros/materializations/models/materialized_view/create_materialized_view_as.sql # core/dbt/include/global_project/macros/materializations/models/materialized_view/get_materialized_view_configuration_changes.sql # core/dbt/include/global_project/macros/materializations/models/materialized_view/materialized_view.sql # core/dbt/include/global_project/macros/materializations/models/materialized_view/refresh_materialized_view.sql # core/dbt/include/global_project/macros/materializations/models/materialized_view/replace_materialized_view.sql # plugins/postgres/dbt/include/postgres/macros/materializations/materialized_view.sql # tests/adapter/dbt/tests/adapter/materialized_views/base.py # tests/functional/materializations/test_materialized_view.py * merge feature branch into story branch * merge feature branch into story branch * added indexes into the workflow * fix error in jinja that caused print error * working on test messaging and skipping tests that might not fit quite into current system * Merge branch 'feature/materialized-views/ADAP-2' into feature/materialized-views/ADAP-395 # Conflicts: # core/dbt/include/global_project/macros/materializations/models/materialized_view/get_materialized_view_configuration_changes.sql # plugins/postgres/dbt/include/postgres/macros/adapters.sql # plugins/postgres/dbt/include/postgres/macros/materializations/materialized_view.sql # tests/adapter/dbt/tests/adapter/materialized_views/test_on_configuration_change.py # tests/functional/materializations/test_materialized_view.py * moved postgres implemention into plugin directory * update index methods to align with the configuration update macro * added native ddl to postgres macros * removed extra docstring * updated references to View, now references MaterializedView * decomposed materialization into macros * refactor index create statement parser, add exceptions for unexpected formats * swapped conditional to check for positive state * removed skipped test now that materialized view is being used * return the results and logs of the run so that additional checks can be applied at the adapter level, add check for refresh to a test * add check for indexes in particular for apply on configuration scenario * removed extra argument * add materialized views to get_relations / list_relations * typos in index change logic * moved full refresh check inside the build sql step --------- Co-authored-by: Matthew McKnight <matthew.mcknight@dbtlabs.com> * removing returns from tests to stop logs from printing * moved test cases into postgres tests, left non-test functionality in base as new methods or fixtures * fixed overwrite issue, simplified assertion method * updated import order to standard * fixed test import paths * updated naming convention for proper test collection with the test runner * still trying to make the test runner happy * rewrite index updates to use a better source in Postgres * break out a large test suite as a separate run * update `skip` and `fail` scenarios with more descriptive results * typo * removed call to skip status * reverting `exceptions_jinja.py` * added FailFastError back, the right way * removed PostgresIndex in favor of the already existing PostgresIndexConfig, pulled it into its own file to avoid circular imports * removed assumed models in method calls, removed odd insert records and replaced with get row count * fixed index issue, removed some indirection in testing * made test more readable * remove the "apply" from the tests and put it on the base as the default * generalized assertion for reuse with dbt-snowflake, fixed bug in record count utility * fixed type to be more generic to accommodate adapters with their own relation types * fixed all the broken index stuff * updated on_configuration_change to use existing patterns * updated on_configuration_change to use existing patterns * reflected update in tests and materialization logic * reflected update in tests and materialization logic * reverted the change to create a config object from the option object, using just the option object now * reverted the change to create a config object from the option object, using just the option object now * modelled database objects to support monitoring all configuration changes * updated "skip" to "continue", throw an error on non-implemented macro defaults * updated "skip" to "continue", throw an error on non-implemented macro defaults * updated "skip" to "continue", throw an error on non-implemented macro defaults * updated "skip" to "continue", throw an error on non-implemented macro defaults * reverted centralized framework, retained a few reusable base classes * updated names to be more consistent * readability updates * added readme specifying that `relation_configs` only supports materialized views for now --------- Co-authored-by: Matthew McKnight <matthew.mcknight@dbtlabs.com> Co-authored-by: Matthew McKnight <91097623+McKnight-42@users.noreply.github.com> * Add SemanticModel Node Type (#7769) * Add dbt-semantic-interfaces as a dependency With the integration with MetricFlow we're taking a dependency on `dbt-semantic-interfaces` which acts as the source of truth for protocols which MetricFlow and dbt-core need to agree on. Additionally we're hard pinning to 0.1.0.dev3 for now. We plan on having a less restrictive specification when dbt-core 1.6 hits GA. * Add implementations of DSI Metadata protocol to nodes.py * CT-2521: Initial work on adding new SemanticModel node * CT-2521: Second rough draft of SemanticModels * CT-2521: Update schema v10 * CT-2521: Update unit tests for new SemanticModel collection in manifest * CT-2521: Add changelog entry * CT-2521: Final touches on initial implementation of SemanticModel parsing * Change name of Metadata class to reduce potential for confusion * Remove "Replaceable" inheritance, per review * CT-2521: Rename internal variables from semantic_models to semantic_nodes * CT-2521: Update manifest schema to reflect change --------- Co-authored-by: Quigley Malcolm <quigley.malcolm@dbtlabs.com> * Update spec MetricNode for dbt x MetricFlow integration and begin outputting semantic manifest artifact (#7812) * Refactor MetricNode definition to satisfy DSI Metric protocol * Fix tests involving metrics to have updated properties * Update UnparsedMetricNode to match new metric yaml spec * Update MetricParser for new unparsed and parsed MetricNodes * Remove `rename_metric_attr` We're intentionally breaking the spec. There will be a separate tool provided for migrating from dbt-metrics to dbt x metricflow. This bit of code was renaming things like `type` to `calculation_method`. This is problematic because `type` is on the new spec, while `calculation_method` is not. Additionally, since we're intentionally breaking the spec, this function, `rename_metric_attr`, shouldn't be used for any property renaming. * Fix tests for Metrics (1.6) changes * Regenerated v10 manifest schema and associated functional test artifact state * Remove no longer needed tests * Skip / comment out tests for metrics functionality that we'll be implementing later * Begin outputting semantic manifest artifact on every run * Drop metrics during upgrade_manifest_json if manifest is v9 or before * Update properties of `minimal_parsed_metric_dict` to match new metric spec * Add changie entry for metric node breaking changes * Add semantic model nodes to semantic manifest * Bumping version to 1.6.0b3 and generate changelog * Rm spaces from NodeType strings (#7842) * Rm space from NodeType strings * Add changelog entry * Add new index.html and changelog yaml files from dbt-docs (#7836) * Move remaining unit tests to `/tests` directory (#7843) * moved remaining unit tests to directory * fix path * Delete profiles.yml * rename remaining test refs * Use project directory in path selector instead of cwd (#7829) * Use contextvar to store and get project_root for path selector method * Changie * Modify test to check Path selector with project-dir * Don't set cv_project_root in base task if no config * update adapters url (#7779) * update adapters url in response to [docs.getedbt.com pr 3465](https://github.com/dbt-labs/docs.getdbt.com/issues/3465), updating this error message to point to the correct URL, which was recently changed. old URL: https://docs.getdbt.com/docs/supported-data-platforms#adapter-installation new URL: https://docs.getdbt.com/docs/connect-adapters#install-using-the-cli thank you @dbeatty10 for your 🦅 👀 ! * adding changie entry * Update .changes/unreleased/Breaking Changes-20230612-161159.yaml --------- Co-authored-by: Emily Rockman <emily.rockman@dbtlabs.com> * Allow ProjectDependency to have extra fields (#7834) * Allow ProjectDependency to have extra fields * changelog * Respect column `quote` config in model contracts (#7537) * Update SemanticModel node to match DSI 0.1.0dev3 protocols (#7848) * Add tests to ensure our semantic layer nodes satisfy the DSI protocols These tests create runtime checkable versions of the protocols defined in DSI. Thus we can instantiate instances of our semantic layer nodes and use `isinstance` to check that they satisfy the protocol. These `runtime_checkable` versions of the protocols should only exist in testing and should never be used in the actual package code. * Update the `Dimension` object of `SemanticModel` node to match DSI protocol * Make `UnparsedDimension` more strict and update schema readers accordingly * Update the `Entity` object of `SemanticModel` node to match DSI protocol * Make `UnparsedEntity` more strict and update schema readers accordingly * Update the `Measure` object of `SemanticModel` node to match DSI protocol * Make `UnparsedMeasure` more strict and update schema readers accordingly * Update the `SemanticModel` node to match DSI protocol A lot of the additions are helper functions which we don't actually use in core. This is a known issue. We're in the process of removing a fair number of them from the DSI protocol spec. However, in the meantime we need to implement them to satisfy the protocol unfortunately. * Make `UnparsedSemanticModel` more strict and update schema readers accordingly * Changie entry for updating SemanticModel node * Bumping version to 1.6.0b4 and generate changelog * Fix: `dbt show --inline` with `private` models (#7838) * Add functional test * Check resource_type before DbtReferenceError * Changelog entry * Bump mypy from 0.981 to 1.0.1 (#7027) * Bump mypy from 0.981 to 1.0.1 Bumps [mypy](https://github.com/python/mypy) from 0.981 to 1.0.1. - [Release notes](https://github.com/python/mypy/releases) - [Commits](https://github.com/python/mypy/compare/v0.981...v1.0.1) --- updated-dependencies: - dependency-name: mypy dependency-type: direct:development update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> * Add automated changelog yaml from template for bot PR * upgrade mypy and fix all errors * fixing some duplicate imports from conflict resolution * fix mypy errors from merging in main --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Github Build Bot <buildbot@fishtownanalytics.com> Co-authored-by: Emily Rockman <emily.rockman@dbtlabs.com> * Add AdapterRegistered event log message (#7862) * Add AdapterRegistered event log message * Add AdapterRegistered to unit test * make versioning and logging consistent * make versioning and logging consistent * add to_version_string * remove extra equals * format fire_event * Allow setting packages in dependencies.yml and move dependencies to runtime config (#7857) * [CT-2696] Skip jinia parsing of metric filters (#7885) * Update metric filters in testing fixtures I incorrectly wrote the tests such that they didn't include curly braces, `{{..}}`, around things like `dimension(..)` for filters. This updates the tests fixtures to have proper filter specifications * Skip jinja rendering of `filter` key of metrics Note that `filter` can show up in multiple places: as a root key on a metric (`metric.filter`), on a metric input (`metric.type_params.metrics[x].filter`), denominator (`metric.type_params.denominator.filter`), numerator (`metric.type_params.numerator.filter`), and a metric input measure (`metric.type_params.measure.filter` and `metric.type_params.measures[x].filter`). In this commit we skip all of them :) * Add changie doc for skipping jinja parsing for metric filters * Update yaml renderer test for metrics * Ct-2690 Support dbt-semantic-interfaces 0.1.0dev5 (#7888) * Upgrade to dbt-semantic-interfaces v0.1.0dev5 This is a fairly simple upgrade. Literally it's just pointing at the the new versions. The v3 schemas are directly compatible with v5 because there were no protocol level changes from v3 to v5. All the changers were updates to tools MetricFlow uses from DSI, not tools that we ourselves are using in core (yet). * Add changie doc for DSI version bump * Upgrade core to support dbt-semantic-interfaces `0.1.0dev7` (#7903) * Bump DSI dependency version to 0.1.0dev7 * Cleaner DSI type enum importing Previoulsy we had to use individual import paths for each type enum that dbt-semantic-interfaces provided. However, dbt-semantic-interfaces has been updated to allow for importing all the type enums from a singular path. * Cleaner DSI protocol importing Previoulsy we had to use individual import paths for each protocol that dbt-semantic-interfaces provided. However, dbt-semantic-interfaces has been updated to allow for importing all the protocols from a singular path. * Add semantic protocol satisifcation test for metric type params * Replace `metric.type_params.measures` with `metric.type_params.input_measures` In DSI 0.1.0dev7 `measures` on metric type params became `input_measures`. Additionally `input_measures` should not be user specified but something we compile at parse time, thus we've removed it from `UnparsedMetricTypeParams`. Finally, actually populating `input_measures` is somewhat complicated due to the existance of derived metrics, thus that work is being pushed off to CT-2707. * Update metric numerator/denominator to be `MetricInput`s In DSI 0.1.0dev7 `metric.type_params.numerator` and `metric.type_params.denominator` switched from being `MetricInputMeasure`s to `MetricInput`s. This commit reflects that change. Additionally, some helper functions on metric type params were removed related to the numerator and denominator. Thus we've removed them respectively in this commit. * Add protocol satisfaction tests for `MetricInput` and `MetricInputMeasure` * Add `post_aggregation_reference` to `MetricInput` and fix typo in `MetricInputMeasure` DSI 0.1.0dev7 added `post_aggregation_reference` to the `MetricInput` protocol, thus we've added it to our implementation in core. Additionally, we had a typo in a method name in our implementation of `MetricInputMeasure`, ironically a similar function to the one we've added for `MetricInput` * Changie doc for upgraded to DSI 0.1.0dev7 * Fix parsing of metric numerator and denominator in schema_yaml_readers Previously numerator and denominator of a metric were `MetricInputMeasure`s, now they're `MetricInput`s. Changing the typing isn't enough though. We have parsing functions in `schema_yaml_readers` which were specifically parsing the numerator and denominator as if they were `MetricInputMeasure`s. Thus we had to updating the schema_yaml_readers to parse them as `MetricInput`s. During this we had some logic in a parsing function `_get_metric_inputs` which could be abstracted to newly added functions. * Fixes #7753: Fix regression in `run-operation` to not require the name of the package to run (#7811) * Reorganize, annotate, and revise dependency pins (#7368) * Reorganize + annotate dependencies * Loosen pins on other dbt-labs packages * Minor pins, no patch pins * Rm support for py37 * Fix protobuf pin * Bump networkx upper bound #6551 * Do not jinja render packages from dependencies yml (#7910) * Add some comments to methods constructing Project/RuntimeConfig * Save flag that packages dict came from dependencies.yml * Test for not rendering packages_dict * Changie * Ensure packages_yml_dict and dependencies_yml_dict are dictionaries * Ensure "packages" passed to render_packages is a dict * Remove --config-dir instead of fixing #7774 (#7793) * Stringfy the dir always to solve the bug. * Add changelog --------- Co-authored-by: Mila Page <versusfacit@users.noreply.github.com> * Refactor/unify public and model nodes (#7891) * Further Integrate Semantic Models (#7917) * CT-2651: Add Semantic Models to the manifest and various pieces of graph linking code * CT-2651: Finish integrating semantic models into the partial parsing system * CT-2651: More semantic model details for partial parsing * CT-2651: Remove merged references to project_dependencies * CT-2651: Revise changelog entry * CT-2651: Disable unit test until partial parsing of semantic models is complete. * CT-2651: Temporarily disable an apparently-flaky test. * Resolve SemanticModel refs in the same way as other refs (#7895) * UninstalledPackagesFoundError references correct packages specified path (#7886) * Fix: safe remove of external nodes from nodes.depends_on (#7923) * Bumping version to 1.6.0b5 and generate changelog * CT-2719: Rename the semantic_nodes collection on the manifest to semantic_models (#7927) * Improved Semantic expr specification handling (#7876) * Update semantic model parsing test to check different measure expr types * Allow semantic model measure exprs to be defined with ints and bools in yaml Sometimes the expr for a measure can defined in yaml with a bool or an int. However, we were only allowing for strings. There was a work around for this, which was wrapping your bool or int in double quotes in the yaml, but this can be fairly annoying for the end user. * Changie doc for fixing measure expr yaml specification * Improved Semantic Model Measure Percentile defaults (#7877) * Update semantic model parsing test to check measure agg params * Make `use_discrete_percentile` and `use_approximate_percentile` non optional and default false This was a mistake in our implementation of the MeasureAggregationParams. We had defined them as optional and defaulting to `None`. However, as the protocol states, they cannot be `None`, they must be a boolean value. Thus now we now ensure them. * Add changie doc for measure percentile fixes * CT-2732: Fix selector methods to include semantic models (#7936) * Bumping version to 1.6.0b6 and generate changelog * update mashumaro to 3.8.1 (#7951) * Update mashumaro to 3.8 * Change to 3.8.1 * Changie * Fix comment for `dbt retry` (#7932) * Factor Out Repeated Logic in the PartialParsing Class (#7952) * CT-2711: Add remove_tests() call to delete_schema_source() so that call sites are more uniform with other node deletion call sites. This will enable further code factorization. * CT-2711: Factor repeated code section (mostly) out of PartialParsing.handle_schema_file_changes() * CT-2711: Factor a repeated code section out of schedule_nodes_for_parsing() * Fix macro namespace search packages (#5804) * allow setting enabled and depends_on_nodes from ModelNodeArgs (#7930) * Set access model node args (#7966) * Add Partial Parsing Support for Semantic Models (#7964) * CT-2711: Add partial parsing support for semantic models * CT-2711: Fix typo identified in code review * Add merge incremental strategy for postgres (#6951) * Add merge incremental strategy * Expect merge to be a valid strategy for Postgres --------- Co-authored-by: Anders Swanson <anders.swanson@dbtlabs.com> Co-authored-by: Doug Beatty <doug.beatty@dbtlabs.com> * Handle state:modified for external nodes (#7925) * Add thread_id context var (#7942) * Add thread_id context var * Changie * Fix context test * Update .changes/unreleased/Features-20230623-173357.yaml Co-authored-by: Doug Beatty <44704949+dbeatty10@users.noreply.github.com> * Fix tests --------- Co-authored-by: Doug Beatty <44704949+dbeatty10@users.noreply.github.com> * Add invocation command flag (#7939) * Add invocation command flag * Add changie entry * Update .changes/unreleased/Features-20230623-111254.yaml * consolidate cross-project ref entrypoint + plugin framework (#7955) * Add restrict-access to dbt_project.yml (#7962) * Bumping version to 1.6.0b7 and generate changelog * Fix CTE insertion position when the model uses WITH RECURSIVE (#7350) (#7414) * Replace space with underscore in NodeType strings (#7947) * Use events.contextvar because of multiprocessing unable to pickle ContextVar (#7949) * Add task contextvars to events/contextvars.py * Use events.contextvars instead of task.contextvars * Changie * CT-2757: Fix unit test which broke due to merge issues (#7978) * `dbt clone` (#7881) Co-authored-by: Matthew McKnight <matthew.mcknight@dbtlabs.com> * CT-2707: Populate metric input measures (#7984) * Fix tests fixtures which were using measures for metric numerator/denominators In our previous upgrade to DSI dev7, numerators and denominators for metrics switched from being `MetricInputMeasure`s to `MetricInput`s. I.e. metric numerators and denominators should references other metrics, not semantic model measures. However, at that time, we weren't actually doing anything with numerators and denominators in core, so no issue got raised. The changes we are about to make though are going to surface these issues.. * Add tests for ensuring a metric's `input_measures` gets properly populated * Begin populating `metric.type_params.input_measures` This isn't my favorite bit of code. Mostly because there are checks for existence which really should be handled before this point, however a good point for that to happen doesn't exist currently. For instance, in an ideal world by the time we get to `_process_metric_node`, if a metric is of type `RATIO` and the nominator and denominator should be guaranteed. * Update test checking that disabled metrics aren't added to the manifest metrics We updated from the metric `number_of_people` to `average_tenure_minus_people` for this test because disabling `number_of_people` raised other exceptions at parse time due to a metric referencing a disabled metric. The metric `average_tenure_minus_people` is a leaf metric, and so for this test, it is a better candidate. * Update `test_disabled_metric_ref_model` to have more disabled metrics There are metrics which depend on the metric `number_of_people`. If `number_of_people` is disabled without the metrics that depend on it being disabled, then a different (expected) exception would be raised than the one this test is testing for. Thus we've disabled those downstream metrics. * Add test which checks that metrics depending on disabled metrics raise an exception * Add changie doc for populating metric input measures * Unified to UTC (#7665) * UnifiedToUTC * Check proximity of dbt_valid_to and deleted time * update the message to print if the assertion fails * add CHANGELOG entries * test only if naive * Added comments about naive and aware * Generalize comparison of datetimes that are "close enough" --------- Co-authored-by: Doug Beatty <doug.beatty@dbtlabs.com> * type + fix typo in ModelNodeArgs.unique_id (#7992) * Remove pin of sqlparse, minor refactoring, add tests (#7993) * [CT-2551] Make state selection MECE (#7773) * ct-2551: adds old and unmodified state selection methods * ct-2551: update check_unmodified_content to simplify * add unit and integration tests for unmodified and old * add changelog entry * ct-2551: reformatting of contingent adapter assignment list * Added note before running integration tests (#7657) Co-authored-by: Kshitij Aranke <kshitij.aranke@dbtlabs.com> * Add time spine table configuration to semantic manifest (#7996) * Bump dbt-semantic-interface to dev8 * Create time_spline_table_configuration in semantic manifest * Add metricflow_time_spin to semantic_models tests * Remove skip from test * Changie * Update exception message Co-authored-by: Quigley Malcolm <QMalcolm@users.noreply.github.com> --------- Co-authored-by: Quigley Malcolm <QMalcolm@users.noreply.github.com> * allow on_schema_change: fail for incremental models with contracts (#8006) * versioned node selection with underscore delimiting (#7995) * Bumping version to 1.6.0b8 and generate changelog * fire proper event for inline query error (#7960) * CT-2752: Extract methods to new SemanticManifest class for better encapsulation of details. (#8012) * CT-2780 [Docs] Fix-toc-links-in-contributing-md (#8017) * docs(contributing): fix-toc-link-in-contributing-md * docs(contributing-md): fix-link2 * Fix-typo * Remove backtick from href * changie new * Cough commit / trigger CI * Nicer error message for contracted model missing 'columns' (#8024) * ModelNodeArgs.unique_id - include v (#8038) * pin click (#8050) * pin click * changelog * add workflow for flaky test testing (#8044) * add workflow for flaky test testing * improve docs * rename workflow * update default input * add min passing tests * partial parse file path (#8032) * Update the PR template (#7892) * add interface changes section to the PR template * update entire template * split up choices for tests and interfaces * minor formatting change * add line breaks * actually put in line breaks * revert split choices in checklist * add line breaks to top * move docs link * typo * Semantic Model Validation (#8049) * Use dbt-semantic-interface validations on semantic models and metrics defined in Core. * Remove empty test, since semantic models don't generate any validation warnings. * Add changelog entry. * Temporarily remove requirement that there must be semantic models definied in order to define metrics * Fix accidental propagation of log messages to root logger (#7882) * Fix accidental propagation of log messages to root logger. * Add changelog entry * Fixed an issue which blocked debug logging to stdout with --log-level debug, unless --debug was also used. * add negative part number test case for split part cross db util (#7200) * add negative test case * changie * missed a comma * Update changelog entry * Add a negative number (rather than subtract a positive number) --------- Co-authored-by: Doug Beatty <44704949+dbeatty10@users.noreply.github.com> Co-authored-by: Doug Beatty <doug.beatty@dbtlabs.com> * Remove `create_metric` as a public facing `SemanticModel.Measure` property (#8068) * Remove `create_metric` as a public facing `SemanticModel.Measure` property We want to add `create_metric`. The `create_metric` property will be incredibly useful. However, at this time it is not hooked up, and we don't have time to hook it up before the code freeze for 1.6.0rc of core. As it doesn't do anything, we shouldn't allow people to specify it, because it won't do what one would expect. We plan on making the implementation of `create_metric` a priority for 1.7 of core * Changie doc for the removal of create_metric property * Flaky Test Workflow (#8055) * add permissions * replace db setup * try with bash instead of just pytest flags * fix test command * remove spaces * remove force-flaky flag * add starting vlaues * add mac and windows postgres isntall * define use bash * fix typo * update output report * tweak last if condition * clarify failures/successful runs * print running success and failure tally * just output pytest instead of capturing it * set shell to not exit immediately on exit code * add formatting around results for easier scanning * more output formatting * add matrix to unlock parallel runners * increase to ten batches * update debug * add comment * clean up comments * Use Ubuntu 22.04.2 LTS (Jammy Jellyfish) since it is a long-term supported release (#8071) * Use Ubuntu 22.04.2 LTS (Jammy Jellyfish) since it is a long-term supported release * Changelog entry * Fixes #7785: fail-fast behavior (#8066) * Add validate_sql method to base adapter with implementation for SQLAdapters (#8001) * Add dry_run method to base adapter with implementation for SQLAdapters resolves #7839 In the CLI integration, MetricFlow will issue dry run queries as part of its warehouse-level validation of the semantic manifest, including all semantic model and metric definitions. In most cases, issuing an `explain` query is adequate, however, BigQuery does not support the `explain` keyword and so we cannot simply pre-pend `explain` to our input queries and expect the correct behavior across all contexts. This commit adds a dry_run() method to the BaseAdapter which mirrors the execute() method in that it simply delegates to the ConnectionManager. It also adds a working implementation to the SQLConnectionManager and includes a few test cases for adapter maintainers to try out on their own. The current implementation should work out of the box with most of our adapters. BigQuery will require us to implement the dry_run method on the BigQueryConnectionManager, and community-maintained adapters can opt in by enabling the test and ensuring their own implementations work as expected. Note - we decided to make these concrete methods that throw runtime exceptions for direct descendants of BaseAdapter in order to avoid forcing community adapter maintainers to implement a method that does not currently have any use cases in dbt proper. * Switch dry_run implementation to be macro-based The common pattern for engine-specific SQL statement construction in dbt is to provide a default macro which can then be overridden on a per-adapter basis by either adapter maintainers or end users. The advantage of this is users can take advantage of alternative SQL syntax for performance or other reasons, or even to enable local usage if an engine relies on a non-standard expression and the adapter maintainer has not updated the package. Although there are some risks here they are minimal, and the benefit of added expressiveness and consistency with other similar constructs is clear, so we adopt this approach here. * Improve error message for InvalidConnectionError in test_invalid_dry_run. * Rename dry_run to validate_sql The validate_sql name has less chance of colliding with dbt's command nomenclature, both now and in some future where we have dry-run operations. * Rename macro and test files to validate_sql * Fix changelog entry * Update validate sql test classes to new nomenclature (#8076) The original implementation of validate_sql was called dry_run, but in the rename the test classes and much of their associated documentation still retained the old naming. This is mainly cosmetic, but since these test classes will be imported into adapter repositories we should fix this now before the wrong name proliferates. * Remove volume declaration (#8069) * Remove volume declaration * Changelog entry --------- Co-authored-by: Doug Beatty <doug.beatty@dbtlabs.com> * update contributing.md reference to test/integration (#8073) * Er/ct 2675 test custom target (#8079) * remove skip * fix retry test * 8030/fix contract checksum (#8072) * Add semantic_models to tracked resource counts (#8078) * Add semantic_models to tracked resource counts * Add changelog entry * Simplify node statistic tabulation. * Remove review comment. Replace with explanation. --------- Co-authored-by: Peter Allen Webb <peter.webb@dbtlabs.com> * CT-2691 Fix the populating of a `Metric`'s `depends_on` property (#8015) * Add metrics from metric type params to a metric's depends_on * Add Lookup utility for finding `SemanticModel`s by measure names * Add the `SemanticModel` of a `Metric`'s measure property to the `Metric`'s `depends_on` * Add `SemanticModelConfig` to `SemanticModel` Some tests were failing due to `Metric`'s referencing `SemanticModels`. Specifically there was a check to see if a referenced node was disabled, and because `SemanticModel`'s didn't have a `config` holding the `enabled` boolean attr, core would blow up. * Checkpoint on test fixing * Correct metricflow_time_spine_sql in test fixtures * Add check for `SemanticModel` nodes in `Linker.link_node` Now that `Metrics` depend on `SemanticModels` and `SemanticModels` have their own dependencies on `Models` they need to be checked for in the `Linker.link_node`. I forget the details but things blow up without it. Basically it adds the SemanticModels to the dependency graph. * Fix artifacts/test_previous_version_state.py tests * fix access/test_access.py tests * Fix function metric tests * Fix functional partial_parsing tests * Add time dimension to semantic model in exposures fixture * Bump DSI version to a minimum of 0.1.0dev10 DSI 0.1.0dev10 fixes an incoherence issue in DSI around `agg_time_dimension` setting. This incoherence was that `measure.agg_time_dimension` was being required, even though it was no longer supposed to be a required attribute (it's specificially typed as optional in the protocol). This was causing a handful of tests to fail because the `semantic_model.defaults.agg_time_dimension` value wasn't being respected. Pulling in the fix from DSI 0.1.0dev10 fixes the issue. Interestingly after bumping the DSI version, the integration tests were still failing. If I ran the tests individually they passed though. To get `make integration` to run properly I ended up having to clear my `.tox` cache, as it seems some outdated state was being persisted. * Add test specifically for checking the `depends_on` of `Metric` nodes * Re-enable test asserting calling metric nodes in models * Migrate `checked_agg_time_dimension` to `checked_agg_time_dimension_for_measure` DSI 0.1.0dev10 moved `checked_agg_time_dimension` from the `Measure` protocol to the `SemanticModel` protocol as `checked_agg_time_dimension_for_measure`. This finishes a change where for a given measure either the `Measure.agg_time_dimension` or the measure's parent `SemanticModel.defaults.agg_time_dimension` needs to be set, instead of always require the measure's `Measure.agg_time_dimension`. * Add changie doc for populating metric --------- Co-authored-by: Gerda Shank <gerda@dbtlabs.com> * add __test__ = False to non-test classes that start with Test (#8086) * gitignore user.yml and profiles.yml (#8087) * [CT-2821] Support `dbt-semantic-interfaces~=0.1.0rc1` (#8085) * Bump version support for `dbt-semantic-interfaces` to `~=0.1.0rc1` * Add tests for asserting WhereFilter satisfies protocol * Add `call_parameter_sets` to `WhereFilter` class to satisfy protocol * Changie doc for moving to DSI 0.1.0rc1 * [CT-2822] Fix `NonAdditiveDimension` Implementation (#8089) * Add test to ensure `NonAdditiveDimension` implementation satisfies protocol * Fix typo in `NonAdditiveDimension`: `window_grouples` -> `window_groupings` * Add changie doc for typo fix in NonAdditiveDimension * Fix query comment tests (#7928) (#7928) * add env vars for datadog ci visibility (#8097) * add env vars for datadog ci visibility * modify pytest command for tracing * fix posargs * move env vars to job that needs them * add test repeater to DD * swap flags * Apply new integration tests to existing framework to identify supported features (#8099) * applied new integration tests to existing framework * applied new integration tests to existing framework * generalized tests for reusability in adapters; fixed drop index issue * generalized tests for reusability in adapters; fixed drop index issue * removed unnecessary overrides in tests * adjusted import to allow for usage in adapters * adjusted import to allow for usage in adapters * removed fixture artifact * generalized the materialized view fixture which will need to be specific to the adapter * unskipped tests in the test runner package * corrected test condition * corrected test condition * added missing initial build for the relation type swap tests * bumpversion 1.7.0a1 (#8111) * Cleanup main after cutting new 1.6.latest branch (#8102) * Publish coverage results to codecov.io (#8127) * wrap deprecation warnings in warn_or_error calls (#8129) * Add new index.html and changelog yaml files from dbt-docs (#8131) * Rm docs changelog entry (#8147) * Rearrange pp_versioned_models test (#8150) * format exception from dbtPlugin.initialize (#8143) * Hotfix for 372: Use JSONEncoder in json.dumps (#8151) * Correcting spelling of partition (#8101) * Correcting spelling of partition * Changie entry * add a node status (#8174) * Add implementation issue template (#8176) * [CT-1849] `_connection_exception_retry` handles `EOFError` exceptions (#8182) * Add test for checking that `_connection_exception_retry` handles `EOFError`s * Update `_connection_exception_retry` to handle `EOFError` exceptions * Add changie docs for `_connection_exception_retry` handling `EOFError` exceptions * Fixed double underline (#7944) * Fixed double-underline * backward compatibility postgres_get_relations * Remove invalid comments * compatibility adapter and get_relation * fix generic for call * fix adapter dispatch grammar issue * [CT-1483] Let macro names include word materialization (#8181) * Add test asserting that a macro with the work materializtion doesn't cause issues * Let macro names include the word `materialization` Previously we were checking if a macro included a materialization based on if the macro name included the word `materialization`. However, a macro name included the word `materialization` isn't guarnteed to actually have a materialization, and a macro that doesn't have `materialization` in the name isn't guaranteed to not have a materialization. This change is to detect macros with materializations based on the detected block type of the macro. * Add changie doc materialization in macro detection * [CT-2594] Fix serialization of `warn_error_options` on Contexts (#8180) * Add test ensuring `warn_error_options` is dictified in `invocation_args_dict` of contexts * Add dictification specific to `warn_error_options` in `args_to_dict` * Changie doc for serialization changes of warn_error_options * Check for existing_relation immediately prior to renaming (#8193) * add param to control maxBytes for single dbt.log file (#8200) * add param to control maxBytes for single dbt.log file * nits * nits * Update core/dbt/cli/params.py Co-authored-by: Peter Webb <peter.webb@dbtlabs.com> --------- Co-authored-by: Peter Webb <peter.webb@dbtlabs.com> * Update Core for Latest dbt-extractor with Version Parsing (#8206) * Update dbt-extractor requirement, and adjust ref handling accordingly * Add changelog entry. * Update implementation-ticket.yml (#8208) * up timeout (#8218) * Bump mypy from 1.3.0 to 1.4.0 (#7912) * Bump mypy from 1.3.0 to 1.4.0 Bumps [mypy](https://github.com/python/mypy) from 1.3.0 to 1.4.0. - [Commits](https://github.com/python/mypy/compare/v1.3.0...v1.4.0) --- updated-dependencies: - dependency-name: mypy dependency-type: direct:development update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> * Add automated changelog yaml from template for bot PR * add to pre-commit config --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Github Build Bot <buildbot@fishtownanalytics.com> Co-authored-by: Emily Rockman <emily.rockman@dbtlabs.com> * Initialize sqlparse lexer and tweak order of setting compilation fields (#8215) * Split integration tests into parallel groups / jobs (#6346) * Ensure that target_schema from snapshot config is promoted to node level (#8117) * pin upper bound for sqlparse (#8236) * pin upper bound for sqlparse * changelog * [CT-2879] Fix unbound variable error in `checked_agg_time_dimension_for_measure` (#8235) * Fix unbound variable error in `checked_agg_time_dimension_for_measure` * Improve assertion error message in `checked_agg_time_dimension_for_measure` * Add changie doc for checked_agg_time_dimension_for_measure unbound variable fix …

Rebase Jerco's clone code

3056eda

cla-bot bot added the cla:yes label Jun 15, 2023

aranke added 6 commits June 15, 2023 10:48

fix unit tests

f63c379

Delete .user.yml

afdada9

Delete profiles.yml

23b4113

get integration test working

dafb250

Merge branch 'main' into 7258_dbt_clone

b78e837

add state_relation to required_model_keys

31870f9

McKnight-42 reviewed Jun 16, 2023

View reviewed changes

core/dbt/include/global_project/macros/adapters/clone.sql Outdated Show resolved Hide resolved

aranke changed the title ~~Rebase Jerco's clone code~~ dbt clone Jun 16, 2023

McKnight-42 reviewed Jun 16, 2023

View reviewed changes

Move macros into models directory

692e542

aranke requested review from jtcohen6, ChenyuLInx and VersusFacit June 16, 2023 15:53

aranke marked this pull request as ready for review June 16, 2023 15:53

aranke requested review from a team as code owners June 16, 2023 15:53

jtcohen6 mentioned this pull request Jun 16, 2023

New command: dbt clone #7258

Closed

9 tasks

McKnight-42 mentioned this pull request Jun 16, 2023

[Feature] dbt_clone sf implementation dbt-labs/dbt-snowflake#664

Merged

6 tasks

Simplify test

69a3dbd

jtcohen6 reviewed Jun 20, 2023

View reviewed changes

Merge branch 'main' into 7258_dbt_clone

125fb0b

ChenyuLInx reviewed Jun 20, 2023

View reviewed changes

aranke added 2 commits June 20, 2023 16:54

add no state test, move state_relation to maybe_keys

b5dde06

Merge branch 'main' into 7258_dbt_clone

5303346

jtcohen6 reviewed Jun 21, 2023

View reviewed changes

core/dbt/task/clone.py Outdated Show resolved Hide resolved

core/dbt/include/global_project/macros/materializations/models/clone/clone.sql Outdated Show resolved Hide resolved

core/dbt/task/clone.py Show resolved Hide resolved

jtcohen6 reviewed Jun 27, 2023

View reviewed changes

core/dbt/task/clone.py Show resolved Hide resolved

McKnight-42 and others added 4 commits June 27, 2023 10:27

Merge branch 'main' of github.com:dbt-labs/dbt into 7258_dbt_clone

824a2fc

add clean_up method to drop alt schema names after tests run

6a8ecb0

Merge branch '7258_dbt_clone' of github.com:dbt-labs/dbt into 7258_db…

d4f2eca

…t_clone

Add context to comments

5be28bc

aranke mentioned this pull request Jun 27, 2023

[CT-2749] [Spike] Investigate merging merge_from_artifact and add_from_artifact #7965

Closed

McKnight-42 mentioned this pull request Jun 27, 2023

dbt_clone macros, materialization and tests to dbt-spark dbt-labs/dbt-spark#816

Merged

6 tasks

gshank reviewed Jun 28, 2023

View reviewed changes

core/dbt/context/providers.py Outdated Show resolved Hide resolved

aranke added 7 commits June 28, 2023 09:41

Merge branch 'main' into 7258_dbt_clone

18ad7ca

Use relation_name instead of constructing string

6efe6e3

Merge branch 'main' into 7258_dbt_clone

48e01e5

Fix add_from_artifact test

81dad80

Merge branch 'main' into 7258_dbt_clone

d3d3014

remove node.relation_name check

88a726b

add if v.relation_name test

020cde5

gshank approved these changes Jun 28, 2023

View reviewed changes

aranke mentioned this pull request Jun 28, 2023

[CT-2756] [Bug] relation_name is always None on SeedNode #7976

Open

aranke added 3 commits June 28, 2023 14:04

fix seed relation_name bug

e761aa0

Merge branch 'main' into 7258_dbt_clone

4eb7aa4

Skip test_semantic_model_deleted_partial_parsing

cd78b64

aranke merged commit 5c7aa7f into main Jun 29, 2023

aranke deleted the 7258_dbt_clone branch June 29, 2023 00:22

dbeatty10 mentioned this pull request Sep 29, 2023

Prototype: test materialization that allows materialization to be set in the config() #8748

Closed

5 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

`dbt clone` #7881

`dbt clone` #7881

aranke commented Jun 15, 2023 •

edited by jtcohen6

Loading

github-actions bot commented Jun 15, 2023

McKnight-42 commented Jun 16, 2023

McKnight-42 Jun 16, 2023

aranke Jun 16, 2023

McKnight-42 Jun 16, 2023

ChenyuLInx Jun 20, 2023

McKnight-42 Jun 21, 2023 •

edited by aranke

Loading

aranke commented Jun 16, 2023

jtcohen6 left a comment

jtcohen6 Jun 20, 2023

aranke Jun 20, 2023

ChenyuLInx Jun 20, 2023

aranke Jun 20, 2023

jtcohen6 Jun 21, 2023

aranke Jun 21, 2023

ChenyuLInx left a comment

ChenyuLInx Jun 20, 2023

ChenyuLInx Jun 20, 2023

jtcohen6 left a comment

ChenyuLInx commented Jun 26, 2023

aranke commented Jun 26, 2023

aranke commented Jun 26, 2023

jtcohen6 left a comment

aranke commented Jun 27, 2023

McKnight-42 commented Jun 27, 2023

sungchun12 commented Jun 29, 2023

dbt clone #7881

dbt clone #7881

Conversation

aranke commented Jun 15, 2023 • edited by jtcohen6 Loading

Description

Checklist

github-actions bot commented Jun 15, 2023

McKnight-42 commented Jun 16, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

McKnight-42 Jun 21, 2023 • edited by aranke Loading

Choose a reason for hiding this comment

aranke commented Jun 16, 2023

jtcohen6 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ChenyuLInx left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jtcohen6 left a comment

Choose a reason for hiding this comment

ChenyuLInx commented Jun 26, 2023

aranke commented Jun 26, 2023

aranke commented Jun 26, 2023

jtcohen6 left a comment

Choose a reason for hiding this comment

aranke commented Jun 27, 2023

McKnight-42 commented Jun 27, 2023

sungchun12 commented Jun 29, 2023

`dbt clone` #7881

`dbt clone` #7881

aranke commented Jun 15, 2023 •

edited by jtcohen6

Loading

McKnight-42 Jun 21, 2023 •

edited by aranke

Loading