Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Sync #69

Merged
merged 308 commits into from
Sep 12, 2023
Merged

Sync #69

merged 308 commits into from
Sep 12, 2023

Conversation

leahwicz
Copy link
Owner

resolves #

Description

Checklist

mikealfare and others added 30 commits May 19, 2023 17:04
* remove DelayedFileHandler

* Changelog

* set_path to no-op

* more no-ops for rpc

* Clearer comments
* CT-2461: Work toward model deprecation

* CT-2461: Remove unneeded conversions

* CT-2461: Fix up unit tests for new fields, correct a couple oversights

* CT-2461: Remaining implementation and tests for model/ref deprecation warnings

* CT-2461: Changelog entry for deprecation warnings

* CT-2461: Refine datetime handling and tests

* CT-2461: Fix up unit test data

* CT-2461: Fix some more unit test data.

* CT-2461: Fix merge issues

* CT-2461: Code review items.

* CT-2461: Improve version -> str conversion
#7609)

* Honor `--skip-profile-setup` parameter when inside an existing project

* Use project name as the profile name

* Use separate file connections for reading and writing

* Raise a custom exception when no adapters are installed

* Test skipping interactive profile setup when inside a dbt project

* Replace `assert_not_called()` since it does not work

* Verbose CLI argument for skipping profile setup

* Use separate file connections for reading and writing

* Check empty list in a Pythonic manner
* Include null checks in utils test base

* Add tests for the schema test

* Add tests for this macro

---------

Co-authored-by: Mila Page <versusfacit@users.noreply.github.com>
* Bump ubuntu from 23.04 to 23.10

Bumps ubuntu from 23.04 to 23.10.

---
updated-dependencies:
- dependency-name: ubuntu
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>

* Add automated changelog yaml from template for bot PR

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Github Build Bot <buildbot@fishtownanalytics.com>
* adding performance baseline for 1.3.4

* Add newline

---------

Co-authored-by: Github Build Bot <buildbot@fishtownanalytics.com>
Co-authored-by: leahwicz <60146280+leahwicz@users.noreply.github.com>
* adding performance baseline for 1.4.1

* Add newline

---------

Co-authored-by: Github Build Bot <buildbot@fishtownanalytics.com>
Co-authored-by: leahwicz <60146280+leahwicz@users.noreply.github.com>
* adding performance baseline for 1.3.0

* Add newline

---------

Co-authored-by: Github Build Bot <buildbot@fishtownanalytics.com>
Co-authored-by: leahwicz <60146280+leahwicz@users.noreply.github.com>
* adding performance baseline for 1.4.6

* Add newline

---------

Co-authored-by: Github Build Bot <buildbot@fishtownanalytics.com>
Co-authored-by: leahwicz <60146280+leahwicz@users.noreply.github.com>
Consolidating to remove Language and Execution and instead default to the `core-team`
* Test table/view/column-level comments with `%` symbol

* Test docs block with `%` symbol

* Changelog entry
* fix overlooked node12 case with abandonded marketplace action

* update slack notification

* remove spaces per formatting

* replace with cli dispatch

* move conditional

* add explicit token, temp comment out slack

* add checkout

* checkout teh right branch

* switch to PAT

* add back repo checkout

* manually check workflow status

* fix notification

* swap to reusable workflow

* fix path

* swap permissions

* remove trigger

* fix secrets

* point to main
* adding performance baseline for 1.2.0

* Adding newline

---------

Co-authored-by: Github Build Bot <buildbot@fishtownanalytics.com>
Co-authored-by: leahwicz <60146280+leahwicz@users.noreply.github.com>
* CT-2478: Template rendering optimization

* CT-2478: Fix type annotation, and accomodate non-string unit test cases.
* add ability to select models by access

* changie

* Update core/dbt/graph/selector_methods.py
* Fix doc link in selector.py

* Ran changie to modify changelog entry
MichelleArk and others added 28 commits August 23, 2023 14:06
…, fix msg format (#8451)

* first pass, tests need updates

* update proto defn

* fixing tests

* more test fixes

* finish fixing test file

* reformat the message

* formatting messages

* changelog

* add event to unit test

* feedback on message structure

* WIP

* fix up event to take in all fields

* fix test
* Test `SemanticModel` satisfies protocol when none of it's `Optionals` are specified

* Add tests ensuring SourceFileMetadata and FileSlice satisfiy DSI protocols

* Add test asserting Defaults obj satisfies protocol

* Add test asserting SemanticModel with optionals specified satisfies protocol

* Split dimension protocol satisfaction tests into with and without optionals

* Simplify DSI Protocol import strategy in protocol satisfaction tests

* Add test asserting DimensionValidtyParams satisfies protocol

* Add test asserting DimensionTypeParams satisfies protocol

* Split entity protocol satisfaction tests into with and without optionals

* Split measure protocol satisfication tests and add measure aggregation params satisficaition test

* Split metric protocol satisfaction test into optional specified an unspecified

Additionally, create where_filter pytest fixture

* Improve protocol satisfaction tests for MetricTypeParams and sub protocols

Specifically we added/improved protocol satisfaction tests for
- MetricTypeParams
- MetricInput
- MetricInputMeasure
- MetricTimeWindow
* Add compiled node properties to run_results.json

* Include compiled-node attributes in run_results.json

* Fix typo

* Bump schema version of run_results

* Fix test assertions

* Update expected run_results to reflect new attributes

* Code review changes

* Fix mypy warnings for ManifestLoader.load() (#8443)

* revert python version for docker images (#8445)

* revert python version for docker images

* add comment to not update python version, update changelog

* Bumping version to 1.7.0b1 and generate changelog

* [CT-3013]  Fix parsing of `window_groupings` (#8454)

* Update semantic model parsing tests to check measure non_additive_dimension spec

* Make `window_groupings` default to empty list if not specified on `non_additive_dimension`

* Add changie doc for `window_groupings`  parsing fix

* update `Number` class to handle integer values (#8306)

* add show test for json data

* oh changie my changie

* revert unecessary cahnge to fixture

* keep decimal class for precision methods, but return __int__ value

* jerco updates

* update integer type

* update other tests

* Update .changes/unreleased/Fixes-20230803-093502.yaml

---------

Co-authored-by: Emily Rockman <emily.rockman@dbtlabs.com>

* Improve docker image README (#8212)

* Improve docker image README

- Fix unnecessary/missing newline escapes
- Remove double whitespace between parameters
- 2-space indent for extra lines in image build commands

* Add changelog entry for #8212

* ADAP-814: Refactor prep for MV updates (#8459)

* apply reformatting changes only for #8449
* add logging back to get_create_materialized_view_as_sql
* changie

* swap trigger (#8463)

* update the implementation template (#8466)

* update the implementation template

* add colon

* Split tests into classes (#8474)

* add flaky decorator

* split up tests into classes

* revert update agate for int (#8478)

* updated typing and methods to meet mypy standards (#8485)

* Convert error to conditional warning for unversioned contracted model, fix msg format (#8451)

* first pass, tests need updates

* update proto defn

* fixing tests

* more test fixes

* finish fixing test file

* reformat the message

* formatting messages

* changelog

* add event to unit test

* feedback on message structure

* WIP

* fix up event to take in all fields

* fix test

* Fix ambiguous reference error for duplicate model names across packages with tests (#8488)

* Safely remove external nodes from manifest (#8495)

* [CT-2840] Improved semantic layer protocol satisfaction tests (#8456)

* Test `SemanticModel` satisfies protocol when none of it's `Optionals` are specified

* Add tests ensuring SourceFileMetadata and FileSlice satisfiy DSI protocols

* Add test asserting Defaults obj satisfies protocol

* Add test asserting SemanticModel with optionals specified satisfies protocol

* Split dimension protocol satisfaction tests into with and without optionals

* Simplify DSI Protocol import strategy in protocol satisfaction tests

* Add test asserting DimensionValidtyParams satisfies protocol

* Add test asserting DimensionTypeParams satisfies protocol

* Split entity protocol satisfaction tests into with and without optionals

* Split measure protocol satisfication tests and add measure aggregation params satisficaition test

* Split metric protocol satisfaction test into optional specified an unspecified

Additionally, create where_filter pytest fixture

* Improve protocol satisfaction tests for MetricTypeParams and sub protocols

Specifically we added/improved protocol satisfaction tests for
- MetricTypeParams
- MetricInput
- MetricInputMeasure
- MetricTimeWindow

* Convert to using mashumaro jsonschema with acceptable performance (#8437)

* Regenerate run_results schema after merging in changes from main.

---------

Co-authored-by: Gerda Shank <gerda@dbtlabs.com>
Co-authored-by: Matthew McKnight <91097623+McKnight-42@users.noreply.github.com>
Co-authored-by: Github Build Bot <buildbot@fishtownanalytics.com>
Co-authored-by: Quigley Malcolm <QMalcolm@users.noreply.github.com>
Co-authored-by: dave-connors-3 <73915542+dave-connors-3@users.noreply.github.com>
Co-authored-by: Emily Rockman <emily.rockman@dbtlabs.com>
Co-authored-by: Jaime Martínez Rincón <jaime@jamezrin.name>
Co-authored-by: Mike Alfare <13974384+mikealfare@users.noreply.github.com>
Co-authored-by: Michelle Ark <MichelleArk@users.noreply.github.com>
* WIP

* WIP

* get group and enabled added

* changelog

* cleanup

* getting measure lookup working

* missed file

* get project level working

* fix last test

* add groups to config tests

* more group tests

* fix path

* clean up manifest.py

* update error message

* fix test assert

* remove extra check

* resolve conflicts in manaifest

* update manifest

* resolve conflict

* add alias
* Improve typing of `ContextMember` functions

* Improve typing of `Var` functions

* Improve typing of `ContextMeta.__new__`

* Improve typing `BaseContext` and functions

In addition to just adding parameter typing and return typing to
`BaseContext` functions. We also declared `_context_members_` and
`_context_attrs_` as properites of `BaseContext` this was necessary
because they're being accessed in the classes functions. However,
because they were being indirectly instantiated by the metaclass
`ContextMeta`, the properties weren't actually known to exist. By
adding declaring the properties on the `BaseContext`, we let mypy
know they exist.

* Remove bare `invocations` of `@contextmember` and `@contextproperty`, and add typing to them

Previously `contextmember` and `contextproperty` were 2-in-1 decorators.
This meant they could be invoked either as `@contextmember` or
`@contextmember('some_string')`. This was fine until we wanted to return
typing to the functions. In the instance where the bare decorator was used
(i.e. no `(...)` were present) an object was expected to be returned. However
in the instance where parameters were passed on the invocation, a callable
was expected to be returned. Putting a union of both in the return type
made the invocations complain about each others' return type. To get around this
we've dropped the bare invocation as acceptable. The parenthesis are now always
required, but passing a string in them is optional.
…ews and vice versa (#8449)

* first draft of adding in table - materialized view swap
* table/view/materialized view can all replace each other
* update renameable relations to a config
* migrate relations macros from `macros/adapters/relations` to `macros/relations` so that generics are close to the relation specific macros that they reference; also aligns with adapter macro files structure, to look more familiar
* move drop macro to drop macro file
* align the behavior of get_drop_sql and drop_relation, adopt existing default from drop_relation
* add explicit ddl for drop statements instead of inheriting the default from dbt-core
* update replace macro dependent macros to align with naming standards
* update type for mashumaro, update related test
* first pass at adding dbt-cloud config

* changelog

* fix test, add direct validation
* Fix #8544: Parse the correct schema version from manifest

* add changie

* add comment
* update `Number` class to handle integer values (#8306)

* add show test for json data

* oh changie my changie

* revert unecessary cahnge to fixture

* keep decimal class for precision methods, but return __int__ value

* jerco updates

* update integer type

* update other tests

* Update .changes/unreleased/Fixes-20230803-093502.yaml

---------

Co-authored-by: Emily Rockman <emily.rockman@dbtlabs.com>

* account for integer vs number on table merges

* add tests for combining number with integer.

* add unit test when nulls are added

* cant none as an Integer

* fix null tests

---------

Co-authored-by: dave-connors-3 <73915542+dave-connors-3@users.noreply.github.com>
Co-authored-by: Dave Connors <dave.connors@fishtownanalytics.com>
* Add docstrings to `contracts/graph/metrics.py` functions to document what they do

Used [#5607](#5607)
for context on what the functions should do.

* Add typing to `reverse_dag_parsing` and update function to work on 1.6+ metrics

* Add typing to `parent_metrics` and `parent_metrics_names`

* Add typing to `base_metric_dependency` and `derived_metric_dependency` and update functions to work on 1.6+ metrics

* Simplify implementations of `basic_metric_dependency` and `derived_metric_dependnecy`

* Add typing to `ResolvedMetricReference` initialization

* Add typing to `derived_metric_dependency_graph`

* Simplify conditional controls in `ResolvedMetricReference` functions

The functions in `ResolvedMetricReference` use `manifest.metric.get(...)`
which will only return either a `Metric` or `None`, never a different
node type. Thus we don't need to check that the returned metric is
a metric.

* Don't recurse on over `depends_on` for non-derived metrics in `reverse_dag_parsing`

The function `reverse_dag_parsing` only cares about derived metrics,
that is metrics that depend on other metrics. Metrics only depend on
other metrics if they are one of the `DERIVED_METRICS` types. Thus
doing a recursive call to `reverse_dag_parsing` for non `DERIVED_METRICS`
types is unnecessary. Previously we were iterating over a metric's
`depends_on` property regardless of whether the metric was a `DERIVED_METRICS`
type. Now we only do this work if the metric is of a `DERIVED_METRICS`
type.

* Simplify `parent_metrics_names` by having it call `parent_metrics`

* Unskip `TestMetricHelperFunctions.test_derived_metric` and update fixture setup

* Add changie doc for metric helper function updates

* Get manifest in `test_derived_metric` from the parse dbt_run invocation

* Remove `Relation` a intiatlization attribute for `ResolvedMetricReference`

* Add return typing to class `__` functions of `ResolvedMetricReference`

* Move from `manifest.metrics.get` to `manifest.expect` in metric helpers

Previously with `manifest.metrics.get` we were just skipping when `None`
was returned. Getting `None` back was expected in that `parent_unique_id`s
that didn't belong to metrics should return `None` when calling
`manifest.metrics.get`, and these are fine to skip. However, there's
an edgecase where a `parent_unique_id` is supposed to be a metric, but
isn't found, thus returning `None`. How likely this edge case could
get hit, I'm not sure, but it's a possible edge case. Using `manifest.metrics.get`
it we can't actually tell if we're in the edge case or not. By moving
to `manifest.expect` we get the error handling built in, and the only
trade off is that we need to change our conditional to skip returned
nodes that aren't metrics.
* move config changes into alter.sql in alignment with other adapters
* move shared relations macros to relations root
* move single models files to models root
* add table to replace
* move create file into relation directory
* implement replace for postgres
* move column specific macros into column directory
* add unit test for can_be_replaced
* update renameable_relations and replaceable_relations to frozensets to set defaults
* fixed tests for new defaults
@leahwicz leahwicz merged commit 6fe80d3 into leahwicz:main Sep 12, 2023
45 of 47 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.