V0.3.0 - Metric Function & Overhaul

.github/actions/end-to-end-test/action.yml

@@ -11,6 +11,7 @@ inputs:
  database-adapter-package:
  description: "Name of database adapter to install"
  required: true
+
 runs:
  using: "composite"
  steps:
@@ -19,7 +20,7 @@ runs:
  run: |
  pip install --user --upgrade pip
  pip --version
- pip install --pre ${{ inputs.database-adapter-package }}
+ pip install -r dev-requirements.txt
 
  - name: Setup dbt profile
  shell: bash
@@ -32,5 +33,5 @@ runs:
  run: |
  cd ${{ inputs.dbt-project }}
  dbt deps --target ${{ inputs.dbt-target }}
- dbt seed --target ${{ inputs.dbt-target }}
+ dbt seed --target ${{ inputs.dbt-target }} --full-refresh
  dbt build --target ${{ inputs.dbt-target }} --full-refresh

.github/workflows/ci.yml

@@ -52,7 +52,7 @@ jobs:
  database-adapter-package: dbt-postgres
 
  snowflake:
- needs: postgres
+ # needs: postgres
  runs-on: ubuntu-latest
  steps:
  - name: Check out the repository
@@ -76,7 +76,7 @@ jobs:
  database-adapter-package: dbt-snowflake
 
  redshift:
- needs: postgres
+ # needs: postgres
  runs-on: ubuntu-latest
  steps:
  - name: Check out the repository
@@ -99,7 +99,7 @@ jobs:
  database-adapter-package: dbt-redshift
 
  bigquery:
- needs: postgres
+ # needs: postgres
  runs-on: ubuntu-latest
  steps:
  - name: Check out the repository

.gitignore

@@ -4,3 +4,5 @@ dbt_modules/
 dbt_packages/
 logs/
 .DS_Store
+integration_tests/target
+model_testing/

README.md

@@ -4,44 +4,50 @@
 * [dbt_metrics](#dbt_metrics)
 * [About](#about)
  * [Installation Instructions](#installation-instructions)
- * [A note on refs](#️-a-note-on-refs)
 * [Usage](#usage)
-* [Use cases](#use-cases)
+* [Use cases and examples](#use-cases-and-examples)
+ * [Jaffle Shop Metrics](#jaffle-shop-metrics)
  * [Inside of dbt Models](#inside-of-dbt-models)
  * [Via the interactive dbt server (coming soon)](#via-the-interactive-dbt-server-coming-soon)
 * [Secondary calculations](#secondary-calculations)
  * [Period over Period (<a href="/macros/secondary_calculations/secondary_calculation_period_over_period.sql">source</a>)](#period-over-period-source)
  * [Period to Date (<a href="/macros/secondary_calculations/secondary_calculation_period_to_date.sql">source</a>)](#period-to-date-source)
  * [Rolling (<a href="/macros/secondary_calculations/secondary_calculation_rolling.sql">source</a>)](#rolling-source)
 * [Customisation](#customisation)
+ * [Expression Metrics](#expression-metrics)
+ * [Multiple Metrics](#multiple-metrics)
  * [Where Clauses](#where-clauses)
  * [Calendar](#calendar)
+ * [Dimensions from calendar tables](#dimensions-from-calendar-tables)
  * [Time Grains](#time-grains)
  * [Custom aggregations](#custom-aggregations)
  * [Secondary calculation column aliases](#secondary-calculation-column-aliases)
-* [Experimental behaviour](#-experimental-behaviour)
- * [Dimensions on calendar tables](#dimensions-on-calendar-tables)
 
 <!-- Created by https://github.com/ekalinin/github-markdown-toc -->
-<!-- Added by: runner, at: Tue Jun 7 13:41:32 UTC 2022 -->
+<!-- Added by: runner, at: Tue Jul 12 18:14:25 UTC 2022 -->
 
 <!--te-->
 
 # About
-This dbt package generates queries based on [metrics](https://docs.getdbt.com/docs/building-a-dbt-project/metrics), introduced to dbt Core in v1.0. 
+This dbt package generates queries based on [metrics](https://docs.getdbt.com/docs/building-a-dbt-project/metrics), introduced to dbt Core in v1.0. For more information on metrics, such as available types, properties, and other definition parameters, please reference the documentation linked above.
 
 ## Installation Instructions
 Check [dbt Hub](https://hub.getdbt.com/dbt-labs/metrics/latest/) for the latest installation instructions, or [read the docs](https://docs.getdbt.com/docs/package-management) for more information on installing packages.
 
-## ⚠️ A note on `ref`s
-To enable the dynamic referencing of models necessary for macro queries through the dbt Server, queries generated by this package do not participate in the DAG and `ref`'d nodes will not necessarily be built before they are accessed. Refer to the docs on [forcing dependencies](https://docs.getdbt.com/reference/dbt-jinja-functions/ref#forcing-dependencies) for more details.
+Include in your `package.yml`
+
+```yaml
+packages:
+ - package: dbt-labs/metrics
+ version: 0.3.0
+```
 
 # Usage
 Access metrics [like any other macro](https://docs.getdbt.com/docs/building-a-dbt-project/jinja-macros#using-a-macro-from-a-package): 
 ```sql
 select * 
-from {{ metrics.metric(
- metric_name='new_customers',
+from {{ metrics.calculate(
+ metric('new_customers'),
  grain='week',
  dimensions=['plan', 'country'],
  secondary_calculations=[
@@ -62,13 +68,16 @@ from {{ metrics.metric(
 
 `start_date` and `end_date` are optional. When not provided, the spine will span all dates from oldest to newest in the metric's dataset. This default is likely to be correct in most cases, but you can use the arguments to either narrow the resulting table or expand it (e.g. if there was no new customers until 3 January but you want to include the first two days as well). Both values are inclusive.
 
-# Use cases
+# Use cases and examples
+
+## Jaffle Shop Metrics
+For those curious about how to implement metrics in a dbt project, please reference the [`jaffle_shop_metrics`](https://github.com/dbt-labs/jaffle_shop_metrics). 
 
 ## Inside of dbt Models
 You may want to materialize the results as a fixed table for querying. This is not the way we expect the dbt Metrics layer to add the most value, but is a way to experiment with the project without needing access to the interactive server. 
 
 ## Via the interactive dbt server (coming soon)
-When the [dbt server](https://blog.getdbt.com/licensing-dbt/) is released later in 2022, you will be able to access these macros interactively, without needing to build each variant as a single dbt model. For more information, check out the [keynote presentation from Coalesce 2021](https://www.getdbt.com/coalesce-2021/keynote-the-metrics-system/).
+When [dbt server](https://blog.getdbt.com/licensing-dbt/) is released in late 2022, you will be able to access these macros interactively, without needing to build each variant as a single dbt model. For more information, check out the [keynote presentation from Coalesce 2021](https://www.getdbt.com/coalesce-2021/keynote-the-metrics-system/).
 
 # Secondary calculations
 Secondary calculations are window functions which act on the primary metric. You can use them to compare a metric's value to an earlier period and calculate year-to-date sums or rolling averages.
@@ -117,6 +126,25 @@ Constructor: `metrics.rolling(aggregate, interval [, alias])`
 # Customisation
 Most behaviour in the package can be overridden or customised.
 
+## Expression Metrics 
+Version `0.3.0` of this package, and beyond, offer support for `expression` metrics! More information around this type can be found in the[`metrics` page of dbt docs](https://docs.getdbt.com/docs/building-a-dbt-project/metrics)/.
+
+## Multiple Metrics
+There may be instances where you want to return multiple metrics within a single macro. This is possible by providing a list of metrics instead of a single metric. See example below:
+
+```sql
+ select *
+ from 
+ {{ metrics.calculate(
+ [metric('base_sum_metric'), metric('base_average_metric')], 
+ grain='day', 
+ dimensions=['had_discount']
+ }}
+```
+
+**Note**: The metrics must share the `time_grain` selected in the macro AND the `dimensions` selected in the macro. If these are not shared between the 2+ metrics, this behaviour will fail. Additionally, secondary calculations can be used for multiple metrics but each secondary calculation will be applied against each metric and returned in a field that matches the following pattern: `metric_name_secondary_calculation_alias`.
+
+
 ## Where Clauses
 Sometimes you'll want to see the metric in the context of a particular filter but this filter isn't neccesarily part of the metric definition. In this case, you can use the `where` parameter of the metrics package. It takes a list of `sql` statements and adds them in as filters to the first CTE in the produced SQL. This reduces the load on the query planner to run full table scans and will hopefully improve performance.
 
@@ -137,11 +165,14 @@ vars:
  dbt_metrics_calendar_model: my_custom_calendar
 ```
 
+### Dimensions from calendar tables
+You may want to aggregate metrics by a dimension in your custom calendar table, for example `is_weekend`. You can include this within the list of `dimensions` in the macro call **without** it needing to be defined in the metric definition. The macro will correctly recognize that it is coming from the calendar dimension and treat it accordingly.
+
 ## Time Grains 
 The package protects against nonsensical secondary calculations, such as a month-to-date aggregate of data which has been rolled up to the quarter. If you customise your calendar (for example by adding a [4-5-4 retail calendar](https://calogica.com/sql/dbt/2018/11/15/retail-calendar-in-sql.html) month), you will need to override the [`get_grain_order()`](/macros/secondary_calculations/validate_grain_order.sql) macro. In that case, you might remove `month` and replace it with `month_4_5_4`. All date columns must be prefixed with `date_` in the table. Do not include the prefix when defining your metric, it will be added automatically.
 
 ## Custom aggregations 
-To create a custom primary aggregation (as exposed through the `type` config of a metric), create a macro of the form `metric_my_aggregate(expression)`, then override the [`aggregate_primary_metric()`](/macros/aggregate_primary_metric.sql) macro to add it to the dispatch list. The package also protects against nonsensical secondary calculations such as an average of an average; you will need to override the [`get_metric_allowlist()`](/macros/secondary_calculations/validate_aggregate_coherence.sql) macro to both add your new aggregate to to the existing aggregations' allowlists, and to make an allowlist for your new aggregation:
+To create a custom primary aggregation (as exposed through the `type` config of a metric), create a macro of the form `metric_my_aggregate(expression)`, then override the [`gen_primary_metric_aggregate()`](/macros/gen_primary_metric_aggregate.sql) macro to add it to the dispatch list. The package also protects against nonsensical secondary calculations such as an average of an average; you will need to override the [`get_metric_allowlist()`](/macros/secondary_calculations/validate_aggregate_coherence.sql) macro to both add your new aggregate to to the existing aggregations' allowlists, and to make an allowlist for your new aggregation:
 ```
  {% do return ({
  "average": ['max', 'min'],
@@ -156,38 +187,3 @@ To create a custom secondary aggregation (as exposed through the `secondary_calc
 ## Secondary calculation column aliases
 Aliases can be set for a secondary calculation. If no alias is provided, one will be automatically generated. To modify the existing alias logic, or add support for a custom secondary calculation, override [`generate_secondary_calculation_alias()`](/macros/secondary_calculations/generate_secondary_calculation_alias.sql).
 
-# 🧪 Experimental behaviour 
-⚠️ This behaviour is subject to change in future versions of dbt Core and this package.
-
-## Dimensions on calendar tables
-You may want to aggregate metrics by a dimension in your custom calendar table, for example `is_weekend`. _In addition to_ the primary `dimensions` list, add the following `meta` properties to your metric:
-```yaml
-version: 2 
-metrics:
- - name: new_customers
- [...]
- dimensions: 
- - plan
- - country
-
- meta: 
- dimensions: 
- - type: model
- columns:
- - plan
- - country
- - type: calendar
- columns: 
- - is_weekend
-```
-
-You can then access the additional dimensions as normal:
-```sql
-select * 
-from {{ metrics.metric(
- metric_name='new_customers',
- grain='week',
- dimensions=['plan', 'country', 'is_weekend'],
- secondary_calcs=[]
-) }}
-```

dbt_project.yml

@@ -19,7 +19,7 @@ seed-paths: ["seeds"]
 macro-paths: ["macros"]
 snapshot-paths: ["snapshots"]
 
-require-dbt-version: [">=1.0.0", "<2.0.0"]
+require-dbt-version: [">=1.2.0-a1", "<2.0.0"]
 
 target-path: "target" # directory which will store compiled SQL files
 clean-targets: # directories to be removed by `dbt clean`

dev-requirements.txt

@@ -0,0 +1,4 @@
+git+https://github.com/dbt-labs/dbt-redshift.git@c35865b
+git+https://github.com/dbt-labs/dbt-snowflake.git@a017382
+git+https://github.com/dbt-labs/dbt-bigquery.git@d52222e
+git+https://github.com/dbt-labs/dbt-core.git@0db634d#egg=dbt-core&subdirectory=core

examples/metric_jsonschema_example.json

@@ -0,0 +1,25 @@
+{
+ "title": "dbt_metric_file",
+ "type": "object",
+ "required": ["name","label","timestamp","time_grains","type","sql"],
+ "additionalProperties": false,
+ "properties": {
+ "name": { "type": "string"},
+ "label": { "type": "string"},
+ "timestamp": { "type": "string"},
+ "time_grains": { 
+ "type": "array",
+ "items":{
+ "type":"string"
+ }
+ },
+ "type": { "type": "string"},
+ "sql": { "type": "string"},
+ "dimensions": { 
+ "type": "array",
+ "items":{
+ "type":"string"
+ }
+ }
+ }
+}

integration_tests/.gitignore

@@ -2,3 +2,5 @@
 target/
 dbt_packages/
 logs/
+examples/
+model_testing/

integration_tests/dbt_project.yml

@@ -22,4 +22,5 @@ clean-targets:
  - "logs"
 
 vars:
- dbt_metrics_calendar_model: custom_calendar
+ dbt_metrics_calendar_model: custom_calendar
+ testing: fact_orders

integration_tests/macros/get_nodes_testing.sql

@@ -0,0 +1,8 @@
+{% macro get_nodes_testing()%}
+
+ {%- set metric_relation = metric('total_profit') -%}
+ {{ log("MACRO: Node Unique ID: " ~ metric_relation.unique_id, info=true) }}
+ {{ log("MACRO: Depends on: " ~ metric_relation.depends_on, info=true) }}
+ {{ log("MACRO: Depends on nodes: " ~ metric_relation.depends_on.nodes, info=true) }}
+
+{% endmacro %}

integration_tests/macros/notes.sql

@@ -0,0 +1,53 @@
+ {# Now we see if the node already exists in the metric tree and return that if 
+ it does so that we're not creating duplicates #}
+ {# {%- if metric_tree[node.unique_id] is defined -%}
+
+ {% do return(metric_tree[node.unique_id]) -%}
+
+ {%- endif -%}
+
+ {# {{ log("Inside Macro Depends on: " ~ node.depends_on, info=true) }} #}
+
+
+ {# Here we create two sets, sets being the same as lists but they account for uniqueness. 
+ One is the full set, which contains all of the parent metrics and the other is the leaf
+ set, which we'll use to determine the leaf, or base metrics. #}
+ {%- set full_set = [] -%}
+ {%- set leaf_set = [] -%}
+
+ {# We define parent nodes as being the parent nodes that begin with metric, which lets
+ us filter out model nodes #}
+ {%- set parent_nodes = node.depends_on.nodes -%}
+
+ {%- for parent_node in parent_nodes -%}
+
+ {# We set an if condition based on if parent nodes. If there are none, then this metric
+ is a leaf node and any recursive loop should end #}
+ {%- if parent_nodes -%}
+
+ {# Now we finally recurse through the nodes. We begin by filtering the overall list we
+ recurse through by limiting it to depending on metric nodes and not ALL nodes #}
+ {%- for parent_id in parent_nodes -%}
+
+ {# Then we add the parent_id of the metric to the full set. If it already existed
+ then it won't make an impact but we want to make sure it is represented #}
+ {%- do full_set.update(parent_id) -%}
+
+ {# And here we re-run the current macro but fill in the parent_id so that we loop again
+ with that metric information. You may be wondering, why are you using parent_id? Doesn't 
+ the DAG always go from parent to child? Normally, yes! With this, no! We're reversing the 
+ DAG and going up to parents to find the leaf nodes that are really parent nodes. #}
+ {%- set new_parent = metrics_list[parent_id] -%}
+ {%- do full_set.update(get_metric_parents(new_parent,metrics_list,metric_tree)) -%}
+
+ {%- endfor -%}
+
+ {%- else -%}
+
+ {%- do leaf_set.update(node.unique_id) -%}
+
+ {%- endif -%}
+
+ {%- endfor -%}
+
+ {%- do return(full_set) -%} #}