feature/email-event-filter

.buildkite/scripts/run_models.sh

@@ -22,6 +22,5 @@ dbt run --target "$db" --full-refresh
 dbt test --target "$db"
 dbt run --vars '{hubspot_marketing_enabled: true, hubspot_sales_enabled: false}' --target "$db" --full-refresh
 dbt run --vars '{hubspot_marketing_enabled: true, hubspot_contact_merge_audit_enabled: true, hubspot_sales_enabled: false}' --target "$db" --full-refresh
-dbt run --vars '{hubspot_marketing_enabled: false, hubspot_sales_enabled: true}' --target "$db" --full-refresh
-dbt run --vars '{hubspot_marketing_enabled: false, hubspot_sales_enabled: false}' --target "$db" --full-refresh
+dbt run --vars '{hubspot_marketing_enabled: false, hubspot_sales_enabled: true, hubspot__pass_through_all_columns: true, hubspot_using_filtered_events: false}' --target "$db" --full-refresh
 dbt test --target "$db"

CHANGELOG.md

@@ -1,9 +1,15 @@
 # dbt_hubspot v0.9.1
+## 🎉 Feature Updates
+- A new variable was added `hubspot_using_filtered_events` to allow package users to remove filtered email events from the `stg_hubspot__email_event` staging model as well as the relevant downstream reporting models. This is crucial for HubSpot users who greatly take advantage of marking events as filtered in order to provide accurate reporting. ([#104](https://github.com/fivetran/dbt_hubspot/pull/104))
+  - The `hubspot_using_filtered_events` variable is `true` by default. Set the variable to `false` to filter out specified email events in your staging and downstream models. ([#104](https://github.com/fivetran/dbt_hubspot/pull/104))
+
 ## 🪲 Bug Fixes
-[PR #105](https://github.com/fivetran/dbt_hubspot/pull/105) applies the following changes:
-- Introduced new macro `get_email_metrics` to prevent failures that may occur when certain tables are disabled or enabled. This macro removes any metrics that are unavailable from the default `email_metrics` list, ensuring that the models runs smoothly. It's worth noting that you can still manually set the email_metrics list as described in the README's [(Optional) Step 5: Additional configurations](https://github.com/fivetran/dbt_hubspot/#optional-step-5-additional-configurations).
+- Introduced new macro `adjust_email_metrics` to prevent failures that may occur when certain tables are disabled or enabled. This macro removes any metrics that are unavailable from the default `email_metrics` list, ensuring that the models runs smoothly. It's worth noting that you can still manually set the email_metrics list as described in the README's [(Optional) Step 5: Additional configurations](https://github.com/fivetran/dbt_hubspot/#optional-step-5-additional-configurations). ([#105](https://github.com/fivetran/dbt_hubspot/pull/105))
 ## 🚘 Under the Hood
-- Updated the following models to utilize the new macro:
+- The `email_event_data.csv` seed file was updated to include events that are listed as `true` for filtered_events. This is to effectively test the above mentioned feature update. ([#104](https://github.com/fivetran/dbt_hubspot/pull/104))
+- Included `hubspot_using_filtered_events: false` as a variable declared in the final `run_models.sh` step to ensure our integration tests gain coverage over this new feature and variable. ([#104](https://github.com/fivetran/dbt_hubspot/pull/104))
+  - See the source package [CHANGELOG](https://github.com/fivetran/dbt_hubspot_source/blob/main/CHANGELOG.md) for updates made to the staging layer in `dbt_hubspot_source v0.9.1`.
+- Updated the following models to utilize the `adjust_email_metrics` macro ([#105](https://github.com/fivetran/dbt_hubspot/pull/105)):
   - hubspot__contacts
   - hubspot__contact_lists
   - hubspot__email_campaigns

README.md

@@ -191,6 +191,12 @@ vars:
     - name:          "ticket_calculated_field"
       transform_sql: "total_field / other_total_field"
 ```
+### Removing filtered email events
+When leveraging email events, HubSpot customers may take advantage of filtering out specified email events. These filtered email events are present within the `stg_hubspot__email_events` model and are identified by the `is_filtered_event` boolean field. By default, these events are included in the staging and downstream models generated from this package. However, if you wish to remove these filtered events you may do so by setting the `hubspot_using_filtered_events` variable to false. See below for exact configurations you may provide in your `dbt_project.yml` file:
+```yml
+vars:
+  hubspot_using_filtered_events: false # True by default
+```
 
 ### Changing the Build Schema
 By default this package will build the HubSpot staging models within a schema titled (<target_schema> + `_stg_hubspot`) and HubSpot final models within a schema titled (<target_schema> + `hubspot`) in your target database. If this is not where you would like your modeled HubSpot data to be written to, add the following configuration to your root `dbt_project.yml` file:

integration_tests/seeds/email_event_data.csv

@@ -12,9 +12,9 @@ c4337695-9d10-4bd1-89fe-d468407f28e3,2020-03-16 19:01:11.602,2286,,,2020-03-16 1
 a9d93225-97e4-35b0-ae48-cef5c5d96d7a,2020-03-16 19:01:11.598,2286,,,2020-03-16 19:00:50.059,2,,,4727653,2020-03-16 19:00:48.934,c4337695-9d10-4bd1-89fe-d468407f28e3,DELIVERED,,o2wjpPyUfzBfq9OSt9I2NQ==
 d1be99bb-0102-321a-829e-0ea6efbaf477,2020-03-16 19:01:11.610,2286,,,2020-03-16 18:59:22.346,2,,,4727653,2020-03-16 18:59:21.254,38628f09-e146-459f-880b-0159cfe55dd2,DELIVERED,,o2wjpPyUfzBfq9OSt9I2NQ==
 54b21c61-62fb-3998-b434-e5389ecf6195,2020-03-16 19:01:11.612,2286,,,2020-03-16 18:58:06.531,2,,,4727653,2020-03-16 18:58:05.148,63346e66-0578-4fc7-abda-62ba815c7daa,DELIVERED,,o2wjpPyUfzBfq9OSt9I2NQ==
-89067bfe-b6ef-3d09-bc35-cfef5a3e29ae,2020-03-16 19:01:11.596,2286,,,2020-03-16 19:00:50.059,2,,,4727653,2020-03-16 19:00:48.934,c4337695-9d10-4bd1-89fe-d468407f28e3,PROCESSED,,o2wjpPyUfzBfq9OSt9I2NQ==
-7b114d43-154d-31f9-810b-713734c25ea8,2020-03-16 19:01:11.613,2286,,,2020-03-16 18:58:06.531,2,,,4727653,2020-03-16 18:58:05.148,63346e66-0578-4fc7-abda-62ba815c7daa,PROCESSED,,o2wjpPyUfzBfq9OSt9I2NQ==
-ece508cd-a165-3f4d-9317-b642c62e7907,2020-03-16 19:01:11.611,2286,,,2020-03-16 18:59:22.346,2,,,4727653,2020-03-16 18:59:21.254,38628f09-e146-459f-880b-0159cfe55dd2,PROCESSED,,o2wjpPyUfzBfq9OSt9I2NQ==
+89067bfe-b6ef-3d09-bc35-cfef5a3e29ae,2020-03-16 19:01:11.596,2286,,,2020-03-16 19:00:50.059,2,,,4727653,2020-03-16 19:00:48.934,c4337695-9d10-4bd1-89fe-d468407f28e3,PROCESSED,true,o2wjpPyUfzBfq9OSt9I2NQ==
+7b114d43-154d-31f9-810b-713734c25ea8,2020-03-16 19:01:11.613,2286,,,2020-03-16 18:58:06.531,2,,,4727653,2020-03-16 18:58:05.148,63346e66-0578-4fc7-abda-62ba815c7daa,PROCESSED,true,o2wjpPyUfzBfq9OSt9I2NQ==
+ece508cd-a165-3f4d-9317-b642c62e7907,2020-03-16 19:01:11.611,2286,,,2020-03-16 18:59:22.346,2,,,4727653,2020-03-16 18:59:21.254,38628f09-e146-459f-880b-0159cfe55dd2,PROCESSED,true,o2wjpPyUfzBfq9OSt9I2NQ==
 8edbc608-96a9-3318-b34c-806b660ea0ad,2020-04-23 16:25:36.703,2286,,,2020-04-23 16:21:12.604,86863875,,,4727653,2020-04-23 15:55:48.891,e040e9c2-dd3d-4926-9ec3-5ace0834e4fb,OPEN,false,Q5+hDGVJvTBdAVfNDmoNzw==
 5c814bf0-bd56-34b0-aea3-9effa51d3756,2020-05-01 21:10:36.803,2286,,,2020-05-01 21:06:36.623,86863875,,,4727653,2020-04-23 15:56:01.733,acd11faf-e62a-40b3-8ad8-edecbfb7463c,OPEN,false,yyFrTbWxnSxiliuJBnxvHQ==
 a6e576fa-1b65-388d-9573-50ec0a64db3a,2020-05-04 13:35:36.709,2286,,,2020-05-04 13:32:11.729,86863875,,,4727653,2020-04-23 15:55:58.852,ff6dbb56-fd81-432a-a837-07feec296751,OPEN,false,KG2+j8wAdZtIHY97xIGd+Q==

packages.yml

@@ -1,5 +1,10 @@
 packages:
-  - package: fivetran/hubspot_source
-    version: [">=0.9.0", "<0.10.0"]
+  # - package: fivetran/hubspot_source
+  #   version: [">=0.9.0", "<0.10.0"]
+
+  - git: https://github.com/fivetran/dbt_hubspot_source.git
+    revision: feature/email-event-filter
+    warn-unpinned: false 
+
   - package: calogica/dbt_expectations
     version: [">=0.8.0", "<0.9.0"]