Merge pull request #112 from fivetran/releases/v0.10.latest

Releases/v0.10.latest

CHANGELOG.md

@@ -1,3 +1,32 @@
+# dbt_hubspot v0.10.0
+## 🚨 Breaking Changes 🚨
+These changes are made breaking due to changes in the source. 
+- Columns `updated_at` and `created_at` were added to the following sources and their corresponding staging models in the [source package](https://github.com/fivetran/dbt_hubspot_source):
+  - `DEAL_PIPELINE`
+  - `DEAL_PIPELINE_STAGE`
+  - `TICKET_PIPELINE`
+  - `TICKET_PIPELINE_STAGE`
+- As a result, the following columns have been added ([#111](https://github.com/fivetran/dbt_hubspot/pull/111)): 
+  - Model `hubspot__deals`:
+    - `deal_pipeline_created_at`
+    - `deal_pipeline_updated_at`
+  - Model `hubspot__deal_stages`:
+    - `deal_pipeline_stage_created_at`
+    - `deal_pipeline_stage_updated_at`
+- Documentation has also been updated with these new columns. ([#111](https://github.com/fivetran/dbt_hubspot/pull/111))
+
+## 🎉 Feature Updates
+- Updated README to include the variables `hubspot_ticket_deal_enabled` and `hubspot_owner_enabled`. ([#111](https://github.com/fivetran/dbt_hubspot/pull/111))
+
+## 🚘 Under the Hood
+- Modified the `unnest` logic in the `merge_contacts` macro for **Redshift** users to reduce runtime of the `int_hubspot__contact_merge_adjust` model. ([#110](https://github.com/fivetran/dbt_hubspot/pull/110))
+- Updated seed data for testing newly added columns. ([#111](https://github.com/fivetran/dbt_hubspot/pull/111))
+
+## Contributors
+- @kcraig-ats ([#110](https://github.com/fivetran/dbt_hubspot/pull/110))
+
+See upstream `hubspot_source` release notes [here](https://github.com/fivetran/dbt_hubspot_source/blob/main/CHANGELOG.md).
+
 # dbt_hubspot v0.9.1
 
 ## 🎉 Feature Updates

README.md

@@ -63,7 +63,7 @@ Include the following hubspot package version in your `packages.yml` file:
 ```yaml
 packages:
   - package: fivetran/hubspot
-    version: [">=0.9.0", "<0.10.0"] # we recommend using ranges to capture non-breaking changes automatically
+    version: [">=0.10.0", "<0.11.0"] # we recommend using ranges to capture non-breaking changes automatically
 
 ```
 Do **NOT** include the `hubspot_source` package in this file. The transformation package itself has a dependency on it and will install the source package as well.
@@ -86,7 +86,7 @@ vars:
 ```
 
 ## Step 4: Disable models for non-existent sources
-When setting up your Hubspot connection in Fivetran, it is possible that not every table this package expects will be synced. This can occur because you either don't use that functionality in Hubspot or have actively decided to not sync some tables. In order to disable the relevant functionality in the package, you will need to add the relevant variables. By default, all variables are assumed to be `true` (with exception of `hubspot_service_enabled` and `hubspot_contact_merge_audit_enabled`). You only need to add variables within your root `dbt_project.yml` for the tables you would like to disable or enable respectively:
+When setting up your Hubspot connection in Fivetran, it is possible that not every table this package expects will be synced. This can occur because you either don't use that functionality in Hubspot or have actively decided to not sync some tables. In order to disable the relevant functionality in the package, you will need to add the relevant variables. By default, all variables are assumed to be `true` (with exception of `hubspot_service_enabled`, `hubspot_ticket_deal_enabled`, and `hubspot_contact_merge_audit_enabled`). You only need to add variables within your root `dbt_project.yml` for the tables you would like to disable or enable respectively:
 
 ```yml
 vars:
@@ -130,9 +130,11 @@ vars:
   hubspot_engagement_meeting_enabled: false
   hubspot_engagement_note_enabled: false
   hubspot_engagement_task_enabled: false
-
+  hubspot_owner_enabled: false
+  
   # Service
   hubspot_service_enabled: true                           # Enables all service models
+  hubspot_ticket_deal_enabled: true
 ```
 ## (Optional) Step 5: Additional configurations
 <details><summary>Expand for configurations</summary>
@@ -244,7 +246,7 @@ This dbt package is dependent on the following dbt packages. Please be aware tha
 ```yml
 packages:
     - package: fivetran/hubspot_source
-      version: [">=0.9.0", "<0.10.0"]
+      version: [">=0.10.0", "<0.11.0"]
 
     - package: fivetran/fivetran_utils
       version: [">=0.4.0", "<0.5.0"]

dbt_project.yml

@@ -1,5 +1,6 @@
 name: 'hubspot'
-version: '0.9.1'
+version: '0.10.0'
+
 config-version: 2
 require-dbt-version: [">=1.3.0", "<2.0.0"]
 vars:

integration_tests/.gitignore

@@ -2,3 +2,5 @@
 target/
 dbt_modules/
 logs/
+.DS_Store
+dbt_packages/

integration_tests/ci/sample.profiles.yml

@@ -16,13 +16,13 @@ integration_tests:
       pass: "{{ env_var('CI_REDSHIFT_DBT_PASS') }}"
       dbname: "{{ env_var('CI_REDSHIFT_DBT_DBNAME') }}"
       port: 5439
-      schema: hubspot_integration_tests_6
+      schema: hubspot_integration_tests_7
       threads: 8
     bigquery:
       type: bigquery
       method: service-account-json
       project: 'dbt-package-testing'
-      schema: hubspot_integration_tests_6
+      schema: hubspot_integration_tests_7
       threads: 8
       keyfile_json: "{{ env_var('GCLOUD_SERVICE_KEY') | as_native }}"
     snowflake:
@@ -33,7 +33,7 @@ integration_tests:
       role: "{{ env_var('CI_SNOWFLAKE_DBT_ROLE') }}"
       database: "{{ env_var('CI_SNOWFLAKE_DBT_DATABASE') }}"
       warehouse: "{{ env_var('CI_SNOWFLAKE_DBT_WAREHOUSE') }}"
-      schema: hubspot_integration_tests_6
+      schema: hubspot_integration_tests_7
       threads: 8
     postgres:
       type: postgres
@@ -42,13 +42,13 @@ integration_tests:
       pass: "{{ env_var('CI_POSTGRES_DBT_PASS') }}"
       dbname: "{{ env_var('CI_POSTGRES_DBT_DBNAME') }}"
       port: 5432
-      schema: hubspot_integration_tests_6
+      schema: hubspot_integration_tests_7
       threads: 8
     databricks:
       catalog: "{{ env_var('CI_DATABRICKS_DBT_CATALOG') }}"
       host: "{{ env_var('CI_DATABRICKS_DBT_HOST') }}"
       http_path: "{{ env_var('CI_DATABRICKS_DBT_HTTP_PATH') }}"
-      schema: hubspot_integration_tests_6
-      threads: 2
+      schema: hubspot_integration_tests_7
+      threads: 8
       token: "{{ env_var('CI_DATABRICKS_DBT_TOKEN') }}"
       type: databricks

integration_tests/dbt_project.yml

@@ -1,9 +1,10 @@
 name: 'hubspot_integration_tests'
-version: '0.9.1'
+version: '0.10.0'
+
 profile: 'integration_tests'
 config-version: 2
 vars:
-  hubspot_schema: hubspot_integration_tests_6
+  hubspot_schema: hubspot_integration_tests_7
   hubspot_source:
     hubspot_service_enabled: true
     hubspot_company_property_history_identifier: "company_property_history_data"
@@ -146,4 +147,4 @@ seeds:
 
 dispatch:
   - macro_namespace: dbt_utils
-    search_order: ['spark_utils', 'dbt_utils']
+    search_order: ['spark_utils', 'dbt_utils']

integration_tests/seeds/deal_company_data.csv

@@ -1,3 +1,3 @@
-company_id,deal_id
-12345,6789
-56789,1234
+company_id,deal_id,type_id
+12345,6789,101112
+56789,1234,101112

integration_tests/seeds/deal_contact_data.csv

@@ -1,12 +1,12 @@
-contact_id,deal_id,_fivetran_synced
-1499466,86133127,2022-04-14 4:30:42
-1499466,86134193,2022-04-14 4:45:43
-1499368,86014324,2022-04-14 5:30:45
-1499468,86136486,2022-04-14 6:30:45
-1499469,86139887,2022-04-14 7:45:44
-1499469,86142279,2022-04-14 8:30:44
-1499596,86324169,2022-04-16 5:00:42
-1499595,86318106,2022-04-16 4:45:44
-1499597,86328114,2022-04-16 6:45:43
-1499597,86325643,2022-04-16 6:45:43
-1499598,86326974,2022-04-16 8:00:44
+contact_id,deal_id,_fivetran_synced,type_id
+1499466,86133127,2022-04-14 4:30:42,101112
+1499466,86134193,2022-04-14 4:45:43,123456
+1499368,86014324,2022-04-14 5:30:45,567891
+1499468,86136486,2022-04-14 6:30:45,101112
+1499469,86139887,2022-04-14 7:45:44,123456
+1499469,86142279,2022-04-14 8:30:44,101112
+1499596,86324169,2022-04-16 5:00:42,123456
+1499595,86318106,2022-04-16 4:45:44,567891
+1499597,86328114,2022-04-16 6:45:43,101112
+1499597,86325643,2022-04-16 6:45:43,567891
+1499598,86326974,2022-04-16 8:00:44,123456

integration_tests/seeds/deal_pipeline_data.csv

@@ -1,7 +1,7 @@
-pipeline_id,_fivetran_deleted,_fivetran_synced,active,display_order,label
-mp_pipeline_monitoring,false,2020-07-09 15:25:37.051,true,0,83t7Vz1pOW/Fx81JI4o+4g==
-default,false,2020-07-09 15:25:37.053,true,0,FioJAMNdGX8U65mUGCkDwA==
-mp_pipeline_service,false,2020-07-09 15:25:37.024,true,0,eDLdK7glKfd0VRV0G5OcLg==
-mp_pipeline_archived,false,2020-07-09 15:25:37.054,true,3,pHm8C3GMK7LnsWuMcT6sgw==
-mp_pipeline_sales,false,2020-07-09 15:25:37.049,true,4,4sZLZi4xR6kmTh+A8waTCw==
-mp_pipeline_operations,false,2020-07-09 15:25:37.052,true,5,r2LpcB5mggWrgSrGkJERpg==
+pipeline_id,_fivetran_deleted,_fivetran_synced,active,display_order,label,created_at,updated_at
+mp_pipeline_monitoring,FALSE,2020-07-09 15:25:37.051,TRUE,0,83t7Vz1pOW/Fx81JI4o+4g==,2020-07-09 15:25:37.051,2020-07-09 15:25:37.051
+default,FALSE,2020-07-09 15:25:37.053,TRUE,0,FioJAMNdGX8U65mUGCkDwA==,2020-07-09 15:25:37.053,2020-07-09 15:25:37.053
+mp_pipeline_service,FALSE,2020-07-09 15:25:37.024,TRUE,0,eDLdK7glKfd0VRV0G5OcLg==,2020-07-09 15:25:37.024,2020-07-09 15:25:37.024
+mp_pipeline_archived,FALSE,2020-07-09 15:25:37.054,TRUE,3,pHm8C3GMK7LnsWuMcT6sgw==,2020-07-09 15:25:37.054,2020-07-09 15:25:37.054
+mp_pipeline_sales,FALSE,2020-07-09 15:25:37.049,TRUE,4,4sZLZi4xR6kmTh+A8waTCw==,2020-07-09 15:25:37.049,2020-07-09 15:25:37.049
+mp_pipeline_operations,FALSE,2020-07-09 15:25:37.052,TRUE,5,r2LpcB5mggWrgSrGkJERpg==,2020-07-09 15:25:37.052,2020-07-09 15:25:37.052