Service hub updates

README.md

@@ -1,6 +1,6 @@
 # Hubspot (Source)
 
-This package models Hubspot data from [Fivetran's connector](https://fivetran.com/docs/applications/hubspot). It uses data in the format described by the [marketing](https://docs.google.com/presentation/d/1hrPp310SNK2qyESCV_g_JFx_Knm1MwB467wN3dEgy0M/edit#slide=id.g244d368397_0_1) and [sales](https://docs.google.com/presentation/d/1KABQnt8WmtZe7u5l7WFUoPIsWzv63P9gsWF79XGLoZE/edit#slide=id.g244d368397_0_1) ERDs.
+This package models Hubspot data from [Fivetran's connector](https://fivetran.com/docs/applications/hubspot). It uses data in the format described by the [marketing](https://fivetran.com/docs/applications/hubspot#schemainformation), [sales](https://fivetran.com/docs/applications/hubspot#crmandsaleshubschema) and [service](https://fivetran.com/docs/applications/hubspot#servicehubschema) ERDs.
 
 This package enriches your Fivetran data by doing the following:
 
@@ -33,12 +33,12 @@ config-version: 2
 vars:
   hubspot_source:
     hubspot_database: your_database_name
-    hubspot_schema: your_schema_name 
+    hubspot_schema: your_schema_name
 ```
 
 ### Disabling models
 
-When setting up your Hubspot connection in Fivetran, it is possible that not every table this package expects will be synced. This can occur because you either don't use that functionality in Hubspot or have actively decided to not sync some tables. In order to disable the relevant functionality in the package, you will need to add the relevant variables. By default, all variables are assumed to be `true`. You only need to add variables for the tables you would like to disable:  
+When setting up your Hubspot connection in Fivetran, it is possible that not every table this package expects will be synced. This can occur because you either don't use that functionality in Hubspot or have actively decided to not sync some tables. In order to disable the relevant functionality in the package, you will need to add the relevant variables. By default, all variables are assumed to be `true` (with exception of `hubspot_service_enabled`). You only need to add variables for the tables you would like to disable or enable respectively:
 
 ```yml
 # dbt_project.yml
@@ -72,40 +72,83 @@ vars:
 
     # Sales
 
-    hubspot_sales_enabled: false                            # Disables all sales models
-    hubspot_company_enabled: false
-    hubspot_deal_enabled: false
-    hubspot_engagement_enabled: false                       # Disables all engagement models and functionality
-    hubspot_engagement_contact_enabled: false
-    hubspot_engagement_company_enabled: false
-    hubspot_engagement_deal_enabled: false
-    hubspot_engagement_calls_enabled: false
-    hubspot_engagement_emails_enabled: false
-    hubspot_engagement_meetings_enabled: false
-    hubspot_engagement_notes_enabled: false
-    hubspot_engagement_tasks_enabled: false
+    hubspot_sales_enabled:                      false       # Disables all sales models
+    hubspot_company_enabled:                    false
+    hubspot_deal_enabled:                       false
+    hubspot_engagement_enabled:                 false       # Disables all engagement models and functionality
+    hubspot_engagement_contact_enabled:         false
+    hubspot_engagement_company_enabled:         false
+    hubspot_engagement_deal_enabled:            false
+    hubspot_engagement_calls_enabled:           false
+    hubspot_engagement_emails_enabled:          false
+    hubspot_engagement_meetings_enabled:        false
+    hubspot_engagement_notes_enabled:           false
+    hubspot_engagement_tasks_enabled:           false
+
+    # Service
+    hubspot_service_enabled:                    true        # Enables all service models
 ```
+
+
+### Passthrough Columns
+Additionally, this package includes all source columns defined in the macros folder. We highly recommend including custom fields in this package as models now only bring in a few fields for the `company`, `contact`, `deal`, and `ticket` tables. You can add more columns using our pass-through column variables. These variables allow for the pass-through fields to be aliased (`alias`) and casted (`transform_sql`) if desired, but not required. Datatype casting is configured via a sql snippet within the `transform_sql` key. You may add the desired sql while omitting the `as field_name` at the end and your custom pass-though fields will be casted accordingly. Use the below format for declaring the respective pass-through variables.
+
+```yml
+# dbt_project.yml
+
+...
+vars:
+  hubspot__deal_pass_through_columns:
+    - name:           "property_field_new_id"
+      alias:          "new_name_for_this_field_id"
+      transform_sql:  "cast(new_name_for_this_field as int64)"
+    - name:           "this_other_field"
+      alias:          "new_field_name"
+  hubspot__contact_pass_through_columns:
+    - name:           "wow_i_can_add_all_my_custom_fields"
+      alias:          "best_field"
+  hubspot__company_pass_through_columns:
+    - name:           "this_is_radical"
+      alias:          "radical_field"
+      transform_sql:  "cast(radical_field as string)"
+  hubspot__ticket_pass_through_columns:
+    - name:           "property_mmm"
+      alias:          "mmm"
+    - name:           "property_bop"
+      alias:          "bop"
+
+```
+
 ### Changing the Build Schema
-By default this package will build the HubSpot staging models within a schema titled (<target_schema> + `_stg_hubspot`) in your target database. If this is not where you would like your modeled HubSpot data to be written to, add the following configuration to your `dbt_project.yml` file:
+By default this package will build the HubSpot staging models within a schema titled (<target_schema> + `_stg_hubspot`). If this is not where you would like your HubSpot staging models to be written to, add the following configuration to your `dbt_project.yml` file:
 
 ```yml
 # dbt_project.yml
 
 ...
 models:
-    hubspot_source:
-      +schema: my_new_schema_name # leave blank for just the target_schema
+  hubspot_source:
+    +schema: my_new_staging_models_schema # leave blank for just the target_schema
 ```
+
+*Read more about using custom schemas in dbt [here](https://docs.getdbt.com/docs/building-a-dbt-project/building-models/using-custom-schemas).*
+
 ## Contributions
 
 Additional contributions to this package are very welcome! Please create issues
 or open PRs against `master`. Check out 
 [this post](https://discourse.getdbt.com/t/contributing-to-a-dbt-package/657) 
 on the best workflow for contributing to a package.
 
-## Resources:
+## Database support
+This package has been tested on BigQuery, Snowflake, and Redshift.
+
+### Resources:
+- Provide [feedback](https://www.surveymonkey.com/r/DQ7K7WW) on our existing dbt packages or what you'd like to see next
+- Have questions, feedback, or need help? Book a time during our office hours [using Calendly](https://calendly.com/fivetran-solutions-team/fivetran-solutions-team-office-hours) or email us at solutions@fivetran.com
 - Find all of Fivetran's pre-built dbt packages in our [dbt hub](https://hub.getdbt.com/fivetran/)
-- Learn more about Fivetran [in the Fivetran docs](https://fivetran.com/docs)
+- Learn how to orchestrate [dbt transformations with Fivetran](https://fivetran.com/docs/transformations/dbt)
+- Learn more about Fivetran overall [in our docs](https://fivetran.com/docs)
 - Check out [Fivetran's blog](https://fivetran.com/blog)
 - Learn more about dbt [in the dbt docs](https://docs.getdbt.com/docs/introduction)
 - Check out [Discourse](https://discourse.getdbt.com/) for commonly asked questions and answers

dbt_project.yml

@@ -3,46 +3,64 @@ version: '0.3.0'
 config-version: 2
 require-dbt-version: [">=0.18.0", "<0.20.0"]
 
-vars:
-  hubspot_source: 
-    company:                    "{{ source('hubspot','company') }}" 
-    company_property_history:   "{{ source('hubspot','company_property_history') }}" 
-    contact:                    "{{ source('hubspot','contact') }}" 
-    contact_list:               "{{ source('hubspot','contact_list') }}" 
-    contact_list_member:        "{{ source('hubspot','contact_list_member') }}" 
-    contact_property_history:   "{{ source('hubspot','contact_property_history') }}" 
-    deal:                       "{{ source('hubspot','deal') }}" 
-    deal_company:               "{{ source('hubspot','deal_company') }}" 
-    deal_pipeline:              "{{ source('hubspot','deal_pipeline') }}" 
-    deal_pipeline_stage:        "{{ source('hubspot','deal_pipeline_stage') }}" 
-    deal_property_history:      "{{ source('hubspot','deal_property_history') }}" 
-    email_campaign:             "{{ source('hubspot','email_campaign') }}" 
-    email_event:                "{{ source('hubspot','email_event') }}" 
-    email_event_bounce:         "{{ source('hubspot','email_event_bounce') }}" 
-    email_event_click:          "{{ source('hubspot','email_event_click') }}" 
-    email_event_deferred:       "{{ source('hubspot','email_event_deferred') }}" 
-    email_event_delivered:      "{{ source('hubspot','email_event_delivered') }}" 
-    email_event_dropped:        "{{ source('hubspot','email_event_dropped') }}" 
-    email_event_forward:        "{{ source('hubspot','email_event_forward') }}" 
-    email_event_open:           "{{ source('hubspot','email_event_open') }}" 
-    email_event_print:          "{{ source('hubspot','email_event_print') }}" 
-    email_event_sent:           "{{ source('hubspot','email_event_sent') }}" 
-    email_event_spam_report:    "{{ source('hubspot','email_event_spam_report') }}" 
-    email_event_status_change:  "{{ source('hubspot','email_event_status_change') }}" 
-    engagement:                 "{{ source('hubspot','engagement') }}" 
-    engagement_call:            "{{ source('hubspot','engagement_call') }}" 
-    engagement_company:         "{{ source('hubspot','engagement_company') }}" 
-    engagement_contact:         "{{ source('hubspot','engagement_contact') }}" 
-    engagement_deal:            "{{ source('hubspot','engagement_deal') }}" 
-    engagement_email:           "{{ source('hubspot','engagement_email') }}" 
-    engagement_meeting:         "{{ source('hubspot','engagement_meeting') }}" 
-    engagement_note:            "{{ source('hubspot','engagement_note') }}" 
-    engagement_task:            "{{ source('hubspot','engagement_task') }}" 
-    owner:                      "{{ source('hubspot','owner') }}" 
-
-models: 
+models:
   hubspot_source:
     +materialized: table
     +schema: stg_hubspot
     tmp:
-      +materialized: view
+      +materialized: view
+
+vars:
+  hubspot_source:
+    company:                    "{{ source('hubspot','company') }}"
+    company_property_history:   "{{ source('hubspot','company_property_history') }}"
+    contact:                    "{{ source('hubspot','contact') }}"
+    contact_list:               "{{ source('hubspot','contact_list') }}"
+    contact_list_member:        "{{ source('hubspot','contact_list_member') }}"
+    contact_property_history:   "{{ source('hubspot','contact_property_history') }}"
+    deal:                       "{{ source('hubspot','deal') }}"
+    deal_pipeline:              "{{ source('hubspot','deal_pipeline') }}"
+    deal_pipeline_stage:        "{{ source('hubspot','deal_pipeline_stage') }}"
+    deal_property_history:      "{{ source('hubspot','deal_property_history') }}"
+    email_campaign:             "{{ source('hubspot','email_campaign') }}"
+    email_event:                "{{ source('hubspot','email_event') }}"
+    email_event_bounce:         "{{ source('hubspot','email_event_bounce') }}"
+    email_event_click:          "{{ source('hubspot','email_event_click') }}"
+    email_event_deferred:       "{{ source('hubspot','email_event_deferred') }}"
+    email_event_delivered:      "{{ source('hubspot','email_event_delivered') }}"
+    email_event_dropped:        "{{ source('hubspot','email_event_dropped') }}"
+    email_event_forward:        "{{ source('hubspot','email_event_forward') }}"
+    email_event_open:           "{{ source('hubspot','email_event_open') }}"
+    email_event_print:          "{{ source('hubspot','email_event_print') }}"
+    email_event_sent:           "{{ source('hubspot','email_event_sent') }}"
+    email_event_spam_report:    "{{ source('hubspot','email_event_spam_report') }}"
+    email_event_status_change:  "{{ source('hubspot','email_event_status_change') }}"
+    engagement:                 "{{ source('hubspot','engagement') }}"
+    engagement_call:            "{{ source('hubspot','engagement_call') }}"
+    engagement_company:         "{{ source('hubspot','engagement_company') }}"
+    engagement_contact:         "{{ source('hubspot','engagement_contact') }}"
+    engagement_deal:            "{{ source('hubspot','engagement_deal') }}"
+    engagement_email:           "{{ source('hubspot','engagement_email') }}"
+    engagement_meeting:         "{{ source('hubspot','engagement_meeting') }}"
+    engagement_note:            "{{ source('hubspot','engagement_note') }}"
+    engagement_task:            "{{ source('hubspot','engagement_task') }}"
+    owner:                      "{{ source('hubspot','owner') }}"
+
+    #Service Hub Models
+    ticket_company:             "{{ source('hubspot','ticket_company') }}"
+    ticket_contact:             "{{ source('hubspot','ticket_contact') }}"
+    ticket_deal:                "{{ source('hubspot','ticket_deal') }}"
+    ticket_engagement:          "{{ source('hubspot','ticket_engagement') }}"
+    ticket_pipeline:            "{{ source('hubspot','ticket_pipeline') }}"
+    ticket_pipeline_stage:      "{{ source('hubspot','ticket_pipeline_stage') }}"
+    ticket_property_history:    "{{ source('hubspot','ticket_property_history') }}"
+    ticket:                     "{{ source('hubspot','ticket') }}"
+
+    #Service Hub Enable Variables
+    hubspot_service_enabled:    false
+
+    #HubSpot Passthrough Columns
+    hubspot__deal_pass_through_columns: []
+    hubspot__contact_pass_through_columns: []
+    hubspot__company_pass_through_columns: []
+    hubspot__ticket_pass_through_columns: []

integration_tests/data/ticket_company_data.csv

@@ -0,0 +1,10 @@
+_fivetran_synced,ticket_id,company_id
+2020-07-09 11:06:21.056,1,1
+2020-07-09 11:06:21.056,2,1
+2020-07-09 11:06:21.056,3,1
+2020-07-09 11:06:21.056,4,1
+2020-07-09 11:06:21.056,5,1
+2020-07-09 11:06:21.056,6,1
+2020-07-09 11:06:21.056,7,2
+2020-07-09 11:06:21.056,8,2
+2020-07-09 11:06:21.056,9,2

integration_tests/data/ticket_contact_data.csv

@@ -0,0 +1,10 @@
+_fivetran_synced,ticket_id,contact_id
+2020-07-09 11:06:21.056,1,1
+2020-07-09 11:06:21.056,2,2
+2020-07-09 11:06:21.056,3,3
+2020-07-09 11:06:21.056,4,1
+2020-07-09 11:06:21.056,5,2
+2020-07-09 11:06:21.056,6,3
+2020-07-09 11:06:21.056,7,1
+2020-07-09 11:06:21.056,8,2
+2020-07-09 11:06:21.056,9,3

integration_tests/data/ticket_data.csv

@@ -0,0 +1,10 @@
+id,property_closed_date,property_createdate,property_hubspot_owner_id,property_hs_pipeline,property_hs_pipeline_stage,is_deleted
+1,2020-07-09 11:06:21.056,2020-07-09 11:06:21.056,1,2,3,false
+2,2020-07-09 11:06:21.056,2020-07-09 11:06:21.056,1,2,3,false
+3,2020-07-09 11:06:21.056,2020-07-09 11:06:21.056,1,2,3,false
+4,2020-07-09 11:06:21.056,2020-07-09 11:06:21.056,2,3,3,false
+5,2020-07-09 11:06:21.056,2020-07-09 11:06:21.056,2,3,3,false
+6,2020-07-09 11:06:21.056,2020-07-09 11:06:21.056,2,3,3,false
+7,2020-07-09 11:06:21.056,2020-07-09 11:06:21.056,2,3,3,false
+8,2020-07-09 11:06:21.056,2020-07-09 11:06:21.056,2,3,3,false
+9,2020-07-09 11:06:21.056,2020-07-09 11:06:21.056,2,3,3,false

integration_tests/data/ticket_deal_data.csv

@@ -0,0 +1,10 @@
+_fivetran_synced,ticket_id,deal_id
+2020-07-09 11:06:21.056,1,1
+2020-07-09 11:06:21.056,2,2
+2020-07-09 11:06:21.056,3,3
+2020-07-09 11:06:21.056,4,1
+2020-07-09 11:06:21.056,5,2
+2020-07-09 11:06:21.056,6,3
+2020-07-09 11:06:21.056,7,1
+2020-07-09 11:06:21.056,8,2
+2020-07-09 11:06:21.056,9,3

integration_tests/data/ticket_engagement_data.csv

@@ -0,0 +1,10 @@
+_fivetran_synced,ticket_id,engagement_id
+2020-07-09 11:06:21.056,1,1
+2020-07-09 11:06:21.056,2,2
+2020-07-09 11:06:21.056,3,3
+2020-07-09 11:06:21.056,4,14
+2020-07-09 11:06:21.056,5,5
+2020-07-09 11:06:21.056,6,12
+2020-07-09 11:06:21.056,7,1
+2020-07-09 11:06:21.056,8,2
+2020-07-09 11:06:21.056,9,3

integration_tests/data/ticket_pipeline_data.csv

@@ -0,0 +1,9 @@
+pipeline_id,_fivetran_deleted,_fivetran_synced,active,display_order,label,object_type_id
+5386331,FALSE,2021-03-30 14:16:02,TRUE,15,1277d6630760da4e7e889e5d1874bad9,0-5
+1486381,FALSE,2021-03-30 14:16:02,TRUE,5,43a68e729d0696fdd1014a83cd45267e,0-5
+0,FALSE,2021-03-30 14:16:02,TRUE,0,98d937dcebec9d0f6e040220cbf0db31,0-5
+1588311,FALSE,2021-03-30 14:16:02,TRUE,8,fca0ecdf8bcf71156501587a85f286a2,0-5
+2869229,FALSE,2021-03-30 14:16:02,TRUE,13,09f0c5159c5e34504e453eff3fc70324,0-5
+1727805,FALSE,2021-03-30 14:16:02,TRUE,12,0c95da5ea7ede3f7b9ad1df4eadc3fb8,0-5
+900909,FALSE,2021-03-30 14:16:02,TRUE,1,db5eb84117d06047c97c9a0191b5fffe,0-5
+3136178,FALSE,2021-03-30 14:16:02,TRUE,14,3249fce0a7cf3ef72ef9ef04a5578d62,0-5

integration_tests/data/ticket_pipeline_stage_data.csv

@@ -0,0 +1,12 @@
+stage_id,_fivetran_deleted,_fivetran_synced,active,display_order,is_closed,label,pipeline_id,ticket_state
+1,FALSE,2021-03-30 14:16:02,TRUE,0,FALSE,03c2e7e41ffc181a4e84080b4710e81e,0,OPEN
+3,FALSE,2021-03-30 14:16:02,TRUE,2,FALSE,76a756f37ed4f9039fbe152954566e7a,0,OPEN
+7806423,FALSE,2021-03-30 14:16:02,TRUE,3,FALSE,c5f73dc5584f3189388b4294865e9300,0,OPEN
+2074552,FALSE,2021-03-30 14:16:02,TRUE,1,FALSE,0f7532f707b3eff137f9df6f358fa86f,0,OPEN
+8165139,FALSE,2021-03-30 14:16:02,TRUE,1,FALSE,bad7c4a1299dbc47b4742e044d88070c,900909,OPEN
+900910,FALSE,2021-03-30 14:16:02,TRUE,0,FALSE,03c2e7e41ffc181a4e84080b4710e81e,900909,OPEN
+9450484,FALSE,2021-03-30 14:16:02,TRUE,2,FALSE,fb347e3972f2bd02631bfabbfb1dfc87,900909,OPEN
+900911,FALSE,2021-03-30 14:16:02,TRUE,3,FALSE,619e755f83b2b1e9940e321c8baa2299,900909,OPEN
+900912,FALSE,2021-03-30 14:16:02,TRUE,4,FALSE,2bd583bf8d793a60703dea6a9cfc38ec,900909,OPEN
+11543638,FALSE,2021-03-30 14:16:02,TRUE,5,FALSE,84456c8c47378ffdfaef98cbc9a49a89,900909,OPEN
+1486382,FALSE,2021-03-30 14:16:02,TRUE,0,FALSE,4f5372f07271c52cf8420846e945323c,1486381,OPEN

integration_tests/data/ticket_property_history_data.csv

@@ -0,0 +1,12 @@
+_fivetran_synced,ticket_id,name,source,source_id,timestamp_instant,value
+2020-07-09 11:06:21.056,1,seed name, source name, sg5923, 2020-07-09 11:06:21.056,value
+2020-07-09 11:06:21.056,1,seed data is cool, source name1, sg5923, 2020-07-09 11:06:21.056,value
+2020-07-09 11:06:21.056,1,i can make these names whatever i want, source name2, sg5923, 2020-07-09 11:06:21.056,value
+2020-07-09 11:06:21.056,1,seed name1, source name3, sg5923, 2020-07-09 11:06:21.056,value
+2020-07-09 11:06:21.056,1,seed name2, source name4, sg5923, 2020-07-09 11:06:21.056,value
+2020-07-09 11:06:21.056,1,seed name3, source name5, sg5923, 2020-07-09 11:06:21.056,value
+2020-07-09 11:06:21.056,1,stop being lazy with names, source name99, sg5923, 2020-07-09 11:06:21.056,value
+2020-07-09 11:06:21.056,1,seed name4, source name6?, sg5923, 2020-07-09 11:06:21.056,value
+2020-07-09 11:06:21.056,1,seed name5, source name7, sg5923, 2020-07-09 11:06:21.056,value
+2020-07-09 11:06:21.056,1,seed name6, source name8, sg5923, 2020-07-09 11:06:21.056,value
+2020-07-09 11:06:21.056,1,i got lazy with names, source name99, sg5923, 2020-07-09 11:06:21.056,value