expand LEAD default columns (#35)

* stage changes * source docs * changelog * make breaking change and add other stuff * docs * changelog * changelog
fivetran · Apr 2, 2024 · 70ad275 · 70ad275
1 parent e348042
commit 70ad275
Show file tree

Hide file tree

Showing 14 changed files with 294 additions and 53 deletions.
diff --git a/.github/PULL_REQUEST_TEMPLATE/maintainer_pull_request_template.md b/.github/PULL_REQUEST_TEMPLATE/maintainer_pull_request_template.md
@@ -4,48 +4,27 @@
 **This PR will result in the following new package version:**
 <!--- Please add details around your decision for breaking vs non-breaking version upgrade. If this is a breaking change, were backwards-compatible options explored? -->
 
-**Please detail what change(s) this PR introduces and any additional information that should be known during the review of this PR:**
+**Please provide the finalized CHANGELOG entry which details the relevant changes included in this PR:**
+<!--- Copy/paste the CHANGELOG for this version below. -->
 
 ## PR Checklist
 ### Basic Validation
 Please acknowledge that you have successfully performed the following commands locally:
-- [ ] dbt compile
-- [ ] dbt run –full-refresh
-- [ ] dbt run
-- [ ] dbt test
-- [ ] dbt run –vars (if applicable)
+- [ ] dbt run –full-refresh && dbt test
+- [ ] dbt run (if incremental models are present) && dbt test
 
 Before marking this PR as "ready for review" the following have been applied:
-- [ ] The appropriate issue has been linked and tagged
-- [ ] You are assigned to the corresponding issue and this PR
+- [ ] The appropriate issue has been linked, tagged, and properly assigned
+- [ ] All necessary documentation and version upgrades have been applied
+    <!--- Be sure to update the package version in the dbt_project.yml, integration_tests/dbt_project.yml, and README if necessary. -->
+- [ ] docs were regenerated (unless this PR does not include any code or yml updates)
 - [ ] BuildKite integration tests are passing
+- [ ] Detailed validation steps have been provided below
 
 ### Detailed Validation
-Please acknowledge that the following validation checks have been performed prior to marking this PR as "ready for review":
-- [ ] You have validated these changes and assure this PR will address the respective Issue/Feature.
-- [ ] You are reasonably confident these changes will not impact any other components of this package or any dependent packages.
-- [ ] You have provided details below around the validation steps performed to gain confidence in these changes.
+Please share any and all of your validation steps:
 <!--- Provide the steps you took to validate your changes below. -->
 
-### Standard Updates
-Please acknowledge that your PR contains the following standard updates:
-- Package versioning has been appropriately indexed in the following locations:
-    - [ ] indexed within dbt_project.yml
-    - [ ] indexed within integration_tests/dbt_project.yml
-- [ ] CHANGELOG has individual entries for each respective change in this PR
-    <!--- If there is a parallel upstream change, remember to reference the corresponding CHANGELOG as an individual entry.  -->
-- [ ] README updates have been applied (if applicable)
-    <!--- Remember to check the following README locations for common updates. →
-        <!--- Suggested install range (needed for breaking changes) →
-        <!--- Dependency matrix is appropriately updated (if applicable) →
-        <!--- New variable documentation (if applicable) -->
-- [ ] DECISIONLOG updates have been updated (if applicable)
-- [ ] Appropriate yml documentation has been added (if applicable)
-
-### dbt Docs
-Please acknowledge that after the above were all completed the below were applied to your branch:
-- [ ] docs were regenerated (unless this PR does not include any code or yml updates)
-
 ### If you had to summarize this PR in an emoji, which would it be?
 <!--- For a complete list of markdown compatible emojis check our this git repo (https://gist.github.com/rxaviers/7360908)  --> 
-:dancer:
+:dancer:
diff --git a/.github/workflows/auto-release.yml b/.github/workflows/auto-release.yml
@@ -0,0 +1,13 @@
+name: 'auto release'
+on:
+  pull_request:
+    types:
+      - closed
+    branches:
+      - main
+
+jobs:
+  call-workflow-passing-data:
+    if: github.event.pull_request.merged
+    uses: fivetran/dbt_package_automations/.github/workflows/auto-release.yml@main
+    secrets: inherit
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -1,3 +1,44 @@
+# dbt_marketo_source v0.11.0
+[PR #35](https://github.com/fivetran/dbt_marketo_source/pull/35) includes the following updates:
+
+## Feature Updates (includes 🚨 breaking changes 🚨)
+- Ensures that `stg_marketo__lead` has and documents the below columns, all [standard](https://developers.marketo.com/rest-api/lead-database/fields/list-of-standard-fields/) fields from Marketo. Previously, peristed all fields found in your `LEAD` source table but only _ensured_ that the `id`, `created_at`, `updated_at`, `email`, `first_name`, `last_name`, and `_fivetran_synced` fields were included. If any of the following default columns are missing from your `LEAD` table, `stg_marketo__lead` will create a NULL version with the proper data type:
+  - `phone`
+  - `main_phone`
+  - `mobile_phone`
+  - `company`
+  - `inferred_company`
+  - `address_lead`
+  - `address`
+  - `city`
+  - `state`
+  - `state_code`
+  - `country`
+  - `country_code`
+  - `postal_code`
+  - `billing_street`
+  - `billing_city`
+  - `billing_state`
+  - `billing_state_code`
+  - `billing_country`
+  - `billing_country_code`
+  - `billing_postal_code`
+  - `inferred_city`
+  - `inferred_state_region`
+  - `inferred_country`
+  - `inferred_postal_code`
+  - `inferred_phone_area_code`
+  - `anonymous_ip`
+  - `unsubscribed` -> aliased as `is_unsubscribed` (🚨 breaking change 🚨)
+  - `email_invalid` -> aliased as `is_email_invalid` (🚨 breaking change 🚨)
+  - `do_not_call`
+
+> Note: the above fields will persist downstream into the [transform](https://github.com/fivetran/dbt_marketo/blob/main/models/marketo__leads.sql) `marketo__leads` model.
+
+## Under the Hood
+- Updated the maintainer PR template to resemble the most up to date format.
+- Included auto-releaser GitHub Actions workflow to automate future releases.
+
 # dbt_marketo_source v0.10.0
 ## 🚨 Breaking Changes 🚨:
 [PR #33](https://github.com/fivetran/dbt_marketo_source/pull/33) includes the following updates in connection with the Fivetran Marketo connector's [June 2023](https://fivetran.com/docs/applications/marketo/changelog#june2023) and [May 2023](https://fivetran.com/docs/applications/marketo/changelog#may2023) releases:

diff --git a/README.md b/README.md
@@ -43,7 +43,7 @@ If you  are **not** using the [Marketo transformation package](https://github.co
 ```yml
 packages:
   - package: fivetran/marketo_source
-    version: [">=0.10.0", "<0.11.0"]
+    version: [">=0.11.0", "<0.12.0"]
 ```
 
 ## Step 3: Define database and schema variables

diff --git a/dbt_project.yml b/dbt_project.yml
@@ -1,5 +1,5 @@
 name: 'marketo_source'
-version: '0.10.0'
+version: '0.11.0'
 config-version: 2
 require-dbt-version: [">=1.3.0", "<2.0.0"]
 models:

diff --git a/docs/catalog.json b/docs/catalog.json
diff --git a/docs/index.html b/docs/index.html
diff --git a/docs/manifest.json b/docs/manifest.json
diff --git a/docs/run_results.json b/docs/run_results.json
diff --git a/integration_tests/dbt_project.yml b/integration_tests/dbt_project.yml
@@ -1,5 +1,5 @@
 name: 'marketo_source_integration_tests'
-version: '0.10.0'
+version: '0.11.0'
 profile: 'integration_tests'
 config-version: 2
 vars:

diff --git a/macros/get_lead_columns.sql b/macros/get_lead_columns.sql
@@ -7,9 +7,38 @@
     {"name": "first_name", "datatype": dbt.type_string()},
     {"name": "id", "datatype": dbt.type_int(), "alias": "lead_id"},
     {"name": "last_name", "datatype": dbt.type_string()},
-    {"name": "updated_at", "datatype": dbt.type_timestamp(), "alias": "updated_timestamp"}
+    {"name": "updated_at", "datatype": dbt.type_timestamp(), "alias": "updated_timestamp"},
+    {"name": "phone", "datatype": dbt.type_string()},
+    {"name": "main_phone", "datatype": dbt.type_string()},
+    {"name": "mobile_phone", "datatype": dbt.type_string()},
+    {"name": "company", "datatype": dbt.type_string()},
+    {"name": "inferred_company", "datatype": dbt.type_string()},
+    {"name": "address_lead", "datatype": dbt.type_string()},
+    {"name": "address", "datatype": dbt.type_string()},
+    {"name": "city", "datatype": dbt.type_string()},
+    {"name": "state", "datatype": dbt.type_string()},
+    {"name": "state_code", "datatype": dbt.type_string()},
+    {"name": "country", "datatype": dbt.type_string()},
+    {"name": "country_code", "datatype": dbt.type_string()},
+    {"name": "postal_code", "datatype": dbt.type_string()},
+    {"name": "billing_street", "datatype": dbt.type_string()},
+    {"name": "billing_city", "datatype": dbt.type_string()},
+    {"name": "billing_state", "datatype": dbt.type_string()},
+    {"name": "billing_state_code", "datatype": dbt.type_string()},
+    {"name": "billing_country", "datatype": dbt.type_string()},
+    {"name": "billing_country_code", "datatype": dbt.type_string()},
+    {"name": "billing_postal_code", "datatype": dbt.type_string()},
+    {"name": "inferred_city", "datatype": dbt.type_string()},
+    {"name": "inferred_state_region", "datatype": dbt.type_string()},
+    {"name": "inferred_country", "datatype": dbt.type_string()},
+    {"name": "inferred_postal_code", "datatype": dbt.type_string()},
+    {"name": "inferred_phone_area_code", "datatype": dbt.type_string()},
+    {"name": "anonymous_ip", "datatype": dbt.type_string()},
+    {"name": "unsubscribed", "datatype": dbt.type_boolean(), "alias": "is_unsubscribed"},
+    {"name": "email_invalid", "datatype": dbt.type_boolean(), "alias": "is_email_invalid"},
+    {"name": "do_not_call", "datatype": dbt.type_boolean()}
 ] %}
 
 {{ return(columns) }}
 
-{% endmacro %}
+{% endmacro %}
diff --git a/models/src_marketo.yml b/models/src_marketo.yml
@@ -28,6 +28,93 @@ sources:
           - name: last_name
             description: The last name of the lead.
 
+          - name: phone
+            description: Lead’s Phone Number.
+
+          - name: main_phone
+            description: Primary phone number of the lead’s company
+
+          - name: mobile_phone
+            description: Lead’s mobile phone number
+
+          - name: company
+            description: Lead’s company name
+
+          - name: inferred_company
+            description: Company name inferred by reverse IP lookup of the lead’s first recorded web visit
+
+          - name: address_lead
+            description: Second address associated with the lead.
+
+          - name: address
+            description: Lead’s address. 
+
+          - name: city
+            description: Lead’s city
+
+          - name: state
+            description: Lead’s state
+
+          - name: state_code
+            description: Alpha-2 code of the Lead’s state
+
+          - name: country
+            description: Lead’s country
+
+          - name: country_code
+            description: Alpha-2 code of the Lead’s country
+
+          - name: postal_code
+            description: Lead’s postal code
+
+          - name: billing_street
+            description: Billing street address of the lead’s company
+
+          - name: billing_city
+            description: City of the lead’s billing address
+
+          - name: billing_state
+            description: State or province of the lead’s billing address
+
+          - name: billing_state_code
+            description: Alpha-2 code of the state or province of the lead’s billing address
+
+          - name: billing_country
+            description: Country of the lead’s billing address
+
+          - name: billing_country_code
+            description: Alpha-2 code of the country of the lead’s billing address
+
+          - name: billing_postal_code
+            description: Postal code of the lead’s billing address
+
+          - name: inferred_city
+            description: Lead’s city inferred by reverse IP lookup of lead’s first recorded web visit.
+
+          - name: inferred_state_region
+            description: Lead’s state region inferred by reverse IP lookup of lead’s first recorded web visit.
+
+          - name: inferred_country
+            description: Country inferred by reverse IP lookup of the lead’s first recorded web visit
+
+          - name: inferred_postal_code
+            description: Lead’s postal code inferred by reverse IP lookup of lead’s first recorded web visit.
+
+          - name: inferred_phone_area_code
+            description: Lead’s phone area code inferred by reverse IP lookup of lead’s first recorded web visit.
+
+          - name: anonymous_ip
+            description: IP address of the lead’s first recorded web visit
+
+          - name: unsubscribed
+            description: Lead’s email-unsubscribed status (boolean). Partially system managed. Will prevent receipt of non-operational emails if set to true.
+
+          - name: email_invalid
+            description: Email invalid status (boolean). All emails to the address will be blocked if set to true. Bounces indicating that the email is invalid will automatically set this field to true.
+
+          - name: do_not_call
+            description: Lead’s do-not-call preference (boolean)
+
       - name: lead_describe
         identifier: "{{ var('marketo_lead_describe_identifier', 'lead_describe')}}"
         description: Table with metadata about lead object columns

diff --git a/models/stg_marketo.yml b/models/stg_marketo.yml
@@ -595,6 +595,93 @@ models:
       - name: last_name
         description: The last name of the lead.
 
+      - name: phone
+        description: Lead’s Phone Number.
+
+      - name: main_phone
+        description: Primary phone number of the lead’s company
+
+      - name: mobile_phone
+        description: Lead’s mobile phone number
+
+      - name: company
+        description: Lead’s company name
+
+      - name: inferred_company
+        description: Company name inferred by reverse IP lookup of the lead’s first recorded web visit
+
+      - name: address_lead
+        description: Second address associated with the lead.
+
+      - name: address
+        description: Lead’s address. 
+
+      - name: city
+        description: Lead’s city
+
+      - name: state
+        description: Lead’s state
+
+      - name: state_code
+        description: Alpha-2 code of the Lead’s state
+
+      - name: country
+        description: Lead’s country
+
+      - name: country_code
+        description: Alpha-2 code of the Lead’s country
+
+      - name: postal_code
+        description: Lead’s postal code
+
+      - name: billing_street
+        description: Billing street address of the lead’s company
+
+      - name: billing_city
+        description: City of the lead’s billing address
+
+      - name: billing_state
+        description: State or province of the lead’s billing address
+
+      - name: billing_state_code
+        description: Alpha-2 code of the state or province of the lead’s billing address
+
+      - name: billing_country
+        description: Country of the lead’s billing address
+
+      - name: billing_country_code
+        description: Alpha-2 code of the country of the lead’s billing address
+
+      - name: billing_postal_code
+        description: Postal code of the lead’s billing address
+
+      - name: inferred_city
+        description: Lead’s city inferred by reverse IP lookup of lead’s first recorded web visit.
+
+      - name: inferred_state_region
+        description: Lead’s state region inferred by reverse IP lookup of lead’s first recorded web visit.
+
+      - name: inferred_country
+        description: Country inferred by reverse IP lookup of the lead’s first recorded web visit
+
+      - name: inferred_postal_code
+        description: Lead’s postal code inferred by reverse IP lookup of lead’s first recorded web visit.
+
+      - name: inferred_phone_area_code
+        description: Lead’s phone area code inferred by reverse IP lookup of lead’s first recorded web visit.
+
+      - name: anonymous_ip
+        description: IP address of the lead’s first recorded web visit
+
+      - name: is_unsubscribed
+        description: Lead’s email-unsubscribed status (boolean). Partially system managed. Will prevent receipt of non-operational emails if set to true.
+
+      - name: is_email_invalid
+        description: Email invalid status (boolean). All emails to the address will be blocked if set to true. Bounces indicating that the email is invalid will automatically set this field to true.
+
+      - name: do_not_call
+        description: Lead’s do-not-call preference (boolean)
+
   - name: stg_marketo__program
     columns:
       - name: program_id

diff --git a/models/stg_marketo__lead.sql b/models/stg_marketo__lead.sql
@@ -13,7 +13,12 @@ with base as (
             )
         }}
         -- This will check if there are non-default columns to bring in
-        {% set default_cols = ['id', 'created_at', 'updated_at', 'email', 'first_name', 'last_name', '_fivetran_synced'] %}
+        {% set default_cols = ['id', 'created_at', 'updated_at', 'email', 'first_name', 'last_name', '_fivetran_synced',
+        'phone', 'main_phone', 'mobile_phone', 'company', 'inferred_company', 'address_lead', 'address', 'city', 'state',
+        'state_code', 'country', 'country_code', 'postal_code', 'billing_street', 'billing_city', 'billing_state', 
+        'billing_state_code', 'billing_country', 'billing_country_code', 'billing_postal_code', 'inferred_city', 'inferred_state_region', 
+        'inferred_country', 'inferred_postal_code', 'inferred_phone_area_code', 'anonymous_ip', 'unsubscribed', 'email_invalid', 'do_not_call'] %}
+
         {% set new_cols = dbt_utils.star(from=ref('stg_marketo__lead_tmp'), except=default_cols) %}
         {% if new_cols != '/* no columns returned from star() macro */' %}
             ,{{ new_cols }}