Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
PR Overview
This PR will address the following Issue/Feature: Issue #111
This PR will result in the following new package version:
v0.11.0
This will be a breaking change as the upstream dbt_hubspot_source package included major updates as a result of the API v3 changes (see our release notes here from the connector. As a result, many tables were impacted on how they sync data. As such, we should make this a breaking change and require customers to ensure they are on the latest version of the HubSpot connector in order to leverage this package.
I do also want to call out that I was able to make these changes compatible with the old version of the connector (pre v3 API update). That being said, when using the old version of the connector with this package, you will see some fields are not documented as they have been removed from the newer API version. Likewise, due to the backwards compatibility, there are some fields that will always be
null
for users of the new API. These fields have been identified in the yml documentation and notified for deprecation and removal once all connectors are upgraded (next few months).Please detail what change(s) this PR introduces and any additional information that should be known during the review of this PR:
🚨 Breaking Changes 🚨
This change is made breaking in part to updates applied to the upstream dbt_hubspot_source package following upgrades to ensure compatibility with the HubSpot v3 API updates. Please see below for the relevant upstream changes:
property_hs_*
fields. Please be aware that theproperty_hs_*
will be truncated from the field name in the staging and downstream models. The impacted sources (and relevant staging models) are below:Please note that while these changes are breaking, the package has been updated to ensure backwards compatibility with the pre HubSpot v3 API updates. As a result, you may see some
null
fields which are artifacts of the pre v3 API HubSpot version. Be sure to inspect the relevant field descriptions for an understanding of which fields remain for backwards compatibility purposes. These fields will be removed once all HubSpot connectors are upgraded to the v3 API.The
engagements_joined
macro has been adjusted to account for the HubSpot v3 API changes. In particular, the fields that used to only be available in theengagements
source are now available in the individualengagement_[type]
sources. As such, coalesce statements were added to ensure the correct populated fields are used following the join. Further, to avoid ambiguous columns adbt-utils.star
was added to remove the explicitly declared columns within the coalesce statements.Under the Hood
base_model
argument used for theengagements_joined
macro has been updated to be an explicitref
as opposed to the previously usedvar
within the below models:hubspot__engagement_calls
hubspot__engagement_emails
hubspot__engagement_meetings
hubspot__engagement_notes
hubspot__engagement_tasks
var
result was producing inconsistent results during compile and runtime when leveraging thedbt-utils.star
macro. However, the explicitref
always provided consistent results.Documentation Updates
hubspot__engagement_calls
hubspot__engagement_emails
hubspot__engagement_meetings
hubspot__engagement_notes
hubspot__engagement_tasks
PR Checklist
Basic Validation
Please acknowledge that you have successfully performed the following commands locally:
I ran various versions of the package with variables enabled and disabled. In particular I tested to ensure the changed models worked when enabled and disabled.
Before marking this PR as "ready for review" the following have been applied:
Detailed Validation
Please acknowledge that the following validation checks have been performed prior to marking this PR as "ready for review":
This package update has been validated to ensure it works as expected on the new v3 API update while also working on the older api version. Please see the corresponding Height ticket for the schemas used to validate. Additionally, I was able to gain confidence in these changes due to the customer in the linked Issue claiming the fix resolved upgrade issues they were experiencing.
Additionally, I was able to validate that the impacts downstream models updated within this PR saw consistent results when using the old and new versions of the package. See the below screenshots for a few validations. FYI the old schema is listed as
zz_dbt_joe_hubspot_old_api
and the new is simplyzz_dbt_joe_hubspot
.Old hubspot__engagements model vs New
I also was able to validate that for all the
engagement__[type]
end models that the row counts were inconsistent between versions which was expected.Standard Updates
Please acknowledge that your PR contains the following standard updates:
dbt Docs
Please acknowledge that after the above were all completed the below were applied to your branch:
If you had to summarize this PR in an emoji, which would it be?
🙃