Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bugfix/duplicate columns #115

Merged
merged 7 commits into from
Oct 2, 2023
+97 −47
Merged

Bugfix/duplicate columns #115

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
29d4d28
bugfix/duplicate-columns
fivetran-joemarkiewicz Sep 27, 2023
a03b219
lower functions for casing issues
fivetran-joemarkiewicz Sep 27, 2023
1677fd5
Loop twice
greg-finley Sep 27, 2023
2d74ab4
Merge pull request #114 from greg-finley/greg-dupe2
fivetran-joemarkiewicz Sep 27, 2023
078af34
final updates
fivetran-joemarkiewicz Sep 27, 2023
b62f5c5
succinct changelog
fivetran-joemarkiewicz Sep 27, 2023
f31c7c8
remove coalesce from macro
fivetran-joemarkiewicz Oct 2, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 24 additions & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,27 @@
# dbt_hubspot_source v0.12.0
## 🚨 Breaking Changes 🚨
- The following models now use a custom macro to remove the property_hs_ prefix in staging columns, while also preventing duplicates. If de-prefixed columns match existing ones (e.g., `property_hs_meeting_outcome` vs. `meeting_outcome`), the macro favors the `property_hs_`field, aligning with the latest HubSpot API update. ([PR #115](https://github.com/fivetran/dbt_hubspot_source/pull/115))
- `stg_hubspot__engagement_call`
- `stg_hubspot__engagement_company`
- `stg_hubspot__engagement_contact`
- `stg_hubspot__engagement_deal`
- `stg_hubspot__engagement_email`
- `stg_hubspot__engagement_meeting`
- `stg_hubspot__engagement_note`
- `stg_hubspot__engagement_task`
- `stg_hubspot__ticket`
- `stg_hubspot__ticket_company`
- `stg_hubspot__ticket_contact`
- `stg_hubspot__ticket_deal`
- `stg_hubspot__ticket_engagement`
- `stg_hubspot__ticket_property_history`

## Feature Updates
- A new macro `remove_duplicate_and_prefix_from_columns` has been included which expands off the `fivetran_utils.remove_prefix_columns` macro by removing any duplicate columns that result from the prefix removal. ([PR #115](https://github.com/fivetran/dbt_hubspot_source/pull/115) and [PR #114](https://github.com/fivetran/dbt_hubspot_source/pull/114))

## Contributors
- [@greg-finley](https://github.com/greg-finley) ([PR #114](https://github.com/fivetran/dbt_hubspot_source/pull/114))

# dbt_hubspot_source v0.11.0
[PR #112](https://github.com/fivetran/dbt_hubspot_source/pull/112) includes the following updates:
## 🚨 Breaking Changes 🚨
Expand Down
2 changes: 1 addition & 1 deletion README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,7 @@ Include the following hubspot_source package version in your `packages.yml` file
```yaml
packages:
- package: fivetran/hubspot_source
version: [">=0.11.0", "<0.12.0"]
version: [">=0.12.0", "<0.13.0"]
```
## Step 3: Define database and schema variables
By default, this package runs using your destination and the `hubspot` schema. If this is not where your HubSpot data is (for example, if your HubSpot schema is named `hubspot_fivetran`), add the following configuration to your root `dbt_project.yml` file:
Expand Down
2 changes: 1 addition & 1 deletion dbt_project.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
name: 'hubspot_source'
version: '0.11.0'
version: '0.12.0'
config-version: 2
require-dbt-version: [">=1.3.0", "<2.0.0"]
models:
Expand Down
2 changes: 1 addition & 1 deletion docs/catalog.json
View file
Open in desktop

Large diffs are not rendered by default.

24 changes: 12 additions & 12 deletions docs/index.html
View file
Open in desktop

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion docs/manifest.json
View file
Open in desktop

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion docs/run_results.json
View file
Open in desktop

Large diffs are not rendered by default.

1 change: 1 addition & 0 deletions integration_tests/.gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,3 +2,4 @@
target/
dbt_modules/
logs/
env/
2 changes: 1 addition & 1 deletion integration_tests/dbt_project.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
name: 'hubspot_source_integration_tests'
version: '0.11.0'
version: '0.12.0'
profile: 'integration_tests'
config-version: 2
models:
Expand Down
5 changes: 3 additions & 2 deletions integration_tests/seeds/engagement_task_data.csv
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,2 +1,3 @@
engagement_id,_fivetran_deleted,_fivetran_synced,property_hs_createdate,property_hs_object_id,property_hs_task_type,property_hs_timestamp,property_hubspot_owner_id,property_hubspot_team_id,type
32034933897,false,2023-06-08 23:22:51.021000,2023-03-03 10:40:09.556000,32034933897,TODO,2023-03-08 02:30:00.000000,32779820,9747038,TASK
engagement_id,_fivetran_deleted,_fivetran_synced,property_hs_createdate,property_hs_object_id,property_hs_task_type,property_hs_timestamp,property_hubspot_owner_id,property_hubspot_team_id,type,task_type,property_hs_engagement_source
32034933897,false,2023-06-08 23:22:51.021000,2023-03-03 10:40:09.556000,32034933897,TODO,2023-03-08 02:30:00.000000,32779820,9747038,TASK,,Hello
32034933899,false,2023-06-08 23:22:51.021000,2023-03-03 10:40:09.556000,32034933897,,2023-03-08 02:30:00.000000,32779820,,,TODO,
24 changes: 24 additions & 0 deletions macros/remove_duplicate_and_prefix_from_columns.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
{% macro remove_duplicate_and_prefix_from_columns(columns, prefix='', exclude=[]) %}

{# Loop once to find duplicates #}
{%- set duplicate_exclude = [] -%}
{% for col in columns if col.name not in exclude %}
{%- for dupe in columns if col.name[prefix|length:]|lower == dupe.name|lower -%}
Comment on lines +5 to +6
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This nested loop thing is O(n^2) right? I'm still learning Jinja, but seems it also supports dictionaries https://documentation.bloomreach.com/engagement/docs/datastructures#dictionaries

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@greg-finley correct it does result in n^2 and jinja does support dictionaries. How are you proposing we leverage a dictionary to replace the nested loops?

It may be worthwhile to move forward with this solution in the immediate so users can resolve the error and consider optimizing the macro down the road.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep, 100%, let's ship it.

If we had the list of columns in a dictionary or set, we could O(1) look up whether the duplicate name exists vs looking through the column list again.

{%- do duplicate_exclude.append(col.name) -%}
{%- do duplicate_exclude.append(dupe.name) -%}
, {{ col.name }} as {{ col.name[prefix|length:] }}
{%- endfor %}
{% endfor %}

{# Loop again to find non-duplicates #}
{% for col in columns if col.name not in exclude %}
{%- if col.name|lower not in duplicate_exclude|lower -%}
{% if col.name[:prefix|length]|lower == prefix %}
, {{ col.name }} as {{ col.name[prefix|length:] }}
{%- else %}
, {{ col.name }}
{%- endif -%}
{%- endif -%}
{% endfor %}

{% endmacro %}
4 changes: 2 additions & 2 deletions models/stg_hubspot__engagement_call.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,15 +9,15 @@ with base as (

select
{% set default_cols = adapter.get_columns_in_relation(ref('stg_hubspot__engagement_call_tmp')) %}
{% set new_cols = fivetran_utils.remove_prefix_from_columns(columns=default_cols,
{% set new_cols = remove_duplicate_and_prefix_from_columns(columns=default_cols,
prefix='property_hs_',exclude=get_macro_columns(get_engagement_call_columns())) %}
{{
fivetran_utils.fill_staging_columns(source_columns=default_cols,
staging_columns=get_engagement_call_columns()
)
}}
{% if new_cols | length > 0 %}
,{{ new_cols }}
{{ new_cols }}
{% endif %}
from base

Expand Down
4 changes: 2 additions & 2 deletions models/stg_hubspot__engagement_company.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,15 +9,15 @@ with base as (

select
{% set default_cols = adapter.get_columns_in_relation(ref('stg_hubspot__engagement_company_tmp')) %}
{% set new_cols = fivetran_utils.remove_prefix_from_columns(columns=default_cols,
{% set new_cols = remove_duplicate_and_prefix_from_columns(columns=default_cols,
prefix='property_hs_',exclude=get_macro_columns(get_engagement_company_columns())) %}
{{
fivetran_utils.fill_staging_columns(source_columns=default_cols,
staging_columns=get_engagement_company_columns()
)
}}
{% if new_cols | length > 0 %}
,{{ new_cols }}
{{ new_cols }}
{% endif %}
from base
)
Expand Down
4 changes: 2 additions & 2 deletions models/stg_hubspot__engagement_contact.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,15 +9,15 @@ with base as (

select
{% set default_cols = adapter.get_columns_in_relation(ref('stg_hubspot__engagement_contact_tmp')) %}
{% set new_cols = fivetran_utils.remove_prefix_from_columns(columns=default_cols,
{% set new_cols = remove_duplicate_and_prefix_from_columns(columns=default_cols,
prefix='property_hs_',exclude=get_macro_columns(get_engagement_contact_columns())) %}
{{
fivetran_utils.fill_staging_columns(source_columns=default_cols,
staging_columns=get_engagement_contact_columns()
)
}}
{% if new_cols | length > 0 %}
,{{ new_cols }}
{{ new_cols }}
{% endif %}
from base

Expand Down
4 changes: 2 additions & 2 deletions models/stg_hubspot__engagement_deal.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,15 +9,15 @@ with base as (

select
{% set default_cols = adapter.get_columns_in_relation(ref('stg_hubspot__engagement_deal_tmp')) %}
{% set new_cols = fivetran_utils.remove_prefix_from_columns(columns=default_cols,
{% set new_cols = remove_duplicate_and_prefix_from_columns(columns=default_cols,
prefix='property_hs_',exclude=get_macro_columns(get_engagement_deal_columns())) %}
{{
fivetran_utils.fill_staging_columns(source_columns=default_cols,
staging_columns=get_engagement_deal_columns()
)
}}
{% if new_cols | length > 0 %}
,{{ new_cols }}
{{ new_cols }}
{% endif %}
from base

Expand Down
4 changes: 2 additions & 2 deletions models/stg_hubspot__engagement_email.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,15 +9,15 @@ with base as (

select
{% set default_cols = adapter.get_columns_in_relation(ref('stg_hubspot__engagement_email_tmp')) %}
{% set new_cols = fivetran_utils.remove_prefix_from_columns(columns=default_cols,
{% set new_cols = remove_duplicate_and_prefix_from_columns(columns=default_cols,
prefix='property_hs_',exclude=get_macro_columns(get_engagement_email_columns())) %}
{{
fivetran_utils.fill_staging_columns(source_columns=default_cols,
staging_columns=get_engagement_email_columns()
)
}}
{% if new_cols | length > 0 %}
,{{ new_cols }}
{{ new_cols }}
{% endif %}
from base

Expand Down
4 changes: 2 additions & 2 deletions models/stg_hubspot__engagement_meeting.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,15 +9,15 @@ with base as (

select
{% set default_cols = adapter.get_columns_in_relation(ref('stg_hubspot__engagement_meeting_tmp')) %}
{% set new_cols = fivetran_utils.remove_prefix_from_columns(columns=default_cols,
{% set new_cols = remove_duplicate_and_prefix_from_columns(columns=default_cols,
prefix='property_hs_',exclude=get_macro_columns(get_engagement_meeting_columns())) %}
{{
fivetran_utils.fill_staging_columns(source_columns=default_cols,
staging_columns=get_engagement_meeting_columns()
)
}}
{% if new_cols | length > 0 %}
,{{ new_cols }}
{{ new_cols }}
{% endif %}
from base

Expand Down
4 changes: 2 additions & 2 deletions models/stg_hubspot__engagement_note.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,15 +9,15 @@ with base as (

select
{% set default_cols = adapter.get_columns_in_relation(ref('stg_hubspot__engagement_note_tmp')) %}
{% set new_cols = fivetran_utils.remove_prefix_from_columns(columns=default_cols,
{% set new_cols = remove_duplicate_and_prefix_from_columns(columns=default_cols,
prefix='property_hs_',exclude=get_macro_columns(get_engagement_note_columns())) %}
{{
fivetran_utils.fill_staging_columns(source_columns=default_cols,
staging_columns=get_engagement_note_columns()
)
}}
{% if new_cols | length > 0 %}
,{{ new_cols }}
{{ new_cols }}
{% endif %}
from base

Expand Down
4 changes: 2 additions & 2 deletions models/stg_hubspot__engagement_task.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,15 +9,15 @@ with base as (

select
{% set default_cols = adapter.get_columns_in_relation(ref('stg_hubspot__engagement_task_tmp')) %}
{% set new_cols = fivetran_utils.remove_prefix_from_columns(columns=default_cols,
{% set new_cols = remove_duplicate_and_prefix_from_columns(columns=default_cols,
prefix='property_hs_',exclude=get_macro_columns(get_engagement_task_columns())) %}
{{
fivetran_utils.fill_staging_columns(source_columns=default_cols,
staging_columns=get_engagement_task_columns()
)
}}
{% if new_cols | length > 0 %}
,{{ new_cols }}
{{ new_cols }}
{% endif %}
from base
)
Expand Down
2 changes: 1 addition & 1 deletion models/stg_hubspot__ticket.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@ with base as (
{% if all_passthrough_column_check('stg_hubspot__ticket_tmp',get_ticket_columns()) > 0 %}
-- just pass everything through if extra columns are present, but ensure required columns are present.
,{{
fivetran_utils.remove_prefix_from_columns(
remove_duplicate_and_prefix_from_columns(
columns=adapter.get_columns_in_relation(ref('stg_hubspot__ticket_tmp')),
prefix='property_', exclude=get_macro_columns(get_ticket_columns()))
}}
Expand Down
4 changes: 2 additions & 2 deletions models/stg_hubspot__ticket_company.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,15 +9,15 @@ with base as (

select
{% set default_cols = adapter.get_columns_in_relation(ref('stg_hubspot__ticket_company_tmp')) %}
{% set new_cols = fivetran_utils.remove_prefix_from_columns(columns=default_cols,
{% set new_cols = remove_duplicate_and_prefix_from_columns(columns=default_cols,
prefix='property_hs_',exclude=get_macro_columns(get_ticket_company_columns())) %}
{{
fivetran_utils.fill_staging_columns(source_columns=default_cols,
staging_columns=get_ticket_company_columns()
)
}}
{% if new_cols | length > 0 %}
,{{ new_cols }}
{{ new_cols }}
{% endif %}
from base

Expand Down
4 changes: 2 additions & 2 deletions models/stg_hubspot__ticket_contact.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,15 +9,15 @@ with base as (

select
{% set default_cols = adapter.get_columns_in_relation(ref('stg_hubspot__ticket_contact_tmp')) %}
{% set new_cols = fivetran_utils.remove_prefix_from_columns(columns=default_cols,
{% set new_cols = remove_duplicate_and_prefix_from_columns(columns=default_cols,
prefix='property_hs_',exclude=get_macro_columns(get_ticket_contact_columns())) %}
{{
fivetran_utils.fill_staging_columns(source_columns=default_cols,
staging_columns=get_ticket_contact_columns()
)
}}
{% if new_cols | length > 0 %}
,{{ new_cols }}
{{ new_cols }}
{% endif %}
from base

Expand Down
4 changes: 2 additions & 2 deletions models/stg_hubspot__ticket_deal.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,15 +9,15 @@ with base as (

select
{% set default_cols = adapter.get_columns_in_relation(ref('stg_hubspot__ticket_deal_tmp')) %}
{% set new_cols = fivetran_utils.remove_prefix_from_columns(columns=default_cols,
{% set new_cols = remove_duplicate_and_prefix_from_columns(columns=default_cols,
prefix='property_hs_',exclude=get_macro_columns(get_ticket_deal_columns())) %}
{{
fivetran_utils.fill_staging_columns(source_columns=default_cols,
staging_columns=get_ticket_deal_columns()
)
}}
{% if new_cols | length > 0 %}
,{{ new_cols }}
{{ new_cols }}
{% endif %}
from base

Expand Down
4 changes: 2 additions & 2 deletions models/stg_hubspot__ticket_engagement.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,15 +9,15 @@ with base as (

select
{% set default_cols = adapter.get_columns_in_relation(ref('stg_hubspot__ticket_engagement_tmp')) %}
{% set new_cols = fivetran_utils.remove_prefix_from_columns(columns=default_cols,
{% set new_cols = remove_duplicate_and_prefix_from_columns(columns=default_cols,
prefix='property_hs_',exclude=get_macro_columns(get_ticket_engagement_columns())) %}
{{
fivetran_utils.fill_staging_columns(source_columns=default_cols,
staging_columns=get_ticket_engagement_columns()
)
}}
{% if new_cols | length > 0 %}
,{{ new_cols }}
{{ new_cols }}
{% endif %}
from base

Expand Down
4 changes: 2 additions & 2 deletions models/stg_hubspot__ticket_property_history.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,15 +9,15 @@ with base as (

select
{% set default_cols = adapter.get_columns_in_relation(ref('stg_hubspot__ticket_property_history_tmp')) %}
{% set new_cols = fivetran_utils.remove_prefix_from_columns(columns=default_cols,
{% set new_cols = remove_duplicate_and_prefix_from_columns(columns=default_cols,
prefix='property_hs_',exclude=get_macro_columns(get_ticket_property_history_columns())) %}
{{
fivetran_utils.fill_staging_columns(source_columns=default_cols,
staging_columns=get_ticket_property_history_columns()
)
}}
{% if new_cols | length > 0 %}
,{{ new_cols }}
{{ new_cols }}
{% endif %}
from base

Expand Down