Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

pass through custom data columns #35

Closed
wants to merge 1 commit into from
Closed

pass through custom data columns #35

wants to merge 1 commit into from

Conversation

jamesrayoub
Copy link
Contributor

Rather than simply pass along all source columns sans property_, why not set a default set of core columns and allow the user to add custom data columns using variable maps. Some tables have 100+ fields, with many that have very little value. hubspot.deal is particularly awful; for example, what can I do with property_hs_date_exited_1215013...

This behavior also ensures that all renaming and casting is captured in the staging and allows for simpler, downstream models

I've mocked up what this could look like for HubSpot deals but this could also be used for the company, contact, and soon-to-come ticket models.

Sample var for deal columns:

dbt_project.yml

...

vars:

  deal_pass_through_columns:
  
        - name: "property_sales_development_representative"
          alias: "sales_development_representative_owner_id"
  
        - name: "property_energy_consultant"
          alias: "energy_consultant_owner_id"

@fivetran-joemarkiewicz
Copy link
Contributor

Closing this PR as it was incorporated within PR #30.

Thanks @jamesrayoub for your valued contribution, we really appreciate it!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants