bugfix/union-data-no-union-source

[fivetran-joemarkiewicz]

PR Overview

This PR will address the following Issue/Feature: #107

This PR will result in the following new package version: v0.4.5

This will not be a breaking change as it will only make an adjustment to the union_data macro to ensure source connection when the macro is utilized and unioning is not needed.

Please detail what change(s) this PR introduces and any additional information that should be known during the review of this PR:

Currently the union_data macro works in three scenarios:

1. A package user is leveraging the [connector]_union_schemas variable to union multiple projects across schemas.
2. A package user is leveraging the [connector]_union_databases variable to union multiple projects across databases.
3. Is not trying to union multiple projects and just wants to use one schema.

The macro currently works seamlessly by connecting the source to the downstream models for the union variable use cases. However, the macro does not work to connect the source if only the one schema is being used. This is due to the "else" logic in the model not pointing to the source like the other versions of the macro.

This is not a glaring error (it will not actually cause runs to fail), but it does have an adverse affect on the generated docs. Since the relation is not leveraging the source macro, the package will generate the models with floating sources (see the pic below where I am using QuickBooks as an example).

This is particularly not ideal because it also has an effect on Fivetran Transformations since it is not able to establish a connection to the upstream connector (as the source.yml file is utilized to create this connection). See the below pics for the connector not being identified in the Fivetran Transformations portal. In particular notice how the Connectors column shows none indicating the connector link has not been established.

As seen above, this means integrated scheduling will not work and it will create a poor user experience since it seems to be disconnected (although it isn't and will still succeed).

The changes in this PR make an update to the "else" conditional in the union_data macro to ensure a source connection may be established when a package user is not unioning datasets, but instead just using one. The reason this section of the macro didn't work in the previous build is because we were leveraging only the vars. This allowed us to make a connection and read data from the proper place. However, it did not actually make a link to the source. Therefore, in the updates within this PR we are explicitly leveraging the source function and using the variables as well to establish the source connection.

PR Checklist

Basic Validation

Please acknowledge that you have successfully performed the following commands locally:

• dbt compile
• dbt run –full-refresh
• dbt run
• dbt test
• dbt run –vars (if applicable)

Before marking this PR as “ready for review” the following have been applied:

• The appropriate issue has been linked and tagged
• You are assigned to the corresponding issue and this PR
• BuildKite integration tests are passing

Detailed Validation

Please acknowledge that the following validation checks have been performed prior to marking this PR as “ready for review”:

• You have validated these changes and assure this PR will address the respective Issue/Feature.
• You are reasonably confident these changes will not impact any other components of this package or any dependent packages.
• You have provided details below around the validation steps performed to gain confidence in these changes.

To validate these changes I performed the following:

• Installed the latest version of the quickbooks package and confirmed the issue (see pics above).
• Installed the below version of the quickbooks package which installs this version of fivetran_utils as opposed to the latest version.

packages:
  - git: https://github.com/fivetran/dbt_quickbooks.git
    revision: test/union-data-source
    warn-unpinned: false 

• Ran the package and generated the docs using the updated fivetran_utils from this branch
• Confirmed the sources are now properly linked (see screenshot below and notice how QuickBooks is now shown in the DLG and as a connected connector in the transformation jobs).
  
• Additionally, pushed the changes to my hosted repo for Fivetran Transformations and confirmed the connector now shows as connected and integrated scheduling works now (see the screenshots below)!
  
  

Standard Updates

Please acknowledge that your PR contains the following standard updates:

• Package versioning has been appropriately indexed in the following locations:
  • indexed within dbt_project.yml
  • indexed within integration_tests/dbt_project.yml
• CHANGELOG has individual entries for each respective change in this PR
• README updates have been applied (if applicable)
• [N/A] DECISIONLOG updates have been updated (if applicable)
• [N/A] Appropriate yml documentation has been added (if applicable)

dbt Docs

Please acknowledge that after the above were all completed the below were applied to your branch:

• [N/A] docs were regenerated (unless this PR does not include any code or yml updates)

If you had to summarize this PR in an emoji, which would it be?

🔗

[fivetran-jamie]

looks great to me! just left one comment

also i realize now that the error could've been anticipated by looking at the generated docs, but it's much clearer to be an actual issue in a Quickstart environment. When deploying changes to packages that have Quickstart models enabled, is there a way we can test out a working branch in a quickstart or quickstart-like enviroment?

[fivetran-joemarkiewicz]

When deploying changes to packages that have Quickstart models enabled, is there a way we can test out a working branch in a quickstart or quickstart-like enviroment?

This is a great question and possibly something we should consider. We would likely need to work with other teams but I feel this would be helpful in ensuring their are no unseen bugs that could go unnoticed until being deployed in QuickStart.