Employ base tables to resolve unioning-null issue #25

fivetran-jamie · 2025-02-19T23:48:23Z

PR Overview

This PR will address the following Issue/Feature:
#22

This PR will result in the following new package version:

v0.1.0-a7

Please provide the finalized CHANGELOG entry which details the relevant changes included in this PR:

TO BE FINALIZED AFTER APPROVAL

Bug Fixes

Fixed an issue in which unioned source connections were producing null models.
- The solution required the addition of a base staging model layer. For each staging model, there is a *_base counterpart in which we are running our union_data macro. This framework is necessary to the cooperation of our unioning and column-filling macros, which ensure the models do not fail if you are missing an expected column.
- For each connector type, this adds:
  - 10 more models if Hubspot is enabled
  - 5 more models if Jira is enabled
  - 3 more models if Zendsk is enabled

PR Checklist

Basic Validation

Please acknowledge that you have successfully performed the following commands locally:

dbt run –full-refresh && dbt test
dbt run (if incremental models are present) && dbt test

Before marking this PR as "ready for review" the following have been applied:

The appropriate issue has been linked, tagged, and properly assigned
All necessary documentation and version upgrades have been applied
docs were regenerated (unless this PR does not include any code or yml updates)
BuildKite integration tests are passing
Detailed validation steps have been provided below

Detailed Validation

Please share any and all of your validation steps:

Running on prod using the the union schema/db variables (without the single schema variable set) on test data:

Running the same configs with this branch:

As for the consistency data validation tests, I am getting a peculiar error that makes it seem like this PR is resolving another bug 🤔

This is an example chunk from prod. Note the UNKOWN comment details

Deal Name : test_deal_3

Created By : UNKNOWN (test_contact_1053@test.com)
Created On : 2022-11-06 09:07:13.862+00
Company Name: test_company_3
Engagement Type: NOTE


## COMMENTS

Engagement type: Note
### message from UNKNOWN (UNKNOWN)
##### sent @ 2023-03-04 14:26:46.392+00
n88

The same record in dev differs only in that it has known comment details

Deal Name : test_deal_3

Created By : UNKNOWN (test_contact_1053@test.com)
Created On : 2022-11-06 09:07:13.862+00
Company Name: test_company_3
Engagement Type: NOTE


## COMMENTS

Engagement type: Note
### message from Fivetran Developers (email@name.com)
##### sent @ 2023-03-04 14:26:46.392+00
n88

This is the case for all failed rows. They are different on dev bc they are more seemingly complete

If you had to summarize this PR in an emoji, which would it be?

💒

fivetran-joemarkiewicz

@fivetran-jamie I haven't finished the review, but wanted to call out this one comment regarding the join to make sure we're taking the appropriate approach.

fivetran-joemarkiewicz · 2025-02-21T22:13:30Z

models/unstructured/rag_hubspot__document.sql

@@ -29,7 +29,7 @@ final as (
            "grouped.comments_group_markdown"]) }}
            as chunk
    from deal_document
-    join grouped
+    inner join grouped


It looks like in PR #24 @levonkorganyan is proposing to change this to a left join instead. Let's sync to make sure we're making the right update.

Bumping this message as this is the only other remaining question in this PR before approval.

pinged Levon in his PR

fivetran-joemarkiewicz

@fivetran-jamie this PR looks great! I have two callouts that I'd like to be looked into before approval. Let me know if you have any questions!

CHANGELOG.md

fivetran-joemarkiewicz · 2025-02-24T16:28:46Z

models/unstructured/rag_hubspot__document.sql

@@ -29,7 +29,7 @@ final as (
            "grouped.comments_group_markdown"]) }}
            as chunk
    from deal_document
-    join grouped
+    inner join grouped


Bumping this message as this is the only other remaining question in this PR before approval.

fivetran-joemarkiewicz

LGTM!

CHANGELOG.md

Co-authored-by: Joe Markiewicz <74217849+fivetran-joemarkiewicz@users.noreply.github.com>

CHANGELOG.md

fivetran-avinash · 2025-02-28T19:26:49Z

CHANGELOG.md

+  - `int_rag_hubspot__deal_document`
+  - `int_rag_jira__issue_document`
+  - `int_rag_zendesk__ticket_document`
+


@fivetran-jamie Since Levon contributed this PRto this release, we should probably add him as a contributor?

CHANGELOG.md

fivetran-avinash

@fivetran-jamie Some questions before final approval.

Co-authored-by: Avinash Kunnath <108772760+fivetran-avinash@users.noreply.github.com>

fivetran-avinash

lgtm

fivetran-jamie added 8 commits February 18, 2025 15:00

added to hubspot

658b016

add tmp models to jira and zendesk

be51f7a

try this out in bk

69ad3cc

forgot one zendesk model

ee77bf8

bk

1e99fe1

revert and document

daa0fc0

typo

99af7a8

docs

19822e6

fivetran-jamie linked an issue Feb 20, 2025 that may be closed by this pull request

[Bug] Source not working as expected when using union feature and default single schema not present #22

Closed

4 tasks

readme upgrade

37af711

fivetran-jamie marked this pull request as ready for review February 20, 2025 20:34

fivetran-jamie requested a review from fivetran-joemarkiewicz February 20, 2025 20:35

fivetran-joemarkiewicz assigned fivetran-jamie Feb 21, 2025

fivetran-joemarkiewicz reviewed Feb 21, 2025

View reviewed changes

fivetran-joemarkiewicz requested changes Feb 24, 2025

View reviewed changes

fivetran-jamie added 4 commits February 24, 2025 13:51

owner update

1463066

levon's stuff

2a37dde

coalesces

7a712f0

include records without comments in jira and zendesk as well

b6b03cd

fivetran-joemarkiewicz approved these changes Feb 28, 2025

View reviewed changes

CHANGELOG.md Outdated Show resolved Hide resolved

fivetran-jamie and others added 2 commits February 28, 2025 08:32

license

3ff855a

Update CHANGELOG.md

638dc4d

Co-authored-by: Joe Markiewicz <74217849+fivetran-joemarkiewicz@users.noreply.github.com>

fivetran-avinash reviewed Feb 28, 2025

View reviewed changes

CHANGELOG.md Outdated Show resolved Hide resolved

fivetran-avinash reviewed Feb 28, 2025

View reviewed changes

CHANGELOG.md Outdated Show resolved Hide resolved

fivetran-avinash reviewed Feb 28, 2025

View reviewed changes

fivetran-jamie and others added 2 commits February 28, 2025 16:14

Apply suggestions from code review

1bec0ce

Co-authored-by: Avinash Kunnath <108772760+fivetran-avinash@users.noreply.github.com>

avinash feedback

84ca146

fivetran-avinash approved these changes Mar 1, 2025

View reviewed changes

fivetran-jamie merged commit 11a445c into main Mar 3, 2025
8 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Employ base tables to resolve unioning-null issue #25

Employ base tables to resolve unioning-null issue #25

fivetran-jamie commented Feb 19, 2025 •

edited by fivetran-joemarkiewicz

Loading

fivetran-joemarkiewicz left a comment

fivetran-joemarkiewicz Feb 21, 2025

fivetran-joemarkiewicz Feb 24, 2025

fivetran-jamie Feb 24, 2025

fivetran-joemarkiewicz left a comment

fivetran-joemarkiewicz Feb 24, 2025

fivetran-joemarkiewicz left a comment

fivetran-avinash Feb 28, 2025

fivetran-jamie Mar 1, 2025

fivetran-avinash left a comment

fivetran-avinash left a comment

Employ base tables to resolve unioning-null issue #25

Employ base tables to resolve unioning-null issue #25

Conversation

fivetran-jamie commented Feb 19, 2025 • edited by fivetran-joemarkiewicz Loading

PR Overview

Bug Fixes

PR Checklist

Basic Validation

Detailed Validation

If you had to summarize this PR in an emoji, which would it be?

fivetran-joemarkiewicz left a comment

Choose a reason for hiding this comment

fivetran-joemarkiewicz Feb 21, 2025

Choose a reason for hiding this comment

fivetran-joemarkiewicz Feb 24, 2025

Choose a reason for hiding this comment

fivetran-jamie Feb 24, 2025

Choose a reason for hiding this comment

fivetran-joemarkiewicz left a comment

Choose a reason for hiding this comment

fivetran-joemarkiewicz Feb 24, 2025

Choose a reason for hiding this comment

fivetran-joemarkiewicz left a comment

Choose a reason for hiding this comment

fivetran-avinash Feb 28, 2025

Choose a reason for hiding this comment

fivetran-jamie Mar 1, 2025

Choose a reason for hiding this comment

fivetran-avinash left a comment

Choose a reason for hiding this comment

fivetran-avinash left a comment

Choose a reason for hiding this comment

fivetran-jamie commented Feb 19, 2025 •

edited by fivetran-joemarkiewicz

Loading