Release branch: v0.14.0 #139

fivetran-avinash · 2024-09-18T21:52:18Z

PR Overview

This PR will address the following Issue/Feature: #130, #109

This PR will result in the following new package version: 0.14.0

Please provide the finalized CHANGELOG entry which details the relevant changes included in this PR:

Breaking Changes (Full refresh required after upgrading)

Partitioned models have had the partition_by logic adjusted to include a granularity of a month. This change should only impact BigQuery warehouses and was applied to avoid the common too many partitions error users have experienced due to over-partitioning by day. Therefore, adjusting the partition to a monthly granularity will increase the partition windows and allow for more performant querying.
This change was applied to the following models:
- int_netsuite2__tran_with_converted_amounts
- netsuite2__balance_sheet
- netsuite2__income_statement
- netsuite2__transaction_details

Upstream Netsuite Source Breaking Changes (Full refresh required after upgrading)

Casted specific timestamp fields across all staging models as dates where the Netsuite UI does not perform timezone conversion. Keeping these fields as type timestamp causes issues in reporting tools that perform automatic timezone conversion.
Adds additional commonly used fields within the stg_netsuite2__* models. IMPORTANT: Nearly all of these models where fields are being added have pass-through functionality. So you will need to remove these fields from your passthrough variable setup if they are currently present to avoid errors.
Please refer to the v0.11.0 dbt_netsuite_source release for more details regarding the upstream changes.

Bug Fixes

Updates logic in netsuite2__transaction_details to select the appropriate customer and vendor values based on the whether the transaction type is a customer invoice or credit, or a vendor bill or credit.
- Customer fields impacted: company_name, customer_city, customer_state, customer_zipcode, customer_country, customer_date_first_order, customer_external_id.
- Vendor fields impacted: vendor_category_name, vendor_name, vendor_create_date.

Feature Updates

New fields customer_alt_name and vendor_alt_name were introduced into netsuite2__transaction_details, after being added into the stg_netsuite2__customers and stg_netsuite2__vendors models in the most recent release of dbt_netsuite. You can view the release notes for more details.
We added the employee model in the v0.11.0 release of dbt_netsuite_source, which will materialize stg_netsuite2__employees from the source package by default.
- Since this model is only used by a subset of customers, we've introduced the variable netsuite2__using_employees to allow users who don't utilize the employee table in Netsuite2 the ability to disable that functionality within your dbt_project.yml. Instructions are available in the README.

Under the Hood

Consistency tests added for each Netsuite2 end model in order to be used during integration test validations.

Contributors

@jmongerlyra (PR #131)

PR Checklist

Basic Validation

Please acknowledge that you have successfully performed the following commands locally:

dbt run –full-refresh && dbt test
dbt run (if incremental models are present) && dbt test

Before marking this PR as "ready for review" the following have been applied:

The appropriate issue has been linked, tagged, and properly assigned
All necessary documentation and version upgrades have been applied
docs were regenerated (unless this PR does not include any code or yml updates)
BuildKite integration tests are passing
Detailed validation steps have been provided below

Detailed Validation

Please share any and all of your validation steps:

If you had to summarize this PR in an emoji, which would it be?

💃

Update customer and vendor joins

…com/fivetran/dbt_netsuite into bugfix/update-customer-vendor-joins

…oins [Bug Fix]: Update customer and vendor joins in transaction details

* bugfix/too-many-partitions * validation schema addition * docs regen and final changes before review * intermediate model conditional update * Apply suggestions from code review Co-authored-by: Avinash Kunnath <108772760+fivetran-avinash@users.noreply.github.com> * removing conditional logic --------- Co-authored-by: Avinash Kunnath <108772760+fivetran-avinash@users.noreply.github.com>

fivetran-jamie · 2024-09-19T17:52:20Z

CHANGELOG.md

+
+## Upstream Netsuite Source Breaking Changes (Full refresh required after upgrading)
+- Casted specific timestamp fields across all staging models as dates where the Netsuite UI does not perform timezone conversion. Keeping these fields as type timestamp causes issues in reporting tools that perform automatic timezone conversion.   
+- Adds additional commonly used fields within the `stg_netsuite2__*` models. **IMPORTANT**: Nearly all of these models where fields are being added have pass-through functionality. So you will need to remove these fields from your passthrough variable setup if they are currently present to avoid errors.


Suggested change

- Adds additional commonly used fields within the `stg_netsuite2__*` models. **IMPORTANT**: Nearly all of these models where fields are being added have pass-through functionality. So you will need to remove these fields from your passthrough variable setup if they are currently present to avoid errors.

- Adds additional commonly used fields within the `stg_netsuite2__*` models.

> **IMPORTANT**: Nearly all of the affected models have pass-through functionality. If you have already been using passthrough column variables to include the newly added fields (without aliases), you **MUST** remove the fields from your passthrough variable configuration in order to avoid duplicate column errors.

not sure how the > section looks within a bulleted list but just wanna be as clear as possible

fivetran-jamie · 2024-09-19T17:57:11Z

CHANGELOG.md

+  - Vendor fields impacted: `vendor_category_name`, `vendor_name`, `vendor_create_date`.
+
+## Feature Updates
+- New fields `customer_alt_name` and `vendor_alt_name` were introduced into `netsuite2__transaction_details`, after being added into the `stg_netsuite2__customers` and `stg_netsuite2__vendors` models in the most recent release of `dbt_netsuite`. [You can view the release notes](https://github.com/fivetran/dbt_netsuite/releases/tag/v0.14.0) for more details.


should the release notes point to https://github.com/fivetran/dbt_netsuite_source/releases/tag/v0.11.0?

Hmm, actually, this shouldn't be here at all, since a link to the source release notes are on the next line.

It's a little repetitive but I'll keep it in.

fivetran-jamie · 2024-09-19T17:57:42Z

CHANGELOG.md

+## Feature Updates
+- New fields `customer_alt_name` and `vendor_alt_name` were introduced into `netsuite2__transaction_details`, after being added into the `stg_netsuite2__customers` and `stg_netsuite2__vendors` models in the most recent release of `dbt_netsuite`. [You can view the release notes](https://github.com/fivetran/dbt_netsuite/releases/tag/v0.14.0) for more details.
+- We added the `employee` model in the [`v0.11.0` release of `dbt_netsuite_source`](https://github.com/fivetran/dbt_netsuite_source/releases/tag/v0.11.0), which will materialize `stg_netsuite2__employees` from the source package by default.
+  - Since this model is only used by a subset of customers, we've introduced the variable `netsuite2__using_employees` to allow users who don't utilize the `employee` table in Netsuite2 the ability to disable that functionality within your `dbt_project.yml`. [Instructions are available in the README](https://github.com/fivetran/dbt_netsuite/?tab=readme-ov-file#step-5-disable-models-for-non-existent-sources-netsuite2-only).


would be good to mention this is True by default and dynamically enabled for Quickstart users

fivetran-jamie · 2024-09-19T18:00:48Z

packages.yml

+  # - package: fivetran/netsuite_source
+  #   version: [">=0.10.0", "<0.11.0"]
+
+- git: https://github.com/fivetran/dbt_netsuite_source.git


reminder to switch

packages.yml

fivetran-jamie · 2024-09-19T18:02:07Z

CHANGELOG.md

+
+## Upstream Netsuite Source Breaking Changes (Full refresh required after upgrading)
+- Casted specific timestamp fields across all staging models as dates where the Netsuite UI does not perform timezone conversion. Keeping these fields as type timestamp causes issues in reporting tools that perform automatic timezone conversion.   
+- Adds additional commonly used fields within the `stg_netsuite2__*` models. **IMPORTANT**: Nearly all of these models where fields are being added have pass-through functionality. So you will need to remove these fields from your passthrough variable setup if they are currently present to avoid errors.


also which end models had new fields added to them? could be good to highlight here (or are they in the feature updates section?)

This will be called out in the dbt_netsuite_source release since the changes are being made there.

I linked to it in line 17, but I updated the wordage to make it clearer where to find these fields.

fivetran-avinash

@fivetran-jamie Release notes addressed.

fivetran-jamie

lgtm!

jmongerlyra and others added 15 commits March 8, 2024 21:17

Update customer and vendor joins

3621556

customer vendor joins

1dd189c

Merge pull request #131 from jmongerlyra/lyra_cust_vend

15bcce3

Update customer and vendor joins

seeds, docs, test

499cb2f

Merge branch 'bugfix/update-customer-vendor-joins' of https://github.…

c417ec4

…com/fivetran/dbt_netsuite into bugfix/update-customer-vendor-joins

schema bump

2fc4edd

final fixes before PR review

aad186a

tweak schema in integration tests

208ffe6

remove run results

49b767b

PR review updates

6004c75

update quickstart.yml

4a134e1

update run_models.sh

6dac4f1

release-branch/v0.14.0

980bd93

Merge pull request #138 from fivetran/bugfix/update-customer-vendor-j…

a6bd744

…oins [Bug Fix]: Update customer and vendor joins in transaction details

fivetran-avinash self-assigned this Sep 18, 2024

regen docs

c497113

fivetran-jamie reviewed Sep 19, 2024

View reviewed changes

release review

8018f81

fivetran-avinash commented Sep 19, 2024

View reviewed changes

fivetran-jamie approved these changes Sep 23, 2024

View reviewed changes

fivetran-avinash merged commit 0300d33 into main Sep 25, 2024
9 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Release branch: v0.14.0 #139

Release branch: v0.14.0 #139

fivetran-avinash commented Sep 18, 2024 •

edited

Loading

fivetran-jamie Sep 19, 2024

fivetran-jamie Sep 19, 2024

fivetran-jamie Sep 19, 2024

fivetran-avinash Sep 19, 2024 •

edited

Loading

fivetran-avinash Sep 19, 2024

fivetran-jamie Sep 19, 2024

fivetran-jamie Sep 19, 2024

fivetran-jamie Sep 19, 2024

fivetran-avinash Sep 19, 2024

fivetran-avinash left a comment

fivetran-jamie left a comment

	- Adds additional commonly used fields within the `stg_netsuite2__` models. IMPORTANT*: Nearly all of these models where fields are being added have pass-through functionality. So you will need to remove these fields from your passthrough variable setup if they are currently present to avoid errors.
	- Adds additional commonly used fields within the `stg_netsuite2__*` models.
	> IMPORTANT: Nearly all of the affected models have pass-through functionality. If you have already been using passthrough column variables to include the newly added fields (without aliases), you MUST remove the fields from your passthrough variable configuration in order to avoid duplicate column errors.

Release branch: v0.14.0 #139

Release branch: v0.14.0 #139

Conversation

fivetran-avinash commented Sep 18, 2024 • edited Loading

PR Overview

Breaking Changes (Full refresh required after upgrading)

Upstream Netsuite Source Breaking Changes (Full refresh required after upgrading)

Bug Fixes

Feature Updates

Under the Hood

Contributors

PR Checklist

Basic Validation

Detailed Validation

If you had to summarize this PR in an emoji, which would it be?

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fivetran-avinash Sep 19, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fivetran-avinash left a comment

Choose a reason for hiding this comment

fivetran-jamie left a comment

Choose a reason for hiding this comment

fivetran-avinash commented Sep 18, 2024 •

edited

Loading

fivetran-avinash Sep 19, 2024 •

edited

Loading