Release v0.15.0: Bug fixes for account numbers, addition of commonly …

…used fields and foreign keys (#144)

Release v0.15.0

* Fix account_number, add common fields, and subsidiaries_pass_through_columns (#136)

* Fix account_number for system generated accounts.

* Add subsidiaries pass through columns

* Add additional fields

* Fix is_account_intercompany on balance_sheet

* Fix account_number

* Customer PR adding locations fields (#146)
---------
Co-authored-by: Avinash Kunnath <108772760+fivetran-avinash@users.noreply.github.com> 
Co-authored-by: Jared Monger <95247581+jmongerlyra@users.noreply.github.com>
Co-authored-by: Antonio Ruby Barreto <antonio.ruby.barreto@fastnedcharging.com>

CHANGELOG.md

@@ -1,3 +1,49 @@
+# dbt_netsuite v0.15.0
+For Netsuite2, [PR #144](https://github.com/fivetran/dbt_netsuite/pull/144) includes the following updates: 
+
+## Breaking Changes (Full refresh required after upgrading)
+- Corrected `account_number` field logic for the `netsuite2__balance_sheet` model to match the native Balance Sheet report within Netsuite:
+  - Income statement accounts should use the account number of the system-generated retained earnings account. 
+  - Cumulative Translation Adjustment (CTA) accounts should use the account number of the system-generated CTA account.
+  - We modified the logic to ensure the account number is the retained earnings number for income statement accounts in the balance sheet, and CTA rather than null. 
+  - Since this will change the `account_number`, a `--full-refresh` after upgrading will be required. 
+
+## New Fields
+- Added commonly used fields to each end model. They are listed in the below table.
+- Also added foreign keys to each end model to make it easier for customers to join back to source tables for better insights.
+
+| **Models**                | **New Fields**                                                                                                                                |
+| ------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------- |
+| [netsuite2__transaction_details](https://fivetran.github.io/dbt_netsuite/#!/model/model.netsuite.netsuite2__transaction_details)             | New fields:  `is_reversal`, `reversal_transaction_id`, `reversal_date`, `is_reversal_defer`, `is_eliminate`, `exchange_rate`, `department_full_name`,  `subsidiary_full_name`, `subsidiary_currency_symbol`, `transaction_line_amount`, `account_display_name`  <br> <br> New keys: `customer_id`, `vendor_id`, `class_id`, `location_id`, `department_id`, `currency_id`, `parent_account_id`, `vendor_category_id` (if `netsuite2__using_vendor_categories` is enabled)  |
+| [netsuite2__balance_sheet](https://fivetran.github.io/dbt_netsuite/#!/model/model.netsuite.netsuite2__balance_sheet)            | New fields: `account_display_name`, `subsidiary_full_name`, `is_account_intercompany`,  `is_account_leftside` |
+| [netsuite2__income_statement](https://fivetran.github.io/dbt_netsuite/#!/model/model.netsuite.netsuite2__income_statement)             |  New fields: `account_display_name` <br> <br>   New keys: `class_id`, `location_id`, `department_id` |
+
+
+> **IMPORTANT**: All of the affected models have pass-through functionality. If you have already been using passthrough column variables to include the newly added fields (without aliases), you **MUST** remove the fields from your passthrough variable configuration in order to avoid duplicate column errors.
+
+## Feature Updates
+- You can now leverage passthrough columns in `netsuite2__transaction_details` to bring in additional fields from the `locations` and `subsidiaries` source tables. 
+- To add additional columns to this model, do so by adding our pass-through column variables `locations_pass_through_columns` and `subsidiaries_pass_through_columns` to your `dbt_project.yml` file:
+
+```yml
+vars:
+    locations_pass_through_columns: 
+        - name: "location_custom_field"
+    subsidiaries_pass_through_columns: 
+        - name: "sub_field"
+          alias: "subsidiary_field"
+```
+
+- For more details on how to passthrough columns, [please consult our README section](https://github.com/fivetran/dbt_netsuite/blob/main/README.md#passing-through-additional-fields). 
+
+## Under the Hood
+- Additional consistency tests added for each Netsuite2 end model in order to be used during integration test validations.
+- Updated yml documentation with new fields.
+
+## Contributors
+- [@jmongerlyra](https://github.com/jmongerlyra) ([PR #136](https://github.com/fivetran/dbt_netsuite/pull/136))
+- [@fastbarreto](https://github.com/fastbarreto) ([PR #124](https://github.com/fivetran/dbt_netsuite/pull/124))
+
 # dbt_netsuite v0.14.0
 
 For Netsuite2, [PR #138](https://github.com/fivetran/dbt_netsuite/pull/138) and [PR #132](https://github.com/fivetran/dbt_netsuite/pull/132) include the following updates: 
@@ -72,7 +118,7 @@ For Netsuite2, [PR #114](https://github.com/fivetran/dbt_netsuite/pull/114) incl
   - transaction_number
 - ❗Note: If you have already added any of these fields as passthrough columns to the `transactions_pass_through_columns`, `transaction_lines_pass_through_columns`, `accounts_pass_through_columns`, or `departments_pass_through_columns` vars, you will need to remove or alias these fields from the var to avoid duplicate column errors.
 
-- Removed the unnecessary reference to `entities` in the `netsuit2__transaction_details` model.
+- Removed the unnecessary reference to `entities` in the `netsuite2__transaction_details` model.
 
 ## 📝 Documentation Update 📝
 - [Updated DECISIONLOG](https://github.com/fivetran/dbt_netsuite/blob/main/DECISIONLOG.md#why-converted-transaction-amounts-are-null-if-they-are-non-posting) with our reasoning for why we don't bring in future-facing transactions and leave the `converted_amount` in transaction details empty. ([#115](https://github.com/fivetran/dbt_netsuite/issues/115))

README.md

@@ -101,7 +101,7 @@ Include the following netsuite package version in your `packages.yml` file:
 ```yaml
 packages:
   - package: fivetran/netsuite
-    version: [">=0.14.0", "<0.15.0"]
+    version: [">=0.15.0", "<0.16.0"]
 ```
 ### Step 3: Define Netsuite.com or Netsuite2 Source
 As of April 2022 Fivetran made available a new Netsuite connector which leverages the Netsuite2 endpoint opposed to the original Netsuite.com endpoint. This package is designed to run for either or, not both. By default the `netsuite_data_model` variable for this package is set to the original `netsuite` value which runs the netsuite.com version of the package. If you would like to run the package on Netsuite2 data, you may adjust the `netsuite_data_model` variable to run the `netsuite2` version of the package.

dbt_project.yml

@@ -1,6 +1,6 @@
 config-version: 2
 name: 'netsuite'
-version: '0.14.0'
+version: '0.15.0'
 require-dbt-version: [">=1.3.0", "<2.0.0"]
 
 models:
@@ -60,6 +60,8 @@ vars:
     accounts_pass_through_columns: []
     classes_pass_through_columns: []
     departments_pass_through_columns: []
+    locations_pass_through_columns: []
+    subsidiaries_pass_through_columns: []
     transactions_pass_through_columns: []
     transaction_lines_pass_through_columns: []
     balance_sheet_transaction_detail_columns: []

integration_tests/ci/sample.profiles.yml

@@ -16,13 +16,13 @@ integration_tests:
       pass: "{{ env_var('CI_REDSHIFT_DBT_PASS') }}"
       dbname: "{{ env_var('CI_REDSHIFT_DBT_DBNAME') }}"
       port: 5439
-      schema: netsuite_integration_tests_7
+      schema: netsuite_integration_tests_8
       threads: 8
     bigquery:
       type: bigquery
       method: service-account-json
       project: 'dbt-package-testing'
-      schema: netsuite_integration_tests_7
+      schema: netsuite_integration_tests_8
       threads: 8
       keyfile_json: "{{ env_var('GCLOUD_SERVICE_KEY') | as_native }}"
     snowflake:
@@ -33,7 +33,7 @@ integration_tests:
       role: "{{ env_var('CI_SNOWFLAKE_DBT_ROLE') }}"
       database: "{{ env_var('CI_SNOWFLAKE_DBT_DATABASE') }}"
       warehouse: "{{ env_var('CI_SNOWFLAKE_DBT_WAREHOUSE') }}"
-      schema: netsuite_integration_tests_7
+      schema: netsuite_integration_tests_8
       threads: 8
     postgres:
       type: postgres
@@ -42,13 +42,13 @@ integration_tests:
       pass: "{{ env_var('CI_POSTGRES_DBT_PASS') }}"
       dbname: "{{ env_var('CI_POSTGRES_DBT_DBNAME') }}"
       port: 5432
-      schema: netsuite_integration_tests_7
+      schema: netsuite_integration_tests_8
       threads: 8
     databricks:
       catalog: "{{ env_var('CI_DATABRICKS_DBT_CATALOG') }}"
       host: "{{ env_var('CI_DATABRICKS_DBT_HOST') }}"
       http_path: "{{ env_var('CI_DATABRICKS_DBT_HTTP_PATH') }}"
-      schema: netsuite_integration_tests_7
+      schema: netsuite_integration_tests_8
       threads: 2
       token: "{{ env_var('CI_DATABRICKS_DBT_TOKEN') }}"
       type: databricks

integration_tests/dbt_project.yml

@@ -1,5 +1,5 @@
 name: 'netsuite_integration_tests'
-version: '0.14.0'
+version: '0.15.0'
 profile: 'integration_tests'
 config-version: 2
 
@@ -8,10 +8,10 @@ models:
   # +schema: "netsuite_{{ var('directed_schema','dev') }}"  ## To be used for validation tests
 
 vars:
-  netsuite_schema: netsuite_integration_tests_7
+  netsuite_schema: netsuite_integration_tests_8
   netsuite_data_model_override: netsuite
 
-  # Enable below when generating docs
+  # # Enable below when generating docs
   # netsuite2__multibook_accounting_enabled: true
   # netsuite2__using_to_subsidiary: true
 

integration_tests/tests/consistency/consistency_balance_sheet_amounts.sql

@@ -0,0 +1,47 @@
+{{ config(
+    tags="fivetran_validations",
+    enabled=var('fivetran_validation_tests_enabled', false)
+) }}
+
+with prod as (
+    select 
+        case when account_id is null then -999999 else account_id end as account_id,
+        date_trunc(accounting_period_ending, month) as prod_account_period_month,
+        count(*) as prod_row_count,
+        sum(converted_amount) as prod_converted_amount 
+    from {{ target.schema }}_netsuite_prod.netsuite2__balance_sheet
+    where date_trunc(accounting_period_ending, month) < date_trunc(current_date(), month) - 1 
+    group by 1, 2
+),
+
+dev as (
+    select 
+        case when account_id is null then -999999 else account_id end as account_id,
+        date_trunc(accounting_period_ending, month) as dev_account_period_month,
+        count(*) as dev_row_count,
+        sum(converted_amount) as dev_converted_amount 
+    from {{ target.schema }}_netsuite_dev.netsuite2__balance_sheet
+    where date_trunc(accounting_period_ending, month) < date_trunc(current_date(), month) - 1 
+    group by 1, 2
+),
+
+final as (
+    select
+        prod.account_id,
+        prod.prod_account_period_month,
+        dev.dev_account_period_month,
+        prod.prod_row_count,
+        dev.dev_row_count,
+        prod.prod_converted_amount,
+        dev.dev_converted_amount
+    from prod
+    full outer join dev
+        on dev.account_id = prod.account_id
+        and dev.dev_account_period_month = prod.prod_account_period_month
+)
+
+
+select *
+from final 
+where abs(prod_converted_amount - dev_converted_amount) >= 0.01
+or abs(prod_row_count - dev_row_count) >= 0.01

integration_tests/tests/consistency/consistency_balance_sheet_count.sql

@@ -0,0 +1,33 @@
+{{ config(
+    tags="fivetran_validations",
+    enabled=var('fivetran_validation_tests_enabled', false)
+) }}
+
+with prod as (
+    select 
+        1 as join_key,
+        count(*) as total_balance_sheet_prod_rows
+    from {{ target.schema }}_netsuite_prod.netsuite2__balance_sheet
+    group by 1
+),
+
+dev as (
+    select 
+        1 as join_key,
+        count(*) as total_balance_sheet_dev_rows
+    from {{ target.schema }}_netsuite_dev.netsuite2__balance_sheet
+    group by 1
+),
+
+final as (
+    select
+        total_balance_sheet_prod_rows,
+        total_balance_sheet_dev_rows
+    from prod
+    full outer join dev
+        on dev.join_key = prod.join_key
+)
+
+select *
+from final
+where total_balance_sheet_prod_rows != total_balance_sheet_dev_rows

integration_tests/tests/consistency/consistency_income_statement_amounts.sql

@@ -0,0 +1,37 @@
+{{ config(
+    tags="fivetran_validations",
+    enabled=var('fivetran_validation_tests_enabled', false)
+) }}
+
+with prod as (
+    select 
+        accounting_period_id, 
+        sum(converted_amount) as prod_converted_amount 
+    from {{ target.schema }}_netsuite_prod.netsuite2__income_statement
+    where cast(accounting_period_ending as date) < current_date - 1
+    group by 1
+),
+
+dev as (
+    select 
+        accounting_period_id,
+        sum(converted_amount) as dev_converted_amount 
+    from {{ target.schema }}_netsuite_dev.netsuite2__income_statement
+    where cast(accounting_period_ending as date) < current_date - 1
+    group by 1
+),
+
+final as (
+    select
+        prod.accounting_period_id, 
+        prod.prod_converted_amount,
+        dev.dev_converted_amount
+    from prod
+    full outer join dev
+        on dev.accounting_period_id = prod.accounting_period_id
+)
+
+
+select *
+from final  
+where abs(prod_converted_amount - dev_converted_amount) >= 0.01 

integration_tests/tests/consistency/consistency_income_statement_count.sql

@@ -0,0 +1,33 @@
+{{ config(
+    tags="fivetran_validations",
+    enabled=var('fivetran_validation_tests_enabled', false)
+) }}
+
+with prod as (
+    select 
+        1 as join_key,
+        count(*) as total_income_statement_prod_rows
+    from {{ target.schema }}_netsuite_prod.netsuite2__income_statement
+    group by 1
+),
+
+dev as (
+    select 
+        1 as join_key,
+        count(*) as total_income_statement_dev_rows
+    from {{ target.schema }}_netsuite_dev.netsuite2__income_statement
+    group by 1
+),
+
+final as (
+    select
+        total_income_statement_prod_rows,
+        total_income_statement_dev_rows
+    from prod
+    full outer join dev
+        on dev.join_key = prod.join_key
+)
+
+select *
+from final
+where total_income_statement_prod_rows != total_income_statement_dev_rows

integration_tests/tests/consistency/consistency_transaction_details_count.sql

@@ -0,0 +1,33 @@
+{{ config(
+    tags="fivetran_validations",
+    enabled=var('fivetran_validation_tests_enabled', false)
+) }}
+
+with prod as (
+    select 
+        1 as join_key,
+        count(*) as total_transaction_detail_prod_rows
+    from {{ target.schema }}_netsuite_prod.netsuite2__transaction_details
+    group by 1
+),
+
+dev as (
+    select 
+        1 as join_key,
+        count(*) as total_transaction_detail_dev_rows
+    from {{ target.schema }}_netsuite_dev.netsuite2__transaction_details
+    group by 1
+),
+
+final as (
+    select
+        total_transaction_detail_prod_rows,
+        total_transaction_detail_dev_rows
+    from prod
+    full outer join dev
+        on dev.join_key = prod.join_key
+)
+
+select *
+from final
+where total_transaction_detail_prod_rows != total_transaction_detail_dev_rows