diff --git a/.github/PULL_REQUEST_TEMPLATE/maintainer_pull_request_template.md b/.github/PULL_REQUEST_TEMPLATE/maintainer_pull_request_template.md index fdc71e6..1e22b09 100644 --- a/.github/PULL_REQUEST_TEMPLATE/maintainer_pull_request_template.md +++ b/.github/PULL_REQUEST_TEMPLATE/maintainer_pull_request_template.md @@ -11,11 +11,12 @@ ### Basic Validation Please acknowledge that you have successfully performed the following commands locally: - [ ] dbt run โ€“full-refresh && dbt test -- [ ] dbt run (if incremental models are present) +- [ ] dbt run (if incremental models are present) && dbt test Before marking this PR as "ready for review" the following have been applied: - [ ] The appropriate issue has been linked, tagged, and properly assigned - [ ] All necessary documentation and version upgrades have been applied + - [ ] docs were regenerated (unless this PR does not include any code or yml updates) - [ ] BuildKite integration tests are passing - [ ] Detailed validation steps have been provided below @@ -23,3 +24,7 @@ Before marking this PR as "ready for review" the following have been applied: ### Detailed Validation Please share any and all of your validation steps: + +### If you had to summarize this PR in an emoji, which would it be? + +:dancer: \ No newline at end of file diff --git a/.github/workflows/auto-release.yml b/.github/workflows/auto-release.yml new file mode 100644 index 0000000..8ed5853 --- /dev/null +++ b/.github/workflows/auto-release.yml @@ -0,0 +1,13 @@ +name: 'auto release' +on: + pull_request: + types: + - closed + branches: + - main + +jobs: + call-workflow-passing-data: + if: github.event.pull_request.merged + uses: fivetran/dbt_package_automations/.github/workflows/auto-release.yml@main + secrets: inherit \ No newline at end of file diff --git a/CHANGELOG.md b/CHANGELOG.md index ffb2807..81b43e6 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,4 +1,18 @@ +# dbt_netsuite v0.13.0 + +## ๐ŸŽ‰ Feature Update ๐ŸŽ‰ +This release supports running the package on multiple Netsuite sources at once! See the [README](https://github.com/fivetran/dbt_netsuite/tree/main?tab=readme-ov-file#step-4-define-database-and-schema-variables) for details on how to leverage this feature ([PR #104](https://github.com/fivetran/dbt_netsuite/pull/104)). +> Please note: For each end model, we have a added a new field, `source_relation`, that points to the source connector from which the record originated. + +## ๐Ÿ› ๏ธ Under the Hood ๐Ÿ› ๏ธ +- Included auto-releaser GitHub Actions workflow to automate future releases. +- Updated the maintainer PR template to resemble the most up to date format. + +## Contributors: +- [@fivetran-poonamagate](https://github.com/fivetran-poonamagate) ([PR #104](https://github.com/fivetran/dbt_netsuite/pull/104)) + # dbt_netsuite v0.12.0 + ## ๐ŸŽ Official release for Netsuite2! ๐ŸŽ [PR #98](https://github.com/fivetran/dbt_netsuite/pull/98) is the official supported release of [dbt_netsuite v0.12.0-b1](https://github.com/fivetran/dbt_netsuite/releases/tag/v0.12.0-b1). diff --git a/README.md b/README.md index e7effb3..f48cdda 100644 --- a/README.md +++ b/README.md @@ -102,7 +102,7 @@ Include the following netsuite package version in your `packages.yml` file: ```yaml packages: - package: fivetran/netsuite - version: [">=0.12.0", "<0.13.0"] + version: [">=0.13.0", "<0.14.0"] ``` ## Step 3: Define Netsuite.com or Netsuite2 Source As of April 2022 Fivetran made available a new Netsuite connector which leverages the Netsuite2 endpoint opposed to the original Netsuite.com endpoint. This package is designed to run for either or, not both. By default the `netsuite_data_model` variable for this package is set to the original `netsuite` value which runs the netsuite.com version of the package. If you would like to run the package on Netsuite2 data, you may adjust the `netsuite_data_model` variable to run the `netsuite2` version of the package. @@ -112,6 +112,7 @@ vars: ``` ## Step 4: Define database and schema variables +### Option 1: Single connector ๐Ÿ’ƒ By default, this package runs using your destination and the `netsuite` schema. If this is not where your Netsuite data is (for example, if your netsuite schema is named `netsuite_fivetran`), add the following configuration to your root `dbt_project.yml` file: ```yml @@ -120,7 +121,54 @@ vars: netsuite_schema: your_schema_name ``` +> **Note**: If you are running the package on one source connector, each model will have a `source_relation` column that is just an empty string. + +### Option 2: Union multiple connectors (Netsuite2 only) ๐Ÿ‘ฏ +If you have multiple Netsuite connectors in Fivetran and would like to use this package on all of them simultaneously, we have provided functionality to do so. The package will union all of the data together and pass the unioned table into the transformations. You will be able to see which source it came from in the `source_relation` column of each model. To use this functionality, you will need to set either the `netsuite_union_schemas` OR `netsuite_union_databases` variables (cannot do both, though a more flexible approach is in the works...) in your root `dbt_project.yml` file: + +```yml +# dbt_project.yml + +vars: + netsuite2_union_schemas: ['netsuite2_usa','netsuite2_canada'] # use this if the data is in different schemas/datasets of the same database/project + netsuite2_union_databases: ['netsuite2_usa','netsuite2_canada'] # use this if the data is in different databases/projects but uses the same schema name +``` + +#### Recommended: Incorporate unioned sources into DAG +By default, this package defines one single-connector source, called `netsuite2`, which will be disabled if you are unioning multiple connectors. This means that your DAG will not include your Netsuite sources, though the package will run successfully. + +To properly incorporate all of your Netsuite2 connectors into your project's DAG: +1. Define each of your sources in a `.yml` file in your project. Utilize the following template for the `source`-level configurations, and, **most importantly**, copy and paste the table and column-level definitions from the package's `src_netsuite2.yml` [file](https://github.com/fivetran/dbt_netsuite_source/blob/main/models/netsuite2/src_netsuite2.yml#L15-L607). + +```yml +# a .yml file in your root project +sources: + - name: # ex: netsuite2_usa + schema: # one of var('netsuite2_union_schemas') if unioning schemas, otherwise just 'netsuite2' + database: # one of var('netsuite2_union_databases') if unioning databases, otherwise whatever DB your netsuite2 schemas all live in + loader: fivetran + loaded_at_field: _fivetran_synced + + freshness: # feel free to adjust to your liking + warn_after: {count: 72, period: hour} + error_after: {count: 168, period: hour} + + tables: # copy and paste from netsuite_source/models/netsuite2/src_netsuite2.yml +``` + +> **Note**: If there are source tables you do not have (see [Step 4](https://github.com/fivetran/dbt_netsuite?tab=readme-ov-file#step-5-disable-models-for-non-existent-sources-netsuite2-only)), you may still include them, as long as you have set the right [variables](https://github.com/fivetran/dbt_netsuite?tab=readme-ov-file#step-5-disable-models-for-non-existent-sources-netsuite2-only) to `False`. Otherwise, you may remove them from your source definitions. + +2. Set the `has_defined_sources` variable (scoped to the `netsuite_source` package) to `True`, like such: +```yml +# dbt_project.yml +vars: + netsuite_source: + has_defined_sources: true +``` + ## Step 5: Disable models for non-existent sources (Netsuite2 only) +> _This step is unnecessary (but still available for use) if you are unioning multiple connectors together in the previous step. That is, the `union_data` macro we use will create completely empty staging models for sources that are not found in any of your Netsuite2 schemas/databases. However, you can still leverage the below variables if you would like to avoid this behavior._ + It's possible that your Netsuite connector does not sync every table that this package expects. If your syncs exclude certain tables, it is because you either don't use that feature in Netsuite or actively excluded some tables from your syncs. To disable the corresponding functionality in the package, you must add the relevant variables. By default, all variables are assumed to be true. Add variables for only the tables you would like to disable: ```yml vars: @@ -134,10 +182,11 @@ vars: > To determine if a table or field is activated by a feature, access the [Records Catalog](https://docs.oracle.com/en/cloud/saas/netsuite/ns-online-help/article_159367781370.html). ## (Optional) Step 6: Additional configurations +
Expand/collapse configurations ### Enable additional features -### Multi-Book (Netsuite2 only) +#### Multi-Book (Netsuite2 only) To include `accounting_book_id` and `accounting_book_name` columns in the end models, set the below variable to `true` in your `dbt_project.yml`. This feature is disabled by default. >โ—Notes: > - If you choose to enable this feature, this will add rows for transactions for any non-primary `accounting_book_id`, and your downstream use cases may need to be adjusted. @@ -148,7 +197,7 @@ vars: netsuite2__multibook_accounting_enabled: true # False by default. ``` -### To Subsidiary (Netsuite2 only) +#### To Subsidiary (Netsuite2 only) To include `to_subsidiary_id` and `to_subsidiary_name` columns in the end models, set the below variable to `true` in your `dbt_project.yml`. This feature is disabled by default. You will also need to be using exchange rates, which is enabled by default. >โ—Notes: @@ -216,13 +265,15 @@ models: +schema: my_new_schema_name # leave blank for just the target_schema ``` -### Change the source table references -If an individual source table has a different name than the package expects, add the table name as it appears in your destination to the respective variable: +### Change the source table references (only if using a single connector) +If an individual source table has a different name than the package expects, add the table name as it appears in your destination to the respective variable (and set `use_table_name_identifer_override` to `true` if using Netsuite2): > IMPORTANT: See this project's [`dbt_project.yml`](https://github.com/fivetran/dbt_netsuite_source/blob/main/dbt_project.yml) variable declarations to see the expected names. ```yml vars: + use_table_name_identifer_override: true # Netsuite2 users must set this to TRUE. default = false + # For all Netsuite source tables netsuite__identifier: your_table_name @@ -233,6 +284,8 @@ vars: ### Override the data models variable This package is designed to run **either** the Netsuite.com or Netsuite2 data models. However, for documentation purposes, an additional variable `netsuite_data_model_override` was created to allow for both data model types to be run at the same time by setting the variable value to `netsuite`. This is only to ensure the [dbt docs](https://fivetran.github.io/dbt_netsuite/) (which is hosted on this repository) is generated for both model types. While this variable is provided, we recommend you do not adjust the variable and instead change the `netsuite_data_model` variable to fit your configuration needs. +
+ ## (Optional) Step 7: Produce Analytics-Ready Reports with Streamlit App (Bigquery and Snowflake users only) For those who want to take their reports a step further, our team has created the [Fivetran Netsuite Streamlit App](https://fivetran-netsuite.streamlit.app/) to generate end model visualizations based off of the reports we created in this package. This way you can replicate much of the reporting you see internally in Netsuite and automate a lot of the work needed to report on your core metrics. @@ -253,7 +306,7 @@ This dbt package is dependent on the following dbt packages. Please be aware tha ```yml packages: - package: fivetran/netsuite_source - version: [">=0.9.0", "<0.10.0"] + version: [">=0.10.0", "<0.11.0"] - package: fivetran/fivetran_utils version: [">=0.4.0", "<0.5.0"] diff --git a/dbt_project.yml b/dbt_project.yml index 61379e3..d74644d 100644 --- a/dbt_project.yml +++ b/dbt_project.yml @@ -1,6 +1,6 @@ config-version: 2 name: 'netsuite' -version: '0.12.0' +version: '0.13.0' require-dbt-version: [">=1.3.0", "<2.0.0"] models: diff --git a/integration_tests/dbt_project.yml b/integration_tests/dbt_project.yml index 36f400c..6b2e490 100644 --- a/integration_tests/dbt_project.yml +++ b/integration_tests/dbt_project.yml @@ -1,5 +1,5 @@ name: 'netsuite_integration_tests' -version: '0.12.0' +version: '0.13.0' profile: 'integration_tests' config-version: 2 models: @@ -7,6 +7,7 @@ models: vars: netsuite_schema: netsuite_integration_tests_5 netsuite_data_model_override: netsuite + use_table_name_identifer_override: true # to use seed identifiers in integration tests # Enable below when generating docs # netsuite2__multibook_accounting_enabled: true diff --git a/models/netsuite2.yml b/models/netsuite2.yml index 124760f..2a120f3 100644 --- a/models/netsuite2.yml +++ b/models/netsuite2.yml @@ -102,7 +102,7 @@ models: - name: to_subsidiary_currency_symbol description: Currency symbol for the consolidated exchange rate. - name: balance_sheet_id - description: Surrogate key hashed on `transaction_line_id`, `transaction_id`, `accounting_period_id`, `account_name`, and `account_id`. Adds `to_subsidiary_id` if using subsidiaries and `accounting_book_id` if using multibook accounting. + description: Surrogate key hashed on `transaction_line_id`, `transaction_id`, `accounting_period_id`, `account_name`, `account_id` and `source_relation`. Adds `to_subsidiary_id` if using subsidiaries and `accounting_book_id` if using multibook accounting. tests: - unique - not_null @@ -172,7 +172,7 @@ models: - name: to_subsidiary_currency_symbol description: Currency symbol for the consolidated exchange rate. - name: income_statement_id - description: Surrogate key hashed on `transaction_line_id`, `transaction_id`, `accounting_period_id`, and `account_name`. Adds `to_subsidiary_id` if using subsidiaries and `accounting_book_id` if using multibook accounting. + description: Surrogate key hashed on `transaction_line_id`, `transaction_id`, `accounting_period_id`, `account_name` and `source_relation`. Adds `to_subsidiary_id` if using subsidiaries and `accounting_book_id` if using multibook accounting. tests: - unique - not_null @@ -300,7 +300,7 @@ models: - name: to_subsidiary_currency_symbol description: Currency symbol for the consolidated exchange rate. - name: transaction_details_id - description: Surrogate key hashed on `transaction_line_id` and `transaction_id`. Adds `to_subsidiary_id` if using subsidiaries and `accounting_book_id` if using multibook accounting. + description: Surrogate key hashed on `transaction_line_id` and `transaction_id`, `source_relation`. Adds `to_subsidiary_id` if using subsidiaries and `accounting_book_id` if using multibook accounting. tests: - unique - not_null \ No newline at end of file diff --git a/models/netsuite2/intermediate/base/int_netsuite2__accounting_periods.sql b/models/netsuite2/intermediate/base/int_netsuite2__accounting_periods.sql index c551bbf..e3f68cb 100644 --- a/models/netsuite2/intermediate/base/int_netsuite2__accounting_periods.sql +++ b/models/netsuite2/intermediate/base/int_netsuite2__accounting_periods.sql @@ -21,6 +21,7 @@ joined as ( from accounting_periods left join accounting_period_fiscal_calendars on accounting_periods.accounting_period_id = accounting_period_fiscal_calendars.accounting_period_id + and accounting_periods.source_relation = accounting_period_fiscal_calendars.source_relation ) select * diff --git a/models/netsuite2/intermediate/base/int_netsuite2__accounts.sql b/models/netsuite2/intermediate/base/int_netsuite2__accounts.sql index d67c22a..73ed639 100644 --- a/models/netsuite2/intermediate/base/int_netsuite2__accounts.sql +++ b/models/netsuite2/intermediate/base/int_netsuite2__accounts.sql @@ -23,6 +23,7 @@ joined as ( from accounts left join account_types on accounts.account_type_id = account_types.account_type_id + and accounts.source_relation = account_types.source_relation ) select * diff --git a/models/netsuite2/intermediate/base/int_netsuite2__customers.sql b/models/netsuite2/intermediate/base/int_netsuite2__customers.sql index 002d0fb..4f7c885 100644 --- a/models/netsuite2/intermediate/base/int_netsuite2__customers.sql +++ b/models/netsuite2/intermediate/base/int_netsuite2__customers.sql @@ -24,6 +24,7 @@ joined as ( from customers left join entity_address on coalesce(customers.default_billing_address_id, customers.default_shipping_address_id) = entity_address.nkey + and customers.source_relation = entity_address.source_relation ) select * diff --git a/models/netsuite2/intermediate/base/int_netsuite2__locations.sql b/models/netsuite2/intermediate/base/int_netsuite2__locations.sql index 0b3d134..b9a72b7 100644 --- a/models/netsuite2/intermediate/base/int_netsuite2__locations.sql +++ b/models/netsuite2/intermediate/base/int_netsuite2__locations.sql @@ -24,6 +24,7 @@ joined as ( from locations left join location_main_address on locations.main_address_id = location_main_address.nkey + and locations.source_relation = location_main_address.source_relation ) select * diff --git a/models/netsuite2/intermediate/base/int_netsuite2__transaction_lines.sql b/models/netsuite2/intermediate/base/int_netsuite2__transaction_lines.sql index 3c709d3..96a963d 100644 --- a/models/netsuite2/intermediate/base/int_netsuite2__transaction_lines.sql +++ b/models/netsuite2/intermediate/base/int_netsuite2__transaction_lines.sql @@ -42,10 +42,12 @@ joined as ( left join transaction_accounting_lines on transaction_lines.transaction_line_id = transaction_accounting_lines.transaction_line_id and transaction_lines.transaction_id = transaction_accounting_lines.transaction_id + and transaction_lines.source_relation = transaction_accounting_lines.source_relation {% if var('netsuite2__multibook_accounting_enabled', false) %} left join accounting_books on accounting_books.accounting_book_id = transaction_accounting_lines.accounting_book_id + and accounting_books.source_relation = transaction_accounting_lines.source_relation union all @@ -65,8 +67,10 @@ joined as ( left join transaction_accounting_lines on transaction_lines.transaction_line_id = transaction_accounting_lines.transaction_line_id and transaction_lines.transaction_id = transaction_accounting_lines.transaction_id + and transaction_lines.source_relation = transaction_accounting_lines.source_relation left join accounting_books on accounting_books.base_book_id = transaction_accounting_lines.accounting_book_id + and accounting_books.source_relation = transaction_accounting_lines.source_relation where accounting_books.base_book_id is not null {% endif %} diff --git a/models/netsuite2/intermediate/int_netsuite2__acctxperiod_exchange_rate_map.sql b/models/netsuite2/intermediate/int_netsuite2__acctxperiod_exchange_rate_map.sql index 723dff7..3575e23 100644 --- a/models/netsuite2/intermediate/int_netsuite2__acctxperiod_exchange_rate_map.sql +++ b/models/netsuite2/intermediate/int_netsuite2__acctxperiod_exchange_rate_map.sql @@ -27,6 +27,15 @@ currencies as ( from {{ var('netsuite2_currencies') }} ), +{% if not var('netsuite2__using_to_subsidiary', false) %} +primary_subsidiaries as ( + select + subsidiary_id, + source_relation + from subsidiaries where parent_id is null +), +{% endif %} + period_exchange_rate_map as ( -- exchange rates used, by accounting period, to convert to parent subsidiary select consolidated_exchange_rates.accounting_period_id, @@ -35,6 +44,7 @@ period_exchange_rate_map as ( -- exchange rates used, by accounting period, to c consolidated_exchange_rates.accounting_book_id, {% endif %} + consolidated_exchange_rates.source_relation, consolidated_exchange_rates.average_rate, consolidated_exchange_rates.current_rate, consolidated_exchange_rates.historical_rate, @@ -46,12 +56,17 @@ period_exchange_rate_map as ( -- exchange rates used, by accounting period, to c left join subsidiaries as to_subsidiaries on consolidated_exchange_rates.to_subsidiary_id = to_subsidiaries.subsidiary_id + and consolidated_exchange_rates.source_relation = to_subsidiaries.source_relation left join currencies on currencies.currency_id = to_subsidiaries.currency_id + and currencies.source_relation = to_subsidiaries.source_relation {% if not var('netsuite2__using_to_subsidiary', false) %} - where consolidated_exchange_rates.to_subsidiary_id in (select subsidiary_id from subsidiaries where parent_id is null) -- constraint - only the primary subsidiary has no parent + join primary_subsidiaries + on consolidated_exchange_rates.to_subsidiary_id = primary_subsidiaries.subsidiary_id + and consolidated_exchange_rates.source_relation = primary_subsidiaries.source_relation + {% endif %} ), @@ -73,10 +88,12 @@ accountxperiod_exchange_rate_map as ( -- account table with exchange rate detail when lower(accounts.general_rate_type) = 'current' then period_exchange_rate_map.current_rate when lower(accounts.general_rate_type) = 'average' then period_exchange_rate_map.average_rate else null - end as exchange_rate + end as exchange_rate, + accounts.source_relation from accounts - cross join period_exchange_rate_map + join period_exchange_rate_map + on accounts.source_relation = period_exchange_rate_map.source_relation ) select * diff --git a/models/netsuite2/intermediate/int_netsuite2__tran_and_reporting_periods.sql b/models/netsuite2/intermediate/int_netsuite2__tran_and_reporting_periods.sql index 046a914..042b986 100644 --- a/models/netsuite2/intermediate/int_netsuite2__tran_and_reporting_periods.sql +++ b/models/netsuite2/intermediate/int_netsuite2__tran_and_reporting_periods.sql @@ -10,9 +10,18 @@ subsidiaries as ( from {{ var('netsuite2_subsidiaries') }} ), +primary_subsidiary_calendar as ( + select + fiscal_calendar_id, + source_relation + from subsidiaries + where parent_id is null +), + transaction_and_reporting_periods as ( select base.accounting_period_id as accounting_period_id, + base.source_relation, multiplier.accounting_period_id as reporting_accounting_period_id from accounting_periods as base @@ -21,11 +30,15 @@ transaction_and_reporting_periods as ( and multiplier.is_quarter = base.is_quarter and multiplier.is_year = base.is_year -- this was year_0 in netsuite1 and multiplier.fiscal_calendar_id = base.fiscal_calendar_id + and multiplier.source_relation = base.source_relation and cast(multiplier.starting_at as {{ dbt.type_timestamp() }}) <= {{ current_timestamp() }} + join primary_subsidiary_calendar + on base.fiscal_calendar_id = primary_subsidiary_calendar.fiscal_calendar_id + and base.source_relation = primary_subsidiary_calendar.source_relation + where not base.is_quarter and not base.is_year - and base.fiscal_calendar_id = (select fiscal_calendar_id from subsidiaries where parent_id is null) -- fiscal calendar will align with parent subsidiary's default calendar ) select * diff --git a/models/netsuite2/intermediate/int_netsuite2__tran_lines_w_accounting_period.sql b/models/netsuite2/intermediate/int_netsuite2__tran_lines_w_accounting_period.sql index 0fbc134..ab02a2a 100644 --- a/models/netsuite2/intermediate/int_netsuite2__tran_lines_w_accounting_period.sql +++ b/models/netsuite2/intermediate/int_netsuite2__tran_lines_w_accounting_period.sql @@ -16,6 +16,7 @@ transaction_lines_w_accounting_period as ( -- transaction line totals, by accoun transaction_lines.transaction_line_id, transaction_lines.subsidiary_id, transaction_lines.account_id, + transaction_lines.source_relation, {% if var('netsuite2__multibook_accounting_enabled', false) %} transaction_lines.accounting_book_id, @@ -26,7 +27,9 @@ transaction_lines_w_accounting_period as ( -- transaction line totals, by accoun coalesce(transaction_lines.amount, 0) as unconverted_amount from transaction_lines - join transactions on transactions.transaction_id = transaction_lines.transaction_id + join transactions + on transactions.transaction_id = transaction_lines.transaction_id + and transactions.source_relation = transaction_lines.source_relation where lower(transactions.transaction_type) != 'revenue arrangement' and transaction_lines.is_posting diff --git a/models/netsuite2/intermediate/int_netsuite2__tran_with_converted_amounts.sql b/models/netsuite2/intermediate/int_netsuite2__tran_with_converted_amounts.sql index 28e3590..63b5cce 100644 --- a/models/netsuite2/intermediate/int_netsuite2__tran_with_converted_amounts.sql +++ b/models/netsuite2/intermediate/int_netsuite2__tran_with_converted_amounts.sql @@ -25,7 +25,7 @@ accounts as ( transactions_in_every_calculation_period_w_exchange_rates as ( select transaction_lines_w_accounting_period.*, - reporting_accounting_period_id + transaction_and_reporting_periods.reporting_accounting_period_id {% if var('netsuite2__using_exchange_rate', true) %} , exchange_reporting_period.exchange_rate as exchange_rate_reporting_period @@ -41,22 +41,26 @@ transactions_in_every_calculation_period_w_exchange_rates as ( from transaction_lines_w_accounting_period left join transaction_and_reporting_periods - on transaction_and_reporting_periods.accounting_period_id = transaction_lines_w_accounting_period.transaction_accounting_period_id + on transaction_and_reporting_periods.accounting_period_id = transaction_lines_w_accounting_period.transaction_accounting_period_id + and transaction_and_reporting_periods.source_relation = transaction_lines_w_accounting_period.source_relation {% if var('netsuite2__using_exchange_rate', true) %} left join accountxperiod_exchange_rate_map as exchange_reporting_period on exchange_reporting_period.accounting_period_id = transaction_and_reporting_periods.reporting_accounting_period_id and exchange_reporting_period.account_id = transaction_lines_w_accounting_period.account_id and exchange_reporting_period.from_subsidiary_id = transaction_lines_w_accounting_period.subsidiary_id + and exchange_reporting_period.source_relation = transaction_lines_w_accounting_period.source_relation {% if var('netsuite2__multibook_accounting_enabled', false) %} and exchange_reporting_period.accounting_book_id = transaction_lines_w_accounting_period.accounting_book_id + and exchange_reporting_period.source_relation = transaction_lines_w_accounting_period.source_relation {% endif %} left join accountxperiod_exchange_rate_map as exchange_transaction_period on exchange_transaction_period.accounting_period_id = transaction_and_reporting_periods.accounting_period_id and exchange_transaction_period.account_id = transaction_lines_w_accounting_period.account_id and exchange_transaction_period.from_subsidiary_id = transaction_lines_w_accounting_period.subsidiary_id + and exchange_transaction_period.source_relation = transaction_lines_w_accounting_period.source_relation {% if var('netsuite2__multibook_accounting_enabled', false) %} and exchange_transaction_period.accounting_book_id = transaction_lines_w_accounting_period.accounting_book_id @@ -94,7 +98,8 @@ transactions_with_converted_amounts as ( from transactions_in_every_calculation_period_w_exchange_rates left join accounts - on accounts.account_id = transactions_in_every_calculation_period_w_exchange_rates.account_id + on accounts.account_id = transactions_in_every_calculation_period_w_exchange_rates.account_id + and accounts.source_relation = transactions_in_every_calculation_period_w_exchange_rates.source_relation ) select * diff --git a/models/netsuite2/netsuite2__balance_sheet.sql b/models/netsuite2/netsuite2__balance_sheet.sql index 8245254..a6eed19 100644 --- a/models/netsuite2/netsuite2__balance_sheet.sql +++ b/models/netsuite2/netsuite2__balance_sheet.sql @@ -28,10 +28,19 @@ subsidiaries as ( from {{ var('netsuite2_subsidiaries') }} ), +primary_subsidiary_calendar as ( + select + fiscal_calendar_id, + source_relation + from subsidiaries + where parent_id is null +), + balance_sheet as ( select transactions_with_converted_amounts.transaction_id, transactions_with_converted_amounts.transaction_line_id, + transactions_with_converted_amounts.source_relation, transactions_with_converted_amounts.subsidiary_id, subsidiaries.name as subsidiary_name, @@ -136,6 +145,7 @@ balance_sheet as ( left join transaction_details on transaction_details.transaction_id = transactions_with_converted_amounts.transaction_id and transaction_details.transaction_line_id = transactions_with_converted_amounts.transaction_line_id + and transaction_details.source_relation = transactions_with_converted_amounts.source_relation {% if var('netsuite2__multibook_accounting_enabled', false) %} and transaction_details.accounting_book_id = transactions_with_converted_amounts.accounting_book_id @@ -149,26 +159,35 @@ balance_sheet as ( left join accounts on accounts.account_id = transactions_with_converted_amounts.account_id + and accounts.source_relation = transactions_with_converted_amounts.source_relation left join accounting_periods as reporting_accounting_periods on reporting_accounting_periods.accounting_period_id = transactions_with_converted_amounts.reporting_accounting_period_id + and reporting_accounting_periods.source_relation = transactions_with_converted_amounts.source_relation left join accounting_periods as transaction_accounting_periods on transaction_accounting_periods.accounting_period_id = transactions_with_converted_amounts.transaction_accounting_period_id + and transaction_accounting_periods.source_relation = transactions_with_converted_amounts.source_relation left join subsidiaries on subsidiaries.subsidiary_id = transactions_with_converted_amounts.subsidiary_id + and subsidiaries.source_relation = transactions_with_converted_amounts.source_relation - where reporting_accounting_periods.fiscal_calendar_id = (select fiscal_calendar_id from subsidiaries where parent_id is null) - and transaction_accounting_periods.fiscal_calendar_id = (select fiscal_calendar_id from subsidiaries where parent_id is null) - and (accounts.is_balancesheet - or transactions_with_converted_amounts.is_income_statement) + join primary_subsidiary_calendar + on reporting_accounting_periods.fiscal_calendar_id = primary_subsidiary_calendar.fiscal_calendar_id + and reporting_accounting_periods.source_relation = primary_subsidiary_calendar.source_relation + and transaction_accounting_periods.fiscal_calendar_id = primary_subsidiary_calendar.fiscal_calendar_id + and transaction_accounting_periods.source_relation = primary_subsidiary_calendar.source_relation + + where accounts.is_balancesheet + or transactions_with_converted_amounts.is_income_statement union all select transactions_with_converted_amounts.transaction_id, transactions_with_converted_amounts.transaction_line_id, + transactions_with_converted_amounts.source_relation, transactions_with_converted_amounts.subsidiary_id, subsidiaries.name as subsidiary_name, @@ -221,6 +240,7 @@ balance_sheet as ( left join transaction_details on transaction_details.transaction_id = transactions_with_converted_amounts.transaction_id and transaction_details.transaction_line_id = transactions_with_converted_amounts.transaction_line_id + and transaction_details.source_relation = transactions_with_converted_amounts.source_relation {% if var('netsuite2__multibook_accounting_enabled', false) %} and transaction_details.accounting_book_id = transactions_with_converted_amounts.accounting_book_id @@ -233,20 +253,26 @@ balance_sheet as ( left join accounts on accounts.account_id = transactions_with_converted_amounts.account_id + and accounts.source_relation = transactions_with_converted_amounts.source_relation left join accounting_periods as reporting_accounting_periods on reporting_accounting_periods.accounting_period_id = transactions_with_converted_amounts.reporting_accounting_period_id + and reporting_accounting_periods.source_relation = transactions_with_converted_amounts.source_relation left join subsidiaries on subsidiaries.subsidiary_id = transactions_with_converted_amounts.subsidiary_id + and subsidiaries.source_relation = transactions_with_converted_amounts.source_relation + + join primary_subsidiary_calendar + on reporting_accounting_periods.fiscal_calendar_id = primary_subsidiary_calendar.fiscal_calendar_id + and reporting_accounting_periods.source_relation = primary_subsidiary_calendar.source_relation - where reporting_accounting_periods.fiscal_calendar_id = (select fiscal_calendar_id from subsidiaries where parent_id is null) - and (accounts.is_balancesheet - or transactions_with_converted_amounts.is_income_statement) + where accounts.is_balancesheet + or transactions_with_converted_amounts.is_income_statement ), surrogate_key as ( - {% set surrogate_key_fields = ['transaction_line_id', 'transaction_id', 'accounting_period_id', 'account_name', 'account_id'] %} + {% set surrogate_key_fields = ['transaction_line_id', 'transaction_id', 'accounting_period_id', 'account_name', 'account_id', 'source_relation'] %} {% do surrogate_key_fields.append('to_subsidiary_id') if var('netsuite2__using_to_subsidiary', false) and var('netsuite2__using_exchange_rate', true) %} {% do surrogate_key_fields.append('accounting_book_id') if var('netsuite2__multibook_accounting_enabled', false) %} diff --git a/models/netsuite2/netsuite2__income_statement.sql b/models/netsuite2/netsuite2__income_statement.sql index 66ef8d1..fe5cb52 100644 --- a/models/netsuite2/netsuite2__income_statement.sql +++ b/models/netsuite2/netsuite2__income_statement.sql @@ -48,10 +48,19 @@ departments as ( from {{ var('netsuite2_departments') }} ), +primary_subsidiary_calendar as ( + select + fiscal_calendar_id, + source_relation + from subsidiaries + where parent_id is null +), + income_statement as ( select transactions_with_converted_amounts.transaction_id, transactions_with_converted_amounts.transaction_line_id, + transactions_with_converted_amounts.source_relation, {% if var('netsuite2__multibook_accounting_enabled', false) %} transactions_with_converted_amounts.accounting_book_id, @@ -116,6 +125,7 @@ income_statement as ( join transaction_lines as transaction_lines on transaction_lines.transaction_line_id = transactions_with_converted_amounts.transaction_line_id and transaction_lines.transaction_id = transactions_with_converted_amounts.transaction_id + and transaction_lines.source_relation = transactions_with_converted_amounts.source_relation {% if var('netsuite2__multibook_accounting_enabled', false) %} and transaction_lines.accounting_book_id = transactions_with_converted_amounts.accounting_book_id @@ -123,27 +133,34 @@ income_statement as ( left join departments on departments.department_id = transaction_lines.department_id + and departments.source_relation = transaction_lines.source_relation left join accounts on accounts.account_id = transactions_with_converted_amounts.account_id + and accounts.source_relation = transactions_with_converted_amounts.source_relation left join locations on locations.location_id = transaction_lines.location_id + and locations.source_relation = transaction_lines.source_relation left join classes on classes.class_id = transaction_lines.class_id + and classes.source_relation = transaction_lines.source_relation left join accounting_periods as reporting_accounting_periods on reporting_accounting_periods.accounting_period_id = transactions_with_converted_amounts.reporting_accounting_period_id + and reporting_accounting_periods.source_relation = transactions_with_converted_amounts.source_relation left join subsidiaries on transactions_with_converted_amounts.subsidiary_id = subsidiaries.subsidiary_id + and transactions_with_converted_amounts.source_relation = subsidiaries.source_relation --Below is only used if income statement transaction detail columns are specified dbt_project.yml file. {% if var('income_statement_transaction_detail_columns') != []%} join transaction_details on transaction_details.transaction_id = transactions_with_converted_amounts.transaction_id and transaction_details.transaction_line_id = transactions_with_converted_amounts.transaction_line_id + and transaction_details.source_relation = transactions_with_converted_amounts.source_relation {% if var('netsuite2__multibook_accounting_enabled', false) %} and transaction_details.accounting_book_id = transactions_with_converted_amounts.accounting_book_id {% endif %} @@ -153,13 +170,16 @@ income_statement as ( {% endif %} {% endif %} - where reporting_accounting_periods.fiscal_calendar_id = (select fiscal_calendar_id from subsidiaries where parent_id is null) - and transactions_with_converted_amounts.transaction_accounting_period_id = transactions_with_converted_amounts.reporting_accounting_period_id + join primary_subsidiary_calendar + on reporting_accounting_periods.fiscal_calendar_id = primary_subsidiary_calendar.fiscal_calendar_id + and reporting_accounting_periods.source_relation = primary_subsidiary_calendar.source_relation + + where transactions_with_converted_amounts.transaction_accounting_period_id = transactions_with_converted_amounts.reporting_accounting_period_id and transactions_with_converted_amounts.is_income_statement ), surrogate_key as ( - {% set surrogate_key_fields = ['transaction_line_id', 'transaction_id', 'accounting_period_id', 'account_name'] %} + {% set surrogate_key_fields = ['transaction_line_id', 'transaction_id', 'accounting_period_id', 'account_name', 'source_relation'] %} {% do surrogate_key_fields.append('to_subsidiary_id') if var('netsuite2__using_to_subsidiary', false) and var('netsuite2__using_exchange_rate', true) %} {% do surrogate_key_fields.append('accounting_book_id') if var('netsuite2__multibook_accounting_enabled', false) %} @@ -171,4 +191,4 @@ surrogate_key as ( ) select * -from surrogate_key +from surrogate_key \ No newline at end of file diff --git a/models/netsuite2/netsuite2__transaction_details.sql b/models/netsuite2/netsuite2__transaction_details.sql index 4e836b2..a7174e4 100644 --- a/models/netsuite2/netsuite2__transaction_details.sql +++ b/models/netsuite2/netsuite2__transaction_details.sql @@ -77,9 +77,17 @@ entities as ( from {{ var('netsuite2_entities') }} ), +primary_subsidiary_calendar as ( + select + fiscal_calendar_id, + source_relation + from subsidiaries + where parent_id is null +), + transaction_details as ( select - + transaction_lines.source_relation, {% if var('netsuite2__multibook_accounting_enabled', false) %} transaction_lines.accounting_book_id, transaction_lines.accounting_book_name, @@ -168,11 +176,13 @@ transaction_details as ( join transactions on transactions.transaction_id = transaction_lines.transaction_id + and transactions.source_relation = transaction_lines.source_relation left join transactions_with_converted_amounts as transactions_with_converted_amounts on transactions_with_converted_amounts.transaction_line_id = transaction_lines.transaction_line_id and transactions_with_converted_amounts.transaction_id = transaction_lines.transaction_id and transactions_with_converted_amounts.transaction_accounting_period_id = transactions_with_converted_amounts.reporting_accounting_period_id + and transactions_with_converted_amounts.source_relation = transaction_lines.source_relation {% if var('netsuite2__multibook_accounting_enabled', false) %} and transactions_with_converted_amounts.accounting_book_id = transaction_lines.accounting_book_id @@ -180,48 +190,63 @@ transaction_details as ( left join accounts on accounts.account_id = transaction_lines.account_id + and accounts.source_relation = transaction_lines.source_relation left join accounts as parent_account on parent_account.account_id = accounts.parent_id + and parent_account.source_relation = accounts.source_relation left join accounting_periods on accounting_periods.accounting_period_id = transactions.accounting_period_id + and accounting_periods.source_relation = transactions.source_relation left join customers on customers.customer_id = coalesce(transaction_lines.entity_id, transactions.entity_id) + and customers.source_relation = coalesce(transaction_lines.source_relation, transactions.source_relation) left join classes on classes.class_id = transaction_lines.class_id + and classes.source_relation = transaction_lines.source_relation left join items on items.item_id = transaction_lines.item_id + and items.source_relation = transaction_lines.source_relation left join locations on locations.location_id = transaction_lines.location_id + and locations.source_relation = transaction_lines.source_relation left join vendors on vendors.vendor_id = coalesce(transaction_lines.entity_id, transactions.entity_id) + and vendors.source_relation = coalesce(transaction_lines.source_relation, transactions.source_relation) {% if var('netsuite2__using_vendor_categories', true) %} left join vendor_categories on vendor_categories.vendor_category_id = vendors.vendor_category_id + and vendor_categories.source_relation = vendors.source_relation {% endif %} left join currencies on currencies.currency_id = transactions.currency_id + and currencies.source_relation = transactions.source_relation left join departments on departments.department_id = transaction_lines.department_id + and departments.source_relation = transaction_lines.source_relation join subsidiaries on subsidiaries.subsidiary_id = transaction_lines.subsidiary_id + and subsidiaries.source_relation = transaction_lines.source_relation + + left join primary_subsidiary_calendar + on accounting_periods.fiscal_calendar_id = primary_subsidiary_calendar.fiscal_calendar_id + and accounting_periods.source_relation = primary_subsidiary_calendar.source_relation - where (accounting_periods.fiscal_calendar_id is null - or accounting_periods.fiscal_calendar_id = (select fiscal_calendar_id from subsidiaries where parent_id is null)) + where accounting_periods.fiscal_calendar_id is null or primary_subsidiary_calendar.fiscal_calendar_id is not null ), surrogate_key as ( - {% set surrogate_key_fields = ['transaction_line_id', 'transaction_id'] %} + {% set surrogate_key_fields = ['transaction_line_id', 'transaction_id', 'source_relation'] %} {% do surrogate_key_fields.append('to_subsidiary_id') if var('netsuite2__using_to_subsidiary', false) and var('netsuite2__using_exchange_rate', true) %} {% do surrogate_key_fields.append('accounting_book_id') if var('netsuite2__multibook_accounting_enabled', false) %} diff --git a/packages.yml b/packages.yml index 4a67196..71bad15 100644 --- a/packages.yml +++ b/packages.yml @@ -1,3 +1,6 @@ packages: - - package: fivetran/netsuite_source - version: [">=0.9.0", "<0.10.0"] + # - package: fivetran/netsuite_source + # version: [">=0.9.0", "<0.10.0"] + - git: https://github.com/fivetran/dbt_netsuite_source.git + revision: feature/add-union-data + warn-unpinned: false \ No newline at end of file