Continuation of #88 - handle multiple disputes in stripe__balance_tra…

…nsactions (#92)

* fix!: handle multiple disputes (#88)

* Documentation Standard Updates (#85)

* MagicBot/documentation-updates

* Apply suggestions from code review

* Apply suggestions from code review

---------

Co-authored-by: Jamie Rodriguez <65564846+fivetran-jamie@users.noreply.github.com>

* fix!: handle multiple disputes

* replace array_agg with string_agg

* update changelog

* bump version

---------

Co-authored-by: Joe Markiewicz <74217849+fivetran-joemarkiewicz@users.noreply.github.com>
Co-authored-by: Jamie Rodriguez <65564846+fivetran-jamie@users.noreply.github.com>

* changes

* aws and changelog

* validations

* Update models/stripe__balance_transactions.sql

Co-authored-by: Renee Li <91097070+fivetran-reneeli@users.noreply.github.com>

* renee feedback

* docs

* update package ref

* changelog

---------

Co-authored-by: bramrodenburg <14278376+bramrodenburg@users.noreply.github.com>
Co-authored-by: Joe Markiewicz <74217849+fivetran-joemarkiewicz@users.noreply.github.com>
Co-authored-by: Renee Li <91097070+fivetran-reneeli@users.noreply.github.com>

CHANGELOG.md

@@ -1,3 +1,31 @@
+# dbt_stripe v0.15.0
+
+## Breaking Changes
+- Updated `stripe__balance_transactions` to correctly handle multiple disputes on the same transaction:
+  - Adjusted [`customer_facing_amount`](https://github.com/fivetran/dbt_stripe/blob/main/models/stripe__balance_transactions.sql#L139-L144) to reflect the `dispute_amount` of the *latest* dispute (if the transaction is not a charge or refund and is associated with any disputes) ([PR #92](https://github.com/fivetran/dbt_stripe/pull/92)).
+  - Added the following the dispute-related columns ([PR #92](https://github.com/fivetran/dbt_stripe/pull/92)):
+    - `latest_dispute_amount_won`: Latest disputed amount that was won in favor of the merchant.
+    - `latest_dispute_amount_lost`: Latest disputed amount that was lost and returned to the customer.
+    - `latest_dispute_amount_under_review`: Latest disputed amount that is currently under review by the bank.
+    - `latest_dispute_amount_needs_response`: Latest disputed amount that currently needs a response (the dispute has been filed but the merchant has not yet responded with evidence).
+    - `latest_dispute_amount_warning_closed`: Latest disputed amount that is currently of status `warning_under_closed` (early fraud warning being closed due to no formal dispute).
+    - `latest_dispute_amount_warning_under_review`: Latest disputed amount that is currently of status `warning_under_review` (card issuer suspects possible fraud but hasn't yet escalated the situation to a full dispute).
+    - `latest_dispute_amount_warning_needs_response`: Latest disputed amount that is currently of status `warning_needs_response` (early fraud warning has been escalated into formal dispute and/or card issuer has requested more information).
+    - `dispute_count`: Count of disputes raised against this transaction. If > 1, join in `dispute` data for additional information on each dispute.
+  - Adjusted the `dispute_id` and `dispute_reason` fields to aggregate together data from multiple disputes if present. They have been renamed to `dispute_ids` and `dispute_reasons` in the following models ([PR #88](https://github.com/fivetran/dbt_stripe/pull/88)):
+    - `stripe__balance_transactions`
+    - `stripe__activity_itemized_2`
+    - `stripe__balance_change_from_activity_itemized_3`
+    - `stripe__ending_balance_reconciliation_itemized_4`
+
+## Under the Hood
+- Updated `dispute` seed data to test the above changes ([PR #92](https://github.com/fivetran/dbt_stripe/pull/92)).
+- Added additional validation tests on the affected models from above ([PR #92](https://github.com/fivetran/dbt_stripe/pull/92)).
+
+## Contributors
+- [@bramrodenburg](https://github.com/bramrodenburg) ([PR #88](https://github.com/fivetran/dbt_stripe/pull/88))
+- [@jsnorthrup](https://github.com/jsnorthrup)
+
 # dbt_stripe v0.14.1
 
 [PR #89](https://github.com/fivetran/dbt_stripe/pull/89) includes the following updates:

README.md

@@ -72,7 +72,7 @@ Include the following stripe package version in your `packages.yml` file:
 ```yaml
 packages:
   - package: fivetran/stripe
-    version: [">=0.14.0", "<0.15.0"]
+    version: [">=0.15.0", "<0.16.0"]
 ```
 Do **NOT** include the `stripe_source` package in this file. The transformation package itself has a dependency on it and will install the source package as well.
 

dbt_project.yml

@@ -1,7 +1,7 @@
 config-version: 2
 name: 'stripe'
 
-version: '0.14.1'
+version: '0.15.0'
 require-dbt-version: [">=1.3.0", "<2.0.0"]
 models:
   stripe:

integration_tests/dbt_project.yml

@@ -1,7 +1,7 @@
 config-version: 2
 
 name: 'stripe_integration_tests'
-version: '0.14.1'
+version: '0.15.0'
 
 profile: 'integration_tests'
 

integration_tests/seeds/dispute_data.csv

@@ -8,4 +8,6 @@ du_0LyvFuig6peNXzgDyBMCjDi6,2022-11-27 14:20:24,221244,txn_0LyvHhig6peNXzgDg5Lan
 du_0LiAl6ig6peNXzgDEejs24CE,2022-11-13 20:20:09,432532,txn_0Lumheig6peNXzgDSp1KIXph,ch_2LVoJuig6peNXzgD1DvyohjD,,2022-09-15 05:37:08,usd,,,,,,,,,,,2022-11-12 23:59:59.000000 UTC,false,false,0,,,,,,,,,,,,,,,,,,false,true,,general,lost
 du_0M9ScVig6peNXzgDkfd7tUNb,2022-12-12 20:19:03,1400,txn_0M9SdPig6peNXzgDgpb4aJDQ,ch_2LqcRzig6peNXzgD06FImTqU,,2022-11-29 12:09:03,usd,,,,,,,,,,,2022-12-25 23:59:59.000000 UTC,false,false,0,,,,,,,,,,,,,,,,,,false,true,,subscription_canceled,lost
 du_0LuGd7ig6peNXzgDUxXx85wF,2022-11-14 14:21:25,1400,txn_0LuGe5ig6peNXzgD7PMfTzJ6,ch_2LfkHjig6peNXzgD1e9ltqQt,,2022-10-18 14:18:53,usd,,,,,,,,,,,2022-11-13 23:59:59.000000 UTC,false,false,0,,,,,,,,,,,,,,,,,,false,true,,subscription_canceled,lost
-du_0LuGbbig6peNXzgDjHA4RfoT,2022-11-14 14:21:25,700,txn_0LuGcrig6peNXzgDKP5y95T9,ch_2LLwgKig6peNXzgD0Sr5X6kL,,2022-10-18 14:17:19,usd,,,,,,,,,,,2022-11-13 23:59:59.000000 UTC,false,false,0,,,,,,,,,,,,,,,,,,false,true,,subscription_canceled,lost
+du_0LuGbbig6peNXzgDjHA4RfoT,2022-11-14 14:21:25,700,txn_0LuGcrig6peNXzgDKP5y95T9,ch_2LLwgKig6peNXzgD0Sr5X6kL,,2022-10-18 14:17:19,usd,,,,,,,,,,,2022-11-13 23:59:59.000000 UTC,false,false,0,,,,,,,,,,,,,,,,,,false,true,,subscription_canceled,warning_needs_response
+du_5345g6peNXzgDjHA4RfoT,2022-11-14 14:21:25,701,txn_0LuGcrig6peNXzgDKP5y95T9,ch_2LLwgKig6peNXzgD0Sr5X6kL,,2023-10-18 14:17:19,usd,,,,,,,,,,,2023-11-13 23:59:59.000000 UTC,false,false,0,,,,,,,,,,,,,,,,,,false,true,,subscription_canceled,under_review
+du_53534DjHA4RfoT,2022-11-14 14:21:25,702,txn_0LuGcrig6peNXzgDKP5y95T9,ch_2LLwgKig6peNXzgD0Sr5X6kL,,2023-10-20 14:17:19,usd,,,,,,,,,,,2023-11-13 23:59:59.000000 UTC,false,false,0,,,,,,,,,,,,,,,,,,false,true,,subscription_canceled,lost

integration_tests/tests/consistency/consistency_activity_itemized_2.sql

@@ -0,0 +1,21 @@
+{{ config(
+    tags="fivetran_validations",
+    enabled=var('fivetran_validation_tests_enabled', false)
+) }}
+
+with prod as (
+    select balance_transaction_id, dispute_id as dispute_ids -- we don't have multi-dispute records in our data
+    from {{ target.schema }}_stripe_prod.stripe__activity_itemized_2
+),
+
+dev as (
+    select balance_transaction_id, dispute_ids
+    from {{ target.schema }}_stripe_dev.stripe__activity_itemized_2
+)
+
+-- test will return values and fail if the values are different (which they shouldn't be in our test data)
+select *
+from prod
+join dev
+    on prod.balance_transaction_id = dev.balance_transaction_id
+where prod.dispute_ids != dev.dispute_ids

integration_tests/tests/consistency/consistency_balance_change_from_activity_itemized_3.sql

@@ -0,0 +1,21 @@
+{{ config(
+    tags="fivetran_validations",
+    enabled=var('fivetran_validation_tests_enabled', false)
+) }}
+
+with prod as (
+    select balance_transaction_id, dispute_reason as dispute_reasons -- we don't have multi-dispute records in our data
+    from {{ target.schema }}_stripe_prod.stripe__balance_change_from_activity_itemized_3
+),
+
+dev as (
+    select balance_transaction_id, dispute_reasons
+    from {{ target.schema }}_stripe_dev.stripe__balance_change_from_activity_itemized_3
+)
+
+-- test will return values and fail if the values are different (which they shouldn't be in our test data)
+select *
+from prod
+join dev
+    on prod.balance_transaction_id = dev.balance_transaction_id
+where prod.dispute_reasons != dev.dispute_reasons

integration_tests/tests/consistency/consistency_balance_transcations_count.sql

@@ -0,0 +1,32 @@
+{{ config(
+    tags="fivetran_validations",
+    enabled=var('fivetran_validation_tests_enabled', false)
+) }}
+
+-- this test is to make sure the rows counts are the same between versions
+with prod as (
+    select 
+        count(*) as prod_rows,
+        sum(customer_facing_amount) as customer_facing_amount,
+        sum(balance_transaction_amount) as balance_transaction_amount,
+        sum(balance_transaction_net) as balance_transaction_net
+    from {{ target.schema }}_stripe_prod.stripe__balance_transactions
+),
+
+dev as (
+    select 
+        count(*) as dev_rows,
+        sum(customer_facing_amount) as customer_facing_amount,
+        sum(balance_transaction_amount) as balance_transaction_amount,
+        sum(balance_transaction_net) as balance_transaction_net
+    from {{ target.schema }}_stripe_dev.stripe__balance_transactions
+)
+
+-- test will return values and fail if the row counts don't match
+select *
+from prod
+join dev
+    on prod.prod_rows != dev.dev_rows
+    or prod.customer_facing_amount != dev.customer_facing_amount
+    or prod.balance_transaction_amount != dev.balance_transaction_amount
+    or prod.balance_transaction_net != dev.balance_transaction_net

integration_tests/tests/consistency/consistency_ending_balance_reconciliation_itemized_4_count.sql

@@ -0,0 +1,21 @@
+{{ config(
+    tags="fivetran_validations",
+    enabled=var('fivetran_validation_tests_enabled', false)
+) }}
+
+with prod as (
+    select balance_transaction_id, dispute_reason as dispute_reasons -- we don't have multi-dispute records in our data
+    from {{ target.schema }}_stripe_prod.stripe__ending_balance_reconciliation_itemized_4
+),
+
+dev as (
+    select balance_transaction_id, dispute_reasons
+    from {{ target.schema }}_stripe_dev.stripe__ending_balance_reconciliation_itemized_4
+)
+
+-- test will return values and fail if the values are different (which they shouldn't be in our test data)
+select *
+from prod
+join dev
+    on prod.balance_transaction_id = dev.balance_transaction_id
+where prod.dispute_reasons != dev.dispute_reasons

integration_tests/tests/integrity/integrity_balance_transactions.sql

@@ -0,0 +1,27 @@
+{{ config(
+    tags="fivetran_validations",
+    enabled=var('fivetran_validation_tests_enabled', false)
+) }}
+
+with staging_model as (
+    select 
+        count(*) as row_count,
+        sum(amount) as balance_transaction_amount,
+        sum(net) as balance_transaction_net
+    from {{ ref('stg_stripe__balance_transaction') }}
+),
+
+end_model as (
+    select 
+        count(*) as row_count,
+        sum(balance_transaction_amount) as balance_transaction_amount,
+        sum(balance_transaction_net) as balance_transaction_net
+    from {{ ref('stripe__balance_transactions') }}
+)
+
+select *
+from staging_model
+join end_model
+    on staging_model.row_count != end_model.row_count
+    or staging_model.balance_transaction_amount != end_model.balance_transaction_amount
+    or staging_model.balance_transaction_net != end_model.balance_transaction_net

models/stripe.yml

@@ -162,10 +162,10 @@ models:
       description: The ID of the subscription is associated with this record.
     - name: charge_statement_descriptor
       description: Extra information about a source. This will appear on your customer’s statement every time you charge the source.
-    - name: dispute_id
-      description: Unique identifier for the object.
-    - name: dispute_reason
-      description: Reason given by cardholder for dispute. Possible values are bank_cannot_process, check_returned, credit_not_processed, customer_initiated, debit_not_authorized, duplicate, fraudulent, general, incorrect_account_details, insufficient_funds, product_not_received, product_unacceptable, subscription_canceled, or unrecognized. 
+    - name: dispute_ids
+      description: Unique identifiers for the object.
+    - name: dispute_reasons
+      description: Reasons given by cardholder for dispute. Possible values are bank_cannot_process, check_returned, credit_not_processed, customer_initiated, debit_not_authorized, duplicate, fraudulent, general, incorrect_account_details, insufficient_funds, product_not_received, product_unacceptable, subscription_canceled, or unrecognized. 
     - name: refund_id
       description: ID of the refund associated with this record.
     - name: transfer_id
@@ -178,6 +178,22 @@ models:
       description: (Beta) For Stripe Connect activity related to a connected account, charge id of the direct charge that happened on connected account.
     - name: source_relation
       description: "{{ doc('source_relation') }}"
+    - name: latest_dispute_amount_won
+      description: Latest disputed amount that was won in favor of the merchant. Usually the amount of the charge, but can differ (usually because of currency fluctuation or because only part of the order is disputed).
+    - name: latest_dispute_amount_lost
+      description: Latest disputed amount that was lost and returned to the customer. Usually the amount of the charge, but can differ (usually because of currency fluctuation or because only part of the order is disputed).
+    - name: latest_dispute_amount_under_review
+      description: Latest disputed amount that is currently under review by the bank. Usually the amount of the charge, but can differ (usually because of currency fluctuation or because only part of the order is disputed).
+    - name: latest_dispute_amount_needs_response
+      description: Latest disputed amount that currently needs a response (the dispute has been filed but the merchant has not yet responded with evidence). Usually the amount of the charge, but can differ (usually because of currency fluctuation or because only part of the order is disputed).
+    - name: latest_dispute_amount_warning_closed
+      description: Latest disputed amount that is currently of status `warning_under_closed` (early fraud warning being closed due to no formal dispute). Usually the amount of the charge, but can differ (usually because of currency fluctuation or because only part of the order is disputed).
+    - name: latest_dispute_amount_warning_under_review
+      description: Latest disputed amount that is currently of status `warning_under_review` (card issuer suspects possible fraud but hasn't yet escalated the situation to a full dispute). Usually the amount of the charge, but can differ (usually because of currency fluctuation or because only part of the order is disputed).
+    - name: latest_dispute_amount_warning_needs_response
+      description: Latest disputed amount that is currently of status `warning_needs_response` (early fraud warning has been escalated into formal dispute or card issuer has requested more information). Usually the amount of the charge, but can differ (usually because of currency fluctuation or because only part of the order is disputed).
+    - name: dispute_count
+      description: Count of disputes raised against this transaction. If > 1, join in `dispute` data for additional information on each dispute.
 
   - name: stripe__customer_overview
     description: Each record represents a customer, enriched with metrics about their associated transactions.
@@ -585,8 +601,8 @@ models:
         description: The unique ID of the related Payment Intent, if any.
       - name: refund_id	
         description: The unique ID of the related refund, if any.
-      - name: dispute_id	
-        description: The unique ID of the related dispute, if any.
+      - name: dispute_ids	
+        description: The unique IDs of the related disputes, if any.
       - name: invoice_id		
         description: Unique ID for the invoice associated with this balance transaction. Available for charges, refunds, and disputes made in connection with a Stripe Billing invoice.
       - name: invoice_number		
@@ -666,7 +682,7 @@ models:
       - name: statement_descriptor		
         description: The dynamic statement descriptor or suffix specified when the related charge was created.
       - name: customer_facing_amount		
-        description: For transactions associated with charges, refunds, or disputes, the amount of the original charge, refund, or dispute. If the customer was charged in a different currency than your account’s default, this field will reflect the amount as seen by the customer.
+        description: For transactions associated with charges, refunds, or disputes, the amount of the original charge, the refund, or the latest dispute. If the customer was charged in a different currency than your account’s default, this field will reflect the amount as seen by the customer.
       - name: balance_transaction_description	
         description: An arbitrary string attached to the balance transaction.
       - name: connected_account_id		
@@ -707,7 +723,7 @@ models:
       - name: description	
         description: An arbitrary string attached to the balance transaction. Often useful for displaying to users.
       - name: customer_facing_amount		
-        description: For transactions associated with charges, refunds, or disputes, the amount of the original charge, refund, or dispute. If the customer was charged in a different currency than your account’s default, this field will reflect the amount as seen by the customer.
+        description: For transactions associated with charges, refunds, or disputes, the amount of the original charge, the refund, or the latest dispute. If the customer was charged in a different currency than your account’s default, this field will reflect the amount as seen by the customer.
       - name: customer_facing_currency		
         description: For transactions associated with charges, refunds, or disputes, the three-letter ISO currency code for customer_facing_amount.
       - name: automatic_payout_id		
@@ -792,8 +808,8 @@ models:
         description: Two-letter ISO code representing the country of the card.
       - name: statement_descriptor		
         description: The dynamic statement descriptor or suffix specified when the related charge was created.
-      - name: dispute_reason		
-        description: Reason given by cardholder for dispute. Read more about dispute reasons.
+      - name: dispute_reasons		
+        description: Reasons given by cardholder for disputes. Read more about [dispute reasons](https://docs.stripe.com/disputes/categories).
       - name: connected_account_id		
         description: For Stripe Connect activity related to a connected account, the unique ID for the account.
       - name: connected_account_country		
@@ -836,7 +852,7 @@ models:
       - name: description
         description: An arbitrary string attached to the balance transaction. Often useful for displaying to users.
       - name: customer_facing_amount
-        description: For transactions associated with charges, refunds, or disputes, the amount of the original charge, refund, or dispute. If the customer was charged in a different currency than your account’s default, this field will reflect the amount as seen by the customer.
+        description: For transactions associated with charges, refunds, or disputes, the amount of the original charge, the refund, or the latest dispute. If the customer was charged in a different currency than your account’s default, this field will reflect the amount as seen by the customer.
       - name: customer_facing_currency
         description: For transactions associated with charges, refunds, or disputes, the three-letter ISO currency code for customer_facing_amount.
       - name: customer_id
@@ -917,8 +933,8 @@ models:
         description: Two-letter ISO code representing the country of the card.
       - name: statement_descriptor
         description: The dynamic statement descriptor or suffix specified when the related charge was created.
-      - name: dispute_reason
-        description: Reason given by cardholder for dispute. Read more about dispute reasons.
+      - name: dispute_reasons
+        description: Reasons given by cardholder for disputes. Read more about [dispute reasons](https://docs.stripe.com/disputes/categories).
       - name: connected_account_id
         description: For Stripe Connect activity related to a connected account, the unique ID for the account.
       - name: connected_account_country