Source Stripe: Add undeclared columns to spec

[erohmensing]

What

In this PR, BasicReadTest.test_read test in the Connector Acceptance Tests was updated to fail if the connector produces
stream records which contain columns that haven't been declared in the spec.

Stripe currently fails the updated test. Its acceptance-test-config.yml was edited with the
fail_on_extra_columns: false parameter in order to avoid this change from making the connector fail CAT.

We want to add the undeclared columns to the spec.

How this will help:

• Column selection is currently blocked for connectors that don't declare all columns that they pass. This
  is because turning on column selection will stop these undeclared columns from being sent, when they were
  previously being sent. Doing this allows us to enable column selection for Stripe!
• Users will better understand the shape of the data they'll receive, since the columns will match the spec.

How

The following descriptions of streams that pass undeclared columns come from results of the failed connector acceptance test:

output.txt-2174-2023-03-24T19:46:57.7167414Z The balance_transactions stream has the following schema errors:
output.txt:2175:2023-03-24T19:46:57.7168512Z Additional properties are not allowed ('reporting_category' was unexpected)
--
output.txt-2229-2023-03-24T19:46:57.7213053Z The charges stream has the following schema errors:
output.txt:2230:2023-03-24T19:46:57.7214913Z Additional properties are not allowed ('billing_details', 'application_fee_amount', 'payment_method', 'receipt_url', 'calculated_statement_descriptor', 'statement_descriptor_suffix', 'failure_balance_transaction', 'amount_captured', 'amount_updates', 'disputed', 'transfer_data' were unexpected)
--
output.txt-2964-2023-03-24T19:46:57.7694149Z The customers stream has the following schema errors:
output.txt:2965:2023-03-24T19:46:57.7694756Z Additional properties are not allowed ('test_clock', 'default_currency' were unexpected)
--
output.txt-3564-2023-03-24T19:46:57.7977004Z The disputes stream has the following schema errors:
output.txt:3565:2023-03-24T19:46:57.7977383Z Additional properties are not allowed ('balance_transaction', 'payment_intent' were unexpected)
--
output.txt-3718-2023-03-24T19:46:57.8020962Z The invoice_items stream has the following schema errors:
output.txt:3719:2023-03-24T19:46:57.8021405Z Additional properties are not allowed ('discounts', 'unit_amount_decimal', 'price', 'tax_rates', 'test_clock' were unexpected)
--
output.txt-3847-2023-03-24T19:46:57.8053707Z The invoice_line_items stream has the following schema errors:
output.txt:3848:2023-03-24T19:46:57.8054292Z Additional properties are not allowed ('unit_amount_excluding_tax', 'discount_amounts', 'discounts', 'amount_excluding_tax', 'price', 'proration_details', 'tax_rates', 'tax_amounts' were unexpected)
--
output.txt-4000-2023-03-24T19:46:57.8092017Z The invoices stream has the following schema errors:
output.txt:4001:2023-03-24T19:46:57.8093835Z Additional properties are not allowed ('default_tax_rates', 'application', 'subtotal_excluding_tax', 'latest_revision', 'total_tax_amounts', 'on_behalf_of', 'footer', 'automatic_tax', 'customer_email', 'account_country', 'last_finalization_error', 'shipping_cost', 'total_discount_amounts', 'customer_tax_exempt', 'default_payment_method', 'default_source', 'payment_settings', 'customer_phone', 'payment_intent', 'test_clock', 'application_fee_amount', 'customer_name', 'quote', 'from_invoice', 'customer_shipping', 'custom_fields', 'total_excluding_tax', 'shipping_details', 'customer_address', 'paid_out_of_band', 'customer_tax_ids', 'account_name', 'transfer_data', 'account_tax_ids', 'pre_payment_credit_notes_amount', 'rendering_options', 'collection_method', 'post_payment_credit_notes_amount', 'amount_shipping' were unexpected)
--
output.txt-4347-2023-03-24T19:46:57.8199833Z The payment_intents stream has the following schema errors:
output.txt:4348:2023-03-24T19:46:57.8200696Z Additional properties are not allowed ('latest_charge', 'automatic_payment_methods', 'amount_details', 'processing', 'statement_descriptor' were unexpected)
--
output.txt-5077-2023-03-24T19:46:57.8505147Z The payouts stream has the following schema errors:
output.txt:5078:2023-03-24T19:46:57.8505603Z Additional properties are not allowed ('source_balance', 'original_payout', 'reconciliation_status', 'reversed_by' were unexpected)
--
output.txt-5166-2023-03-24T19:46:57.8527457Z The plans stream has the following schema errors:
output.txt:5167:2023-03-24T19:46:57.8527759Z Additional properties are not allowed ('amount_decimal' was unexpected)
--
output.txt-5226-2023-03-24T19:46:57.8541744Z The products stream has the following schema errors:
output.txt:5227:2023-03-24T19:46:57.8542079Z Additional properties are not allowed ('tax_code', 'default_price' were unexpected)
--
output.txt-5285-2023-03-24T19:46:57.8556023Z The promotion_codes stream has the following schema errors:
output.txt:5286:2023-03-24T19:46:57.8556417Z Additional properties are not allowed ('times_redeemed' was unexpected)
--
output.txt-5380-2023-03-24T19:46:57.8580366Z The subscription_items stream has the following schema errors:
output.txt:5381:2023-03-24T19:46:57.8580745Z Additional properties are not allowed ('tax_rates', 'price', 'billing_thresholds' were unexpected)
--
output.txt-5520-2023-03-24T19:46:57.8621375Z The subscriptions stream has the following schema errors:
output.txt:5521:2023-03-24T19:46:57.8622643Z Additional properties are not allowed ('default_tax_rates', 'application', 'description', 'on_behalf_of', 'pending_setup_intent', 'schedule', 'pending_invoice_item_interval', 'automatic_tax', 'latest_invoice', 'default_source', 'default_payment_method', 'payment_settings', 'currency', 'test_clock', 'trial_settings', 'pause_collection', 'cancellation_details', 'transfer_data', 'billing_thresholds', 'collection_method', 'next_pending_invoice_item_invoice', 'pending_update' were unexpected)
--
output.txt-6271-2023-03-24T19:51:09.2991041Z The balance_transactions stream has the following schema errors:
output.txt:6272:2023-03-24T19:51:09.2992159Z Additional properties are not allowed ('reporting_category' was unexpected)
--
output.txt-6326-2023-03-24T19:51:09.3086076Z The charges stream has the following schema errors:
output.txt:6327:2023-03-24T19:51:09.3087967Z Additional properties are not allowed ('disputed', 'receipt_url', 'calculated_statement_descriptor', 'application_fee_amount', 'failure_balance_transaction', 'statement_descriptor_suffix', 'transfer_data', 'payment_method', 'billing_details', 'amount_captured', 'amount_updates' were unexpected)
--
output.txt-7061-2023-03-24T19:51:09.3569227Z The customers stream has the following schema errors:
output.txt:7062:2023-03-24T19:51:09.3569827Z Additional properties are not allowed ('default_currency', 'test_clock' were unexpected)
--
output.txt-7661-2023-03-24T19:51:09.3839175Z The disputes stream has the following schema errors:
output.txt:7662:2023-03-24T19:51:09.3839556Z Additional properties are not allowed ('balance_transaction', 'payment_intent' were unexpected)
--
output.txt-7815-2023-03-24T19:51:09.3881253Z The invoice_items stream has the following schema errors:
output.txt:7816:2023-03-24T19:51:09.3881686Z Additional properties are not allowed ('tax_rates', 'discounts', 'unit_amount_decimal', 'test_clock', 'price' were unexpected)
--
output.txt-7944-2023-03-24T19:51:09.3925991Z The invoice_line_items stream has the following schema errors:
output.txt:7945:2023-03-24T19:51:09.3926562Z Additional properties are not allowed ('tax_rates', 'tax_amounts', 'unit_amount_excluding_tax', 'discounts', 'discount_amounts', 'proration_details', 'amount_excluding_tax', 'price' were unexpected)
--
output.txt-8097-2023-03-24T19:51:09.3964534Z The invoices stream has the following schema errors:
output.txt:8098:2023-03-24T19:51:09.3966365Z Additional properties are not allowed ('application_fee_amount', 'customer_phone', 'total_excluding_tax', 'rendering_options', 'on_behalf_of', 'customer_email', 'payment_settings', 'amount_shipping', 'account_name', 'total_tax_amounts', 'test_clock', 'total_discount_amounts', 'default_payment_method', 'quote', 'shipping_cost', 'account_country', 'customer_tax_ids', 'customer_address', 'post_payment_credit_notes_amount', 'shipping_details', 'customer_shipping', 'collection_method', 'default_tax_rates', 'payment_intent', 'customer_tax_exempt', 'custom_fields', 'paid_out_of_band', 'automatic_tax', 'transfer_data', 'application', 'pre_payment_credit_notes_amount', 'customer_name', 'latest_revision', 'account_tax_ids', 'default_source', 'footer', 'subtotal_excluding_tax', 'from_invoice', 'last_finalization_error' were unexpected)
--
output.txt-8444-2023-03-24T19:51:09.4053943Z The payment_intents stream has the following schema errors:
output.txt:8445:2023-03-24T19:51:09.4054465Z Additional properties are not allowed ('statement_descriptor', 'processing', 'amount_details', 'latest_charge', 'automatic_payment_methods' were unexpected)
--
output.txt-9174-2023-03-24T19:51:09.4343133Z The payouts stream has the following schema errors:
output.txt:9175:2023-03-24T19:51:09.4343584Z Additional properties are not allowed ('source_balance', 'reversed_by', 'reconciliation_status', 'original_payout' were unexpected)
--
output.txt-9263-2023-03-24T19:51:09.4365623Z The plans stream has the following schema errors:
output.txt:9264:2023-03-24T19:51:09.4365934Z Additional properties are not allowed ('amount_decimal' was unexpected)
--
output.txt-9323-2023-03-24T19:51:09.4379922Z The products stream has the following schema errors:
output.txt:9324:2023-03-24T19:51:09.4380270Z Additional properties are not allowed ('tax_code', 'default_price' were unexpected)
--
output.txt-9382-2023-03-24T19:51:09.4393704Z The promotion_codes stream has the following schema errors:
output.txt:9383:2023-03-24T19:51:09.4394009Z Additional properties are not allowed ('times_redeemed' was unexpected)
--
output.txt-9477-2023-03-24T19:51:09.4417969Z The subscription_items stream has the following schema errors:
output.txt:9478:2023-03-24T19:51:09.4418356Z Additional properties are not allowed ('billing_thresholds', 'price', 'tax_rates' were unexpected)
--
output.txt-9617-2023-03-24T19:51:09.4461667Z The subscriptions stream has the following schema errors:
output.txt:9618:2023-03-24T19:51:09.4462940Z Additional properties are not allowed ('pending_invoice_item_interval', 'billing_thresholds', 'description', 'on_behalf_of', 'cancellation_details', 'payment_settings', 'schedule', 'test_clock', 'default_payment_method', 'default_tax_rates', 'collection_method', 'automatic_tax', 'transfer_data', 'application', 'latest_invoice', 'next_pending_invoice_item_invoice', 'pending_update', 'pause_collection', 'trial_settings', 'currency', 'default_source', 'pending_setup_intent' were unexpected)
--
output.txt-10367-2023-03-24T19:55:13.3129153Z The balance_transactions stream has the following schema errors:
output.txt:10368:2023-03-24T19:55:13.3130435Z Additional properties are not allowed ('reporting_category' was unexpected)
--
output.txt-10422-2023-03-24T19:55:13.3173167Z The charges stream has the following schema errors:
output.txt:10423:2023-03-24T19:55:13.3175289Z Additional properties are not allowed ('disputed', 'failure_balance_transaction', 'transfer_data', 'amount_captured', 'amount_updates', 'billing_details', 'payment_method', 'statement_descriptor_suffix', 'application_fee_amount', 'calculated_statement_descriptor', 'receipt_url' were unexpected)
--
output.txt-11157-2023-03-24T19:55:13.4014020Z The customers stream has the following schema errors:
output.txt:11158:2023-03-24T19:55:13.4015015Z Additional properties are not allowed ('default_currency', 'test_clock' were unexpected)
--
output.txt-11757-2023-03-24T19:55:13.4467795Z The disputes stream has the following schema errors:
output.txt:11758:2023-03-24T19:55:13.4468481Z Additional properties are not allowed ('balance_transaction', 'payment_intent' were unexpected)
--
output.txt-11911-2023-03-24T19:55:13.4539752Z The invoice_items stream has the following schema errors:
output.txt:11912:2023-03-24T19:55:13.4540553Z Additional properties are not allowed ('tax_rates', 'unit_amount_decimal', 'test_clock', 'discounts', 'price' were unexpected)
--
output.txt-12040-2023-03-24T19:55:13.4599442Z The invoice_line_items stream has the following schema errors:
output.txt:12041:2023-03-24T19:55:13.4600433Z Additional properties are not allowed ('tax_amounts', 'tax_rates', 'proration_details', 'discounts', 'amount_excluding_tax', 'price', 'unit_amount_excluding_tax', 'discount_amounts' were unexpected)
--
output.txt-12193-2023-03-24T19:55:13.4667542Z The invoices stream has the following schema errors:
output.txt:12194:2023-03-24T19:55:13.4670897Z Additional properties are not allowed ('account_name', 'automatic_tax', 'customer_phone', 'account_tax_ids', 'latest_revision', 'application_fee_amount', 'default_tax_rates', 'post_payment_credit_notes_amount', 'customer_tax_ids', 'footer', 'rendering_options', 'last_finalization_error', 'test_clock', 'default_payment_method', 'quote', 'account_country', 'shipping_cost', 'shipping_details', 'total_excluding_tax', 'customer_email', 'customer_tax_exempt', 'transfer_data', 'amount_shipping', 'customer_address', 'total_discount_amounts', 'paid_out_of_band', 'custom_fields', 'default_source', 'customer_shipping', 'payment_intent', 'on_behalf_of', 'pre_payment_credit_notes_amount', 'subtotal_excluding_tax', 'total_tax_amounts', 'from_invoice', 'customer_name', 'payment_settings', 'collection_method', 'application' were unexpected)
--
output.txt-12540-2023-03-24T19:55:13.4825221Z The payment_intents stream has the following schema errors:
output.txt:12541:2023-03-24T19:55:13.4826227Z Additional properties are not allowed ('latest_charge', 'automatic_payment_methods', 'processing', 'amount_details', 'statement_descriptor' were unexpected)
--
output.txt-13270-2023-03-24T19:55:13.5313879Z The payouts stream has the following schema errors:
output.txt:13271:2023-03-24T19:55:13.5314649Z Additional properties are not allowed ('original_payout', 'reconciliation_status', 'source_balance', 'reversed_by' were unexpected)
--
output.txt-13359-2023-03-24T19:55:13.5347600Z The plans stream has the following schema errors:
output.txt:13360:2023-03-24T19:55:13.5347954Z Additional properties are not allowed ('amount_decimal' was unexpected)
--
output.txt-13419-2023-03-24T19:55:13.5361658Z The products stream has the following schema errors:
output.txt:13420:2023-03-24T19:55:13.5361999Z Additional properties are not allowed ('tax_code', 'default_price' were unexpected)
--
output.txt-13478-2023-03-24T19:55:13.5375498Z The promotion_codes stream has the following schema errors:
output.txt:13479:2023-03-24T19:55:13.5375816Z Additional properties are not allowed ('times_redeemed' was unexpected)
--
output.txt-13573-2023-03-24T19:55:13.5400240Z The subscription_items stream has the following schema errors:
output.txt:13574:2023-03-24T19:55:13.5400622Z Additional properties are not allowed ('tax_rates', 'billing_thresholds', 'price' were unexpected)
--
output.txt-13713-2023-03-24T19:55:13.5436300Z The subscriptions stream has the following schema errors:
output.txt:13714:2023-03-24T19:55:13.5437472Z Additional properties are not allowed ('automatic_tax', 'billing_thresholds', 'next_pending_invoice_item_invoice', 'currency', 'latest_invoice', 'pending_invoice_item_interval', 'default_tax_rates', 'pending_setup_intent', 'pause_collection', 'cancellation_details', 'test_clock', 'default_payment_method', 'transfer_data', 'description', 'pending_update', 'default_source', 'on_behalf_of', 'payment_settings', 'trial_settings', 'collection_method', 'schedule', 'application' were unexpected)

1. Add the missing properties indicated by the Additional properties are not allowed ('<column>', 'column' were unexpected) logs to the connector's spec.
2. Please also consult our DataDog Dashboard for Record Unexpected Fields for this connector. The API might be returning even more record properties that need to be added to the catalog.
3. Remove fail_on_extra_columns: false from the acceptance-test-config.yml file.
4. Commit changes to spec and acceptance-test-config.yml and open a PR.
5. Run tests on the connector, either automatically via CI or manually via the /test command
6. Profit!

Definition of done: Sentry passes CAT without declaring fail_on_extra_columns: false. If the
API starts sending over extra columns in the future, we will catch and fix them as part of the #connector-health
movement.

[mrhallak]

@erohmensing this would be a huge boost for Stripe! I assume this would also need to be done for the credit_note missing stream #24666

[erohmensing]

Thanks for that contribution to stripe @mrhallak! I just flagged it so hopefully you will get a response soon. Yes, it would have to be done for the credit_note stream, only if you did not declare all the columns that are passed through, though. If you're able, would be great if you could make sure all the columns for that stream are declared in that PR so there's less to fix up later :)