From a55df39e4853cefe0d52ed882f6c3e7e240d3d2c Mon Sep 17 00:00:00 2001 From: Christo Grabowski Date: Mon, 7 Aug 2023 00:54:10 -0400 Subject: [PATCH 1/6] add API key setup instructions --- docs/integrations/sources/stripe.md | 47 ++++++++++++++++++----------- 1 file changed, 30 insertions(+), 17 deletions(-) diff --git a/docs/integrations/sources/stripe.md b/docs/integrations/sources/stripe.md index f8e632970d28..eb9291166903 100644 --- a/docs/integrations/sources/stripe.md +++ b/docs/integrations/sources/stripe.md @@ -1,33 +1,46 @@ # Stripe +This page contains the setup guide and reference information for the Stripe source connector. + :::warning -Stripe API Restriction: Access to the events endpoint is [guaranteed only for the last 30 days](https://stripe.com/docs/api/events). Using the full-refresh-overwrite sync from Airbyte will delete the events data older than 30 days from your target destination. +**Stripe API Restriction on Events Data**: Access to the events endpoint is [guaranteed only for the last 30 days](https://stripe.com/docs/api/events). If you use the Full Refresh sync, any events data older than 30 days will be **deleted** from your target destination. ::: -This page guides you through the process of setting up the Stripe source connector. - ## Prerequisites - Your [Stripe `Account ID`](https://dashboard.stripe.com/settings/account) - Your [Stripe `Secret Key`](https://dashboard.stripe.com/apikeys) -## Set up the Stripe source connector +## Setup Guide + +To authenticate the Stripe connector, you must provide a Stripe Secret Key. Although you may use an existing key, we recommend that you create a new key specifically for Airbyte and grant it **Read** priviliges. We recommend granting **Read** priviliges to all permissions, and configuring the data you would like to replicate in the connector itself. You can also use the API keys for the [test mode](https://stripe.com/docs/keys#obtain-api-keys) to try out the Stripe integration with Airbyte. + +### Create a Stripe Secret Key + +1. Log in to your [Stripe account](https://dashboard.stripe.com/login). +2. In the top navigation bar, click **Developers**. +3. In the top-left corner, click **API keys**. +4. Click **+ Create restricted key**. +5. Choose a **Key name**, and select **Read** for all permissions. +6. Click **Create key**. + +### Set up the Stripe source connector in Airbyte -1. Log into your [Airbyte Cloud](https://cloud.airbyte.com/workspaces) or Airbyte Open Source account. -2. Click **Sources** and then click **+ New source**. -3. On the Set up the source page, select **Stripe** from the Source type dropdown. -4. Enter a name for your source. -5. For **Account ID**, enter your [Stripe `Account ID`](https://dashboard.stripe.com/settings/account). -6. For **Secret Key**, enter your [Stripe `Secret Key`](https://dashboard.stripe.com/apikeys) +1. Log in to your [Airbyte Cloud](https://cloud.airbyte.com/workspaces) or Airbyte Open Source account. +2. In the left navigation bar, click **Sources**. In the top-right corner, click **+ New source**. +3. Find and select **Stripe** from the list of available sources. +4. For **Source name**, enter a name to help you identify this source. +5. For **Account ID**, enter your Stripe Account ID. This ID begins with `acct_`, and can be found in the top-right corner of your Stripe [account settings page](https://dashboard.stripe.com/settings/account). +6. For **Secret Key**, enter your Stripe API Key. +7. For **Replication Start Date**, use the provided datepicker or enter the UTC date and time programmatically in the format `YYYY-MM-DDTHH:mm:ssZ`. The data added on and after this date will be replicated. +8. (Optional) For **Lookback Window in days**, select the number of days prior to the start date for which you want to sync your data. This value allows the connector to retrieve data that might have been updated after its initial creation. For example, if the start date is set to `2021-01-01T00:00:00Z`, then: - We recommend creating a secret key specifically for Airbyte to control which resources Airbyte can access. For ease of use, we recommend granting read permission to all resources and configuring which resource to replicate in the Airbyte UI. You can also use the API keys for the [test mode](https://stripe.com/docs/keys#obtain-api-keys) to try out the Stripe integration with Airbyte. +- Leaving the Lookback Window in days parameter at its default value of 0 means Airbyte will sync data starting from 2021-01-01T00:00:00Z. +- Setting the Lookback Window in days value to 1 means Airbyte will consider the Replication start date to be one day earlier, or 2020-12-31T00:00:00Z. +- Setting the Lookback Window in days value to 7 means Airbyte will sync data starting from seven days earlier, or 2020-12-25T00:00:00Z. -7. For **Replication start date**, enter the date in `YYYY-MM-DDTHH:mm:ssZ` format. The data added on and after this date will be replicated. -8. For **Lookback Window in days (Optional)**, select the number of days the value in days prior to the start date that you to sync your data with. If your data is updated after setting up this connector, you can use the this option to reload data from the past N days. Example: If the Replication start date is set to `2021-01-01T00:00:00Z`, then: - - If you leave the Lookback Window in days parameter to its the default value of 0, Airbyte will sync data from the Replication start date `2021-01-01T00:00:00Z` - - If the Lookback Window in days value is set to 1, Airbyte will consider the Replication start date to be `2020-12-31T00:00:00Z` - - If the Lookback Window in days value is set to 7, Airbyte will sync data from `2020-12-25T00:00:00Z` -9. Click **Set up source**. +9. (Optional) For **Data Sync Frequency**, select the time increment in days for the connector to use when requesting data from the Stripe API. Setting this value higher will reduce the number of requests, speeding up the sync process. However, this will result in data being updated less frequently. The default value is 365 days. +10. Click **Set up source** and wait for the tests to complete. ## Supported sync modes From 6af724bf500865a690aef26c8aae974020d28fef Mon Sep 17 00:00:00 2001 From: Christo Grabowski Date: Mon, 7 Aug 2023 03:08:58 -0400 Subject: [PATCH 2/6] re-order bullet points --- docs/integrations/sources/stripe.md | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/docs/integrations/sources/stripe.md b/docs/integrations/sources/stripe.md index eb9291166903..42a95fdf667e 100644 --- a/docs/integrations/sources/stripe.md +++ b/docs/integrations/sources/stripe.md @@ -13,7 +13,7 @@ This page contains the setup guide and reference information for the Stripe sour ## Setup Guide -To authenticate the Stripe connector, you must provide a Stripe Secret Key. Although you may use an existing key, we recommend that you create a new key specifically for Airbyte and grant it **Read** priviliges. We recommend granting **Read** priviliges to all permissions, and configuring the data you would like to replicate in the connector itself. You can also use the API keys for the [test mode](https://stripe.com/docs/keys#obtain-api-keys) to try out the Stripe integration with Airbyte. +To authenticate the Stripe connector, you need a Stripe Secret Key. Although you may use an existing key, we recommend that you create a new key specifically for Airbyte and grant it **Read** priviliges. We recommend granting **Read** priviliges to all permissions, and configuring the data you would like to replicate in the connector itself. ### Create a Stripe Secret Key @@ -24,6 +24,8 @@ To authenticate the Stripe connector, you must provide a Stripe Secret Key. Alth 5. Choose a **Key name**, and select **Read** for all permissions. 6. Click **Create key**. +For more information on Stripe API Keys, see the [Stripe documentation](https://stripe.com/docs/keys). + ### Set up the Stripe source connector in Airbyte 1. Log in to your [Airbyte Cloud](https://cloud.airbyte.com/workspaces) or Airbyte Open Source account. @@ -32,14 +34,14 @@ To authenticate the Stripe connector, you must provide a Stripe Secret Key. Alth 4. For **Source name**, enter a name to help you identify this source. 5. For **Account ID**, enter your Stripe Account ID. This ID begins with `acct_`, and can be found in the top-right corner of your Stripe [account settings page](https://dashboard.stripe.com/settings/account). 6. For **Secret Key**, enter your Stripe API Key. -7. For **Replication Start Date**, use the provided datepicker or enter the UTC date and time programmatically in the format `YYYY-MM-DDTHH:mm:ssZ`. The data added on and after this date will be replicated. +7. For **Replication Start Date**, use the provided datepicker or enter a UTC date and time programmatically in the format `YYYY-MM-DDTHH:mm:ssZ`. The data added on and after this date will be replicated. 8. (Optional) For **Lookback Window in days**, select the number of days prior to the start date for which you want to sync your data. This value allows the connector to retrieve data that might have been updated after its initial creation. For example, if the start date is set to `2021-01-01T00:00:00Z`, then: - Leaving the Lookback Window in days parameter at its default value of 0 means Airbyte will sync data starting from 2021-01-01T00:00:00Z. - Setting the Lookback Window in days value to 1 means Airbyte will consider the Replication start date to be one day earlier, or 2020-12-31T00:00:00Z. - Setting the Lookback Window in days value to 7 means Airbyte will sync data starting from seven days earlier, or 2020-12-25T00:00:00Z. -9. (Optional) For **Data Sync Frequency**, select the time increment in days for the connector to use when requesting data from the Stripe API. Setting this value higher will reduce the number of requests, speeding up the sync process. However, this will result in data being updated less frequently. The default value is 365 days. +9. (Optional) For **Data Request Window**, select the time increment in days for the connector to use when requesting data from the Stripe API. Setting this value higher will reduce the number of requests, speeding up the sync process. However, this will result in data being updated less frequently. The default value is 365 days. 10. Click **Set up source** and wait for the tests to complete. ## Supported sync modes From ff48ad433e9791872b8a3a75b7de7658d9723dc9 Mon Sep 17 00:00:00 2001 From: Christo Grabowski Date: Mon, 7 Aug 2023 04:31:40 -0400 Subject: [PATCH 3/6] complete main docs update --- docs/integrations/sources/stripe.md | 50 ++++++++++++++++------------- 1 file changed, 27 insertions(+), 23 deletions(-) diff --git a/docs/integrations/sources/stripe.md b/docs/integrations/sources/stripe.md index 42a95fdf667e..8bd42d2b469a 100644 --- a/docs/integrations/sources/stripe.md +++ b/docs/integrations/sources/stripe.md @@ -2,10 +2,6 @@ This page contains the setup guide and reference information for the Stripe source connector. -:::warning -**Stripe API Restriction on Events Data**: Access to the events endpoint is [guaranteed only for the last 30 days](https://stripe.com/docs/api/events). If you use the Full Refresh sync, any events data older than 30 days will be **deleted** from your target destination. -::: - ## Prerequisites - Your [Stripe `Account ID`](https://dashboard.stripe.com/settings/account) @@ -13,7 +9,7 @@ This page contains the setup guide and reference information for the Stripe sour ## Setup Guide -To authenticate the Stripe connector, you need a Stripe Secret Key. Although you may use an existing key, we recommend that you create a new key specifically for Airbyte and grant it **Read** priviliges. We recommend granting **Read** priviliges to all permissions, and configuring the data you would like to replicate in the connector itself. +To authenticate the Stripe connector, you need a Stripe Secret Key. Although you may use an existing key, we recommend that you create a new key specifically for Airbyte and grant it **Read** privileges only. We also recommend granting **Read** privileges to all available permissions, and configuring the specific data you would like to replicate in the connector itself. ### Create a Stripe Secret Key @@ -35,13 +31,15 @@ For more information on Stripe API Keys, see the [Stripe documentation](https:// 5. For **Account ID**, enter your Stripe Account ID. This ID begins with `acct_`, and can be found in the top-right corner of your Stripe [account settings page](https://dashboard.stripe.com/settings/account). 6. For **Secret Key**, enter your Stripe API Key. 7. For **Replication Start Date**, use the provided datepicker or enter a UTC date and time programmatically in the format `YYYY-MM-DDTHH:mm:ssZ`. The data added on and after this date will be replicated. -8. (Optional) For **Lookback Window in days**, select the number of days prior to the start date for which you want to sync your data. This value allows the connector to retrieve data that might have been updated after its initial creation. For example, if the start date is set to `2021-01-01T00:00:00Z`, then: +8. (Optional) For **Lookback Window**, you may specify a number of days prior to the start date for which you want to sync your data. This allows the connector to retrieve data that might have been updated after its initial creation, and is useful for handling any post-transaction adjustments (tips, refunds, chargebacks, etc). + + For example, if the **Replication Start Date** is set to `2021-01-01T00:00:00Z`, then: -- Leaving the Lookback Window in days parameter at its default value of 0 means Airbyte will sync data starting from 2021-01-01T00:00:00Z. -- Setting the Lookback Window in days value to 1 means Airbyte will consider the Replication start date to be one day earlier, or 2020-12-31T00:00:00Z. -- Setting the Lookback Window in days value to 7 means Airbyte will sync data starting from seven days earlier, or 2020-12-25T00:00:00Z. + - Leaving the **Lookback Window** at its default value of 0 means Airbyte will sync data starting from `2021-01-01T00:00:00Z`. + - Setting the **Lookback Window** to 1 means Airbyte will sync data starting from one day earlier, or `2020-12-31T00:00:00Z`. + - Setting the **Lookback Window** to 7 means Airbyte will sync data starting from seven days earlier, or `2020-12-25T00:00:00Z`. -9. (Optional) For **Data Request Window**, select the time increment in days for the connector to use when requesting data from the Stripe API. Setting this value higher will reduce the number of requests, speeding up the sync process. However, this will result in data being updated less frequently. The default value is 365 days. +9. (Optional) For **Data Request Window**, you may specify the time window in days used by the connector when requesting data from the Stripe API. This value represents the span of time for each data retrieval, with larger values encompassing more data in fewer requests. A larger value will lead to faster syncs as fewer requests are made, but it may also result in less frequent state persistence. Choosing the right balance can optimize both the speed and frequency of your data synchronization. The default value is 365 days. 10. Click **Set up source** and wait for the tests to complete. ## Supported sync modes @@ -55,10 +53,11 @@ The Stripe source connector supports the following [sync modes](https://docs.air Since the Stripe API does not allow querying objects which were updated since the last sync, the Stripe connector uses the `created` field to query for new data in your Stripe account. ::: -## Supported Streams +## Supported streams The Stripe source connector supports the following streams: +- [Accounts](https://stripe.com/docs/api/accounts/list) \(Incremental\) - [Application Fees](https://stripe.com/docs/api/application_fees) \(Incremental\) - [Application Fee Refunds](https://stripe.com/docs/api/fee_refunds/list) - [Authorizations](https://stripe.com/docs/api/issuing/authorizations/list) \(Incremental\) @@ -67,18 +66,26 @@ The Stripe source connector supports the following streams: - [Cardholders](https://stripe.com/docs/api/issuing/cardholders/list) \(Incremental\) - [Cards](https://stripe.com/docs/api/issuing/cards/list) \(Incremental\) - [Charges](https://stripe.com/docs/api/charges/list) \(Incremental\) - - The `amount` column defaults to the smallest currency unit. (See [charge object](https://stripe.com/docs/api/charges/object) for more details) + :::note + The `amount` column defaults to the smallest currency unit. Check [the Stripe docs](https://stripe.com/docs/api/charges/object) for more details. + ::: - [Checkout Sessions](https://stripe.com/docs/api/checkout/sessions/list) - [Checkout Sessions Line Items](https://stripe.com/docs/api/checkout/sessions/line_items) - [Coupons](https://stripe.com/docs/api/coupons/list) \(Incremental\) -- [CreditNotes](https://stripe.com/docs/api/credit_notes/list) \(Full Refresh\) +- [Credit Notes](https://stripe.com/docs/api/credit_notes/list) \(Full Refresh\) - [Customer Balance Transactions](https://stripe.com/docs/api/customer_balance_transactions/list) - [Customers](https://stripe.com/docs/api/customers/list) \(Incremental\) - - This endpoint does not include deleted customers + :::note + This endpoint does not include deleted customers + ::: - [Disputes](https://stripe.com/docs/api/disputes/list) \(Incremental\) - [Early Fraud Warnings](https://stripe.com/docs/api/radar/early_fraud_warnings/list) \(Incremental\) - [Events](https://stripe.com/docs/api/events/list) \(Incremental\) - - The Stripe API does not guarantee access to events older than 30 days, so this stream will only pull events created from the 30 days prior to the initial sync and not from the Replication start date. + :::warning + **Stripe API Restriction on Events Data**: Access to the events endpoint is [guaranteed only for the last 30 days](https://stripe.com/docs/api/events) by Stripe. If you use the Full Refresh sync mode, be aware that any events data older than 30 days will be **deleted** from your target destination and replaced with the data from the last 30 days only. + ::: +- [File Links](https://stripe.com/docs/api/file_links/list) \(Incremental\) +- [Files](https://stripe.com/docs/api/files/list) \(Incremental\) - [Invoice Items](https://stripe.com/docs/api/invoiceitems/list) \(Incremental\) - [Invoice Line Items](https://stripe.com/docs/api/invoices/invoice_lines) - [Invoices](https://stripe.com/docs/api/invoices/list) \(Incremental\) @@ -92,24 +99,21 @@ The Stripe source connector supports the following streams: - [Products](https://stripe.com/docs/api/products/list) \(Incremental\) - [Refunds](https://stripe.com/docs/api/refunds/list) \(Incremental\) - [Reviews](https://stripe.com/docs/api/radar/reviews/list) \(Incremental\) -- [SetupIntents](https://stripe.com/docs/api/setup_intents/list) \(Incremental\) -- [ShippingRates](https://stripe.com/docs/api/shipping_rates/list) \(Incremental\) +- [Setup Attempts](https://stripe.com/docs/api/setup_attempts/list) \(Incremental\) +- [Setup Intents](https://stripe.com/docs/api/setup_intents/list) \(Incremental\) +- [Shipping Rates](https://stripe.com/docs/api/shipping_rates/list) \(Incremental\) - [Subscription Items](https://stripe.com/docs/api/subscription_items/list) - [Subscription Schedule](https://stripe.com/docs/api/subscription_schedules) \(Incremental\) - [Subscriptions](https://stripe.com/docs/api/subscriptions/list) \(Incremental\) +- [Top Ups](https://stripe.com/docs/api/topups/list) \(Incremental\) - [Transactions](https://stripe.com/docs/api/transfers/list) \(Incremental\) - [Transfers](https://stripe.com/docs/api/transfers/list) \(Incremental\) - [Transfer Reversals](https://stripe.com/docs/api/transfer_reversals/list) -- [Accounts](https://stripe.com/docs/api/accounts/list) \(Incremental\) -- [Setup Attempts](https://stripe.com/docs/api/setup_attempts/list) \(Incremental\) - [Usage Records](https://stripe.com/docs/api/usage_records/subscription_item_summary_list) -- [TopUps](https://stripe.com/docs/api/topups/list) \(Incremental\) -- [Files](https://stripe.com/docs/api/files/list) \(Incremental\) -- [FileLinks](https://stripe.com/docs/api/file_links/list) \(Incremental\) ### Data type mapping -The [Stripe API](https://stripe.com/docs/api) uses the same [JSONSchema](https://json-schema.org/understanding-json-schema/reference/index.html) types that Airbyte uses internally \(`string`, `date-time`, `object`, `array`, `boolean`, `integer`, and `number`\), so no type conversions are performed for the Stripe connector. +The [Stripe API](https://stripe.com/docs/api) uses the same [JSON Schema](https://json-schema.org/understanding-json-schema/reference/index.html) types that Airbyte uses internally \(`string`, `date-time`, `object`, `array`, `boolean`, `integer`, and `number`\), so no type conversions are performed for the Stripe connector. ### Performance considerations From a8fc55e8d78308c94c30bb3e829c3b1883117e89 Mon Sep 17 00:00:00 2001 From: Christo Grabowski Date: Mon, 7 Aug 2023 11:05:35 -0400 Subject: [PATCH 4/6] update lookback window explanation --- docs/integrations/sources/stripe.inapp.md | 37 +++++++++++++++-------- docs/integrations/sources/stripe.md | 32 +++++++++++--------- 2 files changed, 42 insertions(+), 27 deletions(-) diff --git a/docs/integrations/sources/stripe.inapp.md b/docs/integrations/sources/stripe.inapp.md index 3920f431b763..ed00a983f957 100644 --- a/docs/integrations/sources/stripe.inapp.md +++ b/docs/integrations/sources/stripe.inapp.md @@ -1,24 +1,35 @@ ## Prerequisites -- Your [Stripe `Account ID`](https://dashboard.stripe.com/settings/account) -- Your [Stripe `Secret Key`](https://dashboard.stripe.com/apikeys) +- Access to the Stripe account containing the data you wish to replicate ## Setup Guide -1. Enter a name for your source. -2. For **Account ID**, enter your [Stripe `Account ID`](https://dashboard.stripe.com/settings/account). -3. For **Secret Key**, enter your [Stripe `Secret Key`](https://dashboard.stripe.com/apikeys) +:::note +To authenticate the Stripe connector, you need a Stripe Secret Key with **Read** privileges for the data to replicate. For steps on obtaining and setting permissions for this key, refer to our [full Stripe documentation](https://docs.airbyte.com/integrations/sources/stripe#setup-guide). +::: - We recommend creating a secret key specifically for Airbyte to control which resources Airbyte can access. For ease of use, we recommend granting read permission to all resources and configuring which resource to replicate in the Airbyte UI. You can also use the API keys for the [test mode](https://stripe.com/docs/keys#obtain-api-keys) to try out the Stripe integration with Airbyte. +1. For **Source name**, enter a name to help you identify this source. +2. For **Account ID**, enter your Stripe Account ID. This ID begins with `acct_`, and can be found in the top-right corner of your Stripe [account settings page](https://dashboard.stripe.com/settings/account). +3. For **Secret Key**, enter your Stripe API Key, which can be found at your Stripe [API keys page](https://dashboard.stripe.com/apikeys). +4. For **Replication Start Date**, use the provided datepicker or enter a UTC date and time programmatically in the format `YYYY-MM-DDTHH:mm:ssZ`. The data added on and after this date will be replicated. +5. (Optional) For **Lookback Window**, you may specify a number of days from the present day to reread data. This allows the connector to retrieve data that might have been updated after its initial creation, and is useful for handling any post-transaction adjustments (such as tips, refunds, chargebacks, etc). -4. For **Replication start date**, enter the date in `YYYY-MM-DDTHH:mm:ssZ` format. Data created on and after this date will be replicated. -5. (Optional) For **Lookback Window in days**, select the number of days the value in days prior to the start date that you to sync your data with. If your data is updated after setting up this connector, you can use the this option to reload data from the past N days. Example: If the Replication start date is set to `2021-01-01T00:00:00Z`, then: - - If you leave the Lookback Window in days parameter to its the default value of 0, Airbyte will sync data from the Replication start date `2021-01-01T00:00:00Z` - - If the Lookback Window in days value is set to 1, Airbyte will consider the Replication start date to be `2020-12-31T00:00:00Z` - - If the Lookback Window in days value is set to 7, Airbyte will sync data from `2020-12-25T00:00:00Z` -6. Click **Set up source**. + - Leaving the **Lookback Window** at its default value of 0 means Airbyte will not re-export data after it has been synced. + - Setting the **Lookback Window** to 1 means Airbyte will re-export data from the past day, capturing any changes made in the last 24 hours. + - Setting the **Lookback Window** to 7 means Airbyte will re-export and capture any data changes within the last week. + +6. (Optional) For **Data Request Window**, you may specify the time window in days for data retrieval from the Stripe API. This window defines how much data is gathered in each request, with larger values retrieving more data but making fewer requests. + + For example, if you are replicating three years worth of data: + + - A **Data Request Window** of 365 days means Airbyte makes 3 requests, each for a year. This is generally faster but risks needing to resync up to a year's data if something goes wrong. + - A **Data Request Window** of 30 days means 36 requests, each for a month. This may be slower but minimizes the amount of data that needs to be resynced if an issue occurs. + + If you are unsure of which value to use, we recommend leaving this setting at its default value of 365 days. +7. Click **Set up source** and wait for the tests to complete. ### Stripe API limitations -- When syncing `events` data from Stripe, data is only [returned for the last 30 days](https://stripe.com/docs/api/events). Using the full-refresh-overwrite sync from Airbyte will delete the events data older than 30 days from your target destination. Use an append sync mode to ensure historical data is retained. + +- When syncing `events` data from Stripe, data is only [returned for the last 30 days](https://stripe.com/docs/api/events). Using the Full Refresh (Overwrite) sync from Airbyte will delete the events data older than 30 days from your target destination. Use an Append sync mode to ensure historical data is retained. For detailed information on supported sync modes, supported streams, performance considerations, refer to the full documentation for [Stripe](https://docs.airbyte.com/integrations/sources/stripe). diff --git a/docs/integrations/sources/stripe.md b/docs/integrations/sources/stripe.md index 8bd42d2b469a..0bb3db43064c 100644 --- a/docs/integrations/sources/stripe.md +++ b/docs/integrations/sources/stripe.md @@ -4,12 +4,11 @@ This page contains the setup guide and reference information for the Stripe sour ## Prerequisites -- Your [Stripe `Account ID`](https://dashboard.stripe.com/settings/account) -- Your [Stripe `Secret Key`](https://dashboard.stripe.com/apikeys) +- Access to the Stripe account containing the data you wish to replicate ## Setup Guide -To authenticate the Stripe connector, you need a Stripe Secret Key. Although you may use an existing key, we recommend that you create a new key specifically for Airbyte and grant it **Read** privileges only. We also recommend granting **Read** privileges to all available permissions, and configuring the specific data you would like to replicate in the connector itself. +To authenticate the Stripe connector, you need to use a Stripe Secret Key. Although you may use an existing key, we recommend that you create a new restricted key specifically for Airbyte and grant it **Read** privileges only. We also recommend granting **Read** privileges to all available permissions, and configuring the specific data you would like to replicate within Airbyte. ### Create a Stripe Secret Key @@ -17,8 +16,8 @@ To authenticate the Stripe connector, you need a Stripe Secret Key. Although you 2. In the top navigation bar, click **Developers**. 3. In the top-left corner, click **API keys**. 4. Click **+ Create restricted key**. -5. Choose a **Key name**, and select **Read** for all permissions. -6. Click **Create key**. +5. Choose a **Key name**, and select **Read** for all available permissions. +6. Click **Create key**. You may be prompted to enter a confirmation code sent to your email address. For more information on Stripe API Keys, see the [Stripe documentation](https://stripe.com/docs/keys). @@ -29,17 +28,22 @@ For more information on Stripe API Keys, see the [Stripe documentation](https:// 3. Find and select **Stripe** from the list of available sources. 4. For **Source name**, enter a name to help you identify this source. 5. For **Account ID**, enter your Stripe Account ID. This ID begins with `acct_`, and can be found in the top-right corner of your Stripe [account settings page](https://dashboard.stripe.com/settings/account). -6. For **Secret Key**, enter your Stripe API Key. +6. For **Secret Key**, enter the restricted key you created for the connection. 7. For **Replication Start Date**, use the provided datepicker or enter a UTC date and time programmatically in the format `YYYY-MM-DDTHH:mm:ssZ`. The data added on and after this date will be replicated. -8. (Optional) For **Lookback Window**, you may specify a number of days prior to the start date for which you want to sync your data. This allows the connector to retrieve data that might have been updated after its initial creation, and is useful for handling any post-transaction adjustments (tips, refunds, chargebacks, etc). +8. (Optional) For **Lookback Window**, you may specify a number of days from the present day to reread data. This allows the connector to retrieve data that might have been updated after its initial creation, and is useful for handling any post-transaction adjustments (such as tips, refunds, chargebacks, etc). - For example, if the **Replication Start Date** is set to `2021-01-01T00:00:00Z`, then: + - Leaving the **Lookback Window** at its default value of 0 means Airbyte will not re-export data after it has been synced. + - Setting the **Lookback Window** to 1 means Airbyte will re-export data from the past day, capturing any changes made in the last 24 hours. + - Setting the **Lookback Window** to 7 means Airbyte will re-export and capture any data changes within the last week. - - Leaving the **Lookback Window** at its default value of 0 means Airbyte will sync data starting from `2021-01-01T00:00:00Z`. - - Setting the **Lookback Window** to 1 means Airbyte will sync data starting from one day earlier, or `2020-12-31T00:00:00Z`. - - Setting the **Lookback Window** to 7 means Airbyte will sync data starting from seven days earlier, or `2020-12-25T00:00:00Z`. +9. (Optional) For **Data Request Window**, you may specify the time window in days for data retrieval from the Stripe API. This window defines how much data is gathered in each request, with larger values retrieving more data but making fewer requests. -9. (Optional) For **Data Request Window**, you may specify the time window in days used by the connector when requesting data from the Stripe API. This value represents the span of time for each data retrieval, with larger values encompassing more data in fewer requests. A larger value will lead to faster syncs as fewer requests are made, but it may also result in less frequent state persistence. Choosing the right balance can optimize both the speed and frequency of your data synchronization. The default value is 365 days. + For example, if you are replicating three years worth of data: + + - A **Data Request Window** of 365 days means Airbyte makes 3 requests, each for a year. This is generally faster but risks needing to resync up to a year's data if something goes wrong. + - A **Data Request Window** of 30 days means 36 requests, each for a month. This may be slower but minimizes the amount of data that needs to be resynced if an issue occurs. + + If you are unsure of which value to use, we recommend leaving this setting at its default value of 365 days. 10. Click **Set up source** and wait for the tests to complete. ## Supported sync modes @@ -76,13 +80,13 @@ The Stripe source connector supports the following streams: - [Customer Balance Transactions](https://stripe.com/docs/api/customer_balance_transactions/list) - [Customers](https://stripe.com/docs/api/customers/list) \(Incremental\) :::note - This endpoint does not include deleted customers + This endpoint does _not_ include deleted customers ::: - [Disputes](https://stripe.com/docs/api/disputes/list) \(Incremental\) - [Early Fraud Warnings](https://stripe.com/docs/api/radar/early_fraud_warnings/list) \(Incremental\) - [Events](https://stripe.com/docs/api/events/list) \(Incremental\) :::warning - **Stripe API Restriction on Events Data**: Access to the events endpoint is [guaranteed only for the last 30 days](https://stripe.com/docs/api/events) by Stripe. If you use the Full Refresh sync mode, be aware that any events data older than 30 days will be **deleted** from your target destination and replaced with the data from the last 30 days only. + **Stripe API Restriction on Events Data**: Access to the events endpoint is [guaranteed only for the last 30 days](https://stripe.com/docs/api/events) by Stripe. If you use the Full Refresh Overwrite sync, be aware that any events data older than 30 days will be **deleted** from your target destination and replaced with the data from the last 30 days only. Use an Append sync mode to ensure historical data is retained. ::: - [File Links](https://stripe.com/docs/api/file_links/list) \(Incremental\) - [Files](https://stripe.com/docs/api/files/list) \(Incremental\) From fdb9394769193418cfaff9278a17cb39552078f6 Mon Sep 17 00:00:00 2001 From: Christo Grabowski Date: Mon, 7 Aug 2023 11:26:47 -0400 Subject: [PATCH 5/6] small fix to lookback window example --- docs/integrations/sources/stripe.inapp.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/integrations/sources/stripe.inapp.md b/docs/integrations/sources/stripe.inapp.md index ed00a983f957..13372766da20 100644 --- a/docs/integrations/sources/stripe.inapp.md +++ b/docs/integrations/sources/stripe.inapp.md @@ -15,7 +15,7 @@ To authenticate the Stripe connector, you need a Stripe Secret Key with **Read** 5. (Optional) For **Lookback Window**, you may specify a number of days from the present day to reread data. This allows the connector to retrieve data that might have been updated after its initial creation, and is useful for handling any post-transaction adjustments (such as tips, refunds, chargebacks, etc). - Leaving the **Lookback Window** at its default value of 0 means Airbyte will not re-export data after it has been synced. - - Setting the **Lookback Window** to 1 means Airbyte will re-export data from the past day, capturing any changes made in the last 24 hours. + - Setting the **Lookback Window** to 1 means Airbyte will re-export and capture any data changes within the last day. - Setting the **Lookback Window** to 7 means Airbyte will re-export and capture any data changes within the last week. 6. (Optional) For **Data Request Window**, you may specify the time window in days for data retrieval from the Stripe API. This window defines how much data is gathered in each request, with larger values retrieving more data but making fewer requests. From 1138f4dbbf83b5918a601b3ff63330c6ab5fc242 Mon Sep 17 00:00:00 2001 From: Christo Grabowski Date: Wed, 9 Aug 2023 08:40:52 -0400 Subject: [PATCH 6/6] update data request description --- docs/integrations/sources/stripe.inapp.md | 10 +++++----- docs/integrations/sources/stripe.md | 13 +++++++------ 2 files changed, 12 insertions(+), 11 deletions(-) diff --git a/docs/integrations/sources/stripe.inapp.md b/docs/integrations/sources/stripe.inapp.md index 13372766da20..5988ba81922d 100644 --- a/docs/integrations/sources/stripe.inapp.md +++ b/docs/integrations/sources/stripe.inapp.md @@ -1,16 +1,16 @@ ## Prerequisites -- Access to the Stripe account containing the data you wish to replicate +- Access to the Stripe account containing the data to replicate ## Setup Guide :::note -To authenticate the Stripe connector, you need a Stripe Secret Key with **Read** privileges for the data to replicate. For steps on obtaining and setting permissions for this key, refer to our [full Stripe documentation](https://docs.airbyte.com/integrations/sources/stripe#setup-guide). +To authenticate the Stripe connector, you need an API key with **Read** privileges for the data to replicate. For steps on obtaining and setting permissions for this key, refer to our [full Stripe documentation](https://docs.airbyte.com/integrations/sources/stripe#setup-guide). ::: 1. For **Source name**, enter a name to help you identify this source. 2. For **Account ID**, enter your Stripe Account ID. This ID begins with `acct_`, and can be found in the top-right corner of your Stripe [account settings page](https://dashboard.stripe.com/settings/account). -3. For **Secret Key**, enter your Stripe API Key, which can be found at your Stripe [API keys page](https://dashboard.stripe.com/apikeys). +3. For **Secret Key**, enter your Stripe API key, which can be found at your Stripe [API keys page](https://dashboard.stripe.com/apikeys). 4. For **Replication Start Date**, use the provided datepicker or enter a UTC date and time programmatically in the format `YYYY-MM-DDTHH:mm:ssZ`. The data added on and after this date will be replicated. 5. (Optional) For **Lookback Window**, you may specify a number of days from the present day to reread data. This allows the connector to retrieve data that might have been updated after its initial creation, and is useful for handling any post-transaction adjustments (such as tips, refunds, chargebacks, etc). @@ -18,11 +18,11 @@ To authenticate the Stripe connector, you need a Stripe Secret Key with **Read** - Setting the **Lookback Window** to 1 means Airbyte will re-export and capture any data changes within the last day. - Setting the **Lookback Window** to 7 means Airbyte will re-export and capture any data changes within the last week. -6. (Optional) For **Data Request Window**, you may specify the time window in days for data retrieval from the Stripe API. This window defines how much data is gathered in each request, with larger values retrieving more data but making fewer requests. +6. (Optional) For **Data Request Window**, you may specify the time window in days used by the connector when requesting data from the Stripe API. This window defines the span of time covered in each request, with larger values encompassing more days in a single request. Generally speaking, the lack of overhead from making fewer requests means a larger window is faster to sync. However, this also means the state of the sync will persist less frequently. If an issue occurs or the sync is interrupted, a larger window means more data will need to be resynced, potentially causing a delay in the overall process. For example, if you are replicating three years worth of data: - - A **Data Request Window** of 365 days means Airbyte makes 3 requests, each for a year. This is generally faster but risks needing to resync up to a year's data if something goes wrong. + - A **Data Request Window** of 365 days means Airbyte makes 3 requests, each for a year. This is generally faster but risks needing to resync up to a year's data if the sync is interrupted. - A **Data Request Window** of 30 days means 36 requests, each for a month. This may be slower but minimizes the amount of data that needs to be resynced if an issue occurs. If you are unsure of which value to use, we recommend leaving this setting at its default value of 365 days. diff --git a/docs/integrations/sources/stripe.md b/docs/integrations/sources/stripe.md index 0bb3db43064c..18a09ec0027f 100644 --- a/docs/integrations/sources/stripe.md +++ b/docs/integrations/sources/stripe.md @@ -8,7 +8,7 @@ This page contains the setup guide and reference information for the Stripe sour ## Setup Guide -To authenticate the Stripe connector, you need to use a Stripe Secret Key. Although you may use an existing key, we recommend that you create a new restricted key specifically for Airbyte and grant it **Read** privileges only. We also recommend granting **Read** privileges to all available permissions, and configuring the specific data you would like to replicate within Airbyte. +To authenticate the Stripe connector, you need to use a Stripe API key. Although you may use an existing key, we recommend that you create a new restricted key specifically for Airbyte and grant it **Read** privileges only. We also recommend granting **Read** privileges to all available permissions, and configuring the specific data you would like to replicate within Airbyte. ### Create a Stripe Secret Key @@ -36,11 +36,11 @@ For more information on Stripe API Keys, see the [Stripe documentation](https:// - Setting the **Lookback Window** to 1 means Airbyte will re-export data from the past day, capturing any changes made in the last 24 hours. - Setting the **Lookback Window** to 7 means Airbyte will re-export and capture any data changes within the last week. -9. (Optional) For **Data Request Window**, you may specify the time window in days for data retrieval from the Stripe API. This window defines how much data is gathered in each request, with larger values retrieving more data but making fewer requests. +9. (Optional) For **Data Request Window**, you may specify the time window in days used by the connector when requesting data from the Stripe API. This window defines the span of time covered in each request, with larger values encompassing more days in a single request. Generally speaking, the lack of overhead from making fewer requests means a larger window is faster to sync. However, this also means the state of the sync will persist less frequently. If an issue occurs or the sync is interrupted, a larger window means more data will need to be resynced, potentially causing a delay in the overall process. For example, if you are replicating three years worth of data: - - A **Data Request Window** of 365 days means Airbyte makes 3 requests, each for a year. This is generally faster but risks needing to resync up to a year's data if something goes wrong. + - A **Data Request Window** of 365 days means Airbyte makes 3 requests, each for a year. This is generally faster but risks needing to resync up to a year's data if the sync is interrupted. - A **Data Request Window** of 30 days means 36 requests, each for a month. This may be slower but minimizes the amount of data that needs to be resynced if an issue occurs. If you are unsure of which value to use, we recommend leaving this setting at its default value of 365 days. @@ -85,9 +85,6 @@ The Stripe source connector supports the following streams: - [Disputes](https://stripe.com/docs/api/disputes/list) \(Incremental\) - [Early Fraud Warnings](https://stripe.com/docs/api/radar/early_fraud_warnings/list) \(Incremental\) - [Events](https://stripe.com/docs/api/events/list) \(Incremental\) - :::warning - **Stripe API Restriction on Events Data**: Access to the events endpoint is [guaranteed only for the last 30 days](https://stripe.com/docs/api/events) by Stripe. If you use the Full Refresh Overwrite sync, be aware that any events data older than 30 days will be **deleted** from your target destination and replaced with the data from the last 30 days only. Use an Append sync mode to ensure historical data is retained. - ::: - [File Links](https://stripe.com/docs/api/file_links/list) \(Incremental\) - [Files](https://stripe.com/docs/api/files/list) \(Incremental\) - [Invoice Items](https://stripe.com/docs/api/invoiceitems/list) \(Incremental\) @@ -115,6 +112,10 @@ The Stripe source connector supports the following streams: - [Transfer Reversals](https://stripe.com/docs/api/transfer_reversals/list) - [Usage Records](https://stripe.com/docs/api/usage_records/subscription_item_summary_list) +:::warning +**Stripe API Restriction on Events Data**: Access to the events endpoint is [guaranteed only for the last 30 days](https://stripe.com/docs/api/events) by Stripe. If you use the Full Refresh Overwrite sync, be aware that any events data older than 30 days will be **deleted** from your target destination and replaced with the data from the last 30 days only. Use an Append sync mode to ensure historical data is retained. +::: + ### Data type mapping The [Stripe API](https://stripe.com/docs/api) uses the same [JSON Schema](https://json-schema.org/understanding-json-schema/reference/index.html) types that Airbyte uses internally \(`string`, `date-time`, `object`, `array`, `boolean`, `integer`, and `number`\), so no type conversions are performed for the Stripe connector.