Skip to content

Commit

Permalink
[Docs] Simplify Connection Settings (#36502)
Browse files Browse the repository at this point in the history
Co-authored-by: Chandler Prall <chandler.prall@gmail.com>
  • Loading branch information
nataliekwong and chandlerprall authored Mar 26, 2024
1 parent 7382c87 commit 1bb83ec
Show file tree
Hide file tree
Showing 16 changed files with 74 additions and 64 deletions.
24 changes: 8 additions & 16 deletions docs/cloud/managing-airbyte-cloud/configuring-connections.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ To configure these settings:

1. In the Airbyte UI, click **Connections** and then click the connection you want to change.

2. Click the **Replication** tab.
2. Click the **Settings** tab.

3. Click the **Configuration** dropdown to expand the options.

Expand All @@ -29,31 +29,23 @@ You can configure the following settings:
| Setting | Description |
|--------------------------------------|-------------------------------------------------------------------------------------|
| Connection Name | A custom name for your connection |
| [Replication frequency](/using-airbyte/core-concepts/sync-schedules.md) | How often data syncs (can be scheduled, cron, API-triggered or manual) |
| [Destination namespace](/using-airbyte/core-concepts/namespaces.md) | Where the replicated data is written to in the destination |
| Destination stream prefix | A prefix added to each table name in the destination |
| [Schedule Type](/using-airbyte/core-concepts/sync-schedules.md) | How often data syncs (can be scheduled, cron, API-triggered or manual) |
| [Destination Namespace](/using-airbyte/core-concepts/namespaces.md) | Where the replicated data is written to in the destination |
| Destination Stream Prefix | A prefix added to each table name in the destination |
| [Detect and propagate schema changes](/cloud/managing-airbyte-cloud/manage-schema-changes.md) | How Airbyte handles schema changes in the source |
| [Connection Data Residency](/cloud/managing-airbyte-cloud/manage-data-residency.md) | Where data will be processed (Cloud only) |

## Modify streams in your connection
## Modify Streams

In the **Activate the streams you want to sync** table, you choose which streams to sync and how they are loaded to the destination.
On the "Schema" tab, you choose which streams to sync and how they are loaded to the destination.

:::info
A connection's schema consists of one or many streams. Each stream is most commonly associated with a database table or an API endpoint. Within a stream, there can be one or many fields or columns.
:::

To modify streams:
To modify streams, click **Connections** and then click the connection you want to change. Click the **Schema** tab to see all the streams Airbyte can sync. To modify an individual stream:

1. In the Airbyte UI, click **Connections** and then click the connection you want to change.

2. Click the **Replication** tab.

3. Scroll down to the **Activate the streams you want to sync** table.

Modify an individual stream:

1. In the **Activate the streams you want to sync** table, toggle **Sync** on or off for your selected stream. To select or deselect all streams, click the checkbox in the table header. To deselect an individual stream, deselect its checkbox in the table.
1. Toggle **Sync** on or off for your selected stream. To select or deselect all streams at once, use "Hide disabled streams" in the table header. To deselect an individual stream, use the toggle in its row.

2. Click the **Sync mode** dropdown and select the sync mode you want to apply. Depending on the sync mode you select, you may need to choose a cursor or primary key.

Expand Down
2 changes: 1 addition & 1 deletion docs/cloud/managing-airbyte-cloud/manage-credits.md
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,7 @@ To purchase credits directly through the UI,

## Automatic reload of credits

You can enroll in automatic top-ups of your credit balance. Thie feature is for those who do not want to manually add credits each time.
You can enroll in automatic top-ups of your credit balance. This feature is for those who do not want to manually add credits each time.

To enroll, [email us](mailto:billing@airbyte.io) with:

Expand Down
37 changes: 19 additions & 18 deletions docs/using-airbyte/core-concepts/namespaces.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,9 +10,10 @@ As a part of connection setup, you select where in the destination you want to w

| Destination Namespace | Description |
| ---------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------- |
| Destination default | All streams will be replicated to the single default namespace defined in the Destination's settings. |
| Mirror source structure | Some sources (for example, databases) provide namespace information for a stream. If a source provides namespace information, the destination will mirror the same namespace when this configuration is set. For sources or streams where the source namespace is not known, the behavior will default to the "Destination default" option. |
| Custom format | All streams will be replicated to a single user-defined namespace. See<a href="/understanding-airbyte/namespaces#--custom-format"> Custom format</a> for more details |
| Custom | All streams will be replicated to a single user-defined namespace. See<a href="/understanding-airbyte/namespaces#--custom-format"> Custom format</a> for more details |
| Destination-defined | All streams will be replicated to the single default namespace defined in the Destination's settings. |
| Source-defined | Some sources (for example, databases) provide namespace information for a stream. If a source provides namespace information, the destination will mirror the same namespace when this configuration is set. For sources or streams where the source namespace is not known, the behavior will default to the "Destination default" option. |


Most of our destinations support this feature. To learn if your connector supports this, head to the individual connector page to learn more. If your desired destination doesn't support it, you can ignore this feature.

Expand All @@ -26,7 +27,19 @@ In a source, the namespace is the location from where the data is replicated to

Airbyte supports namespaces and allows Sources to define namespaces, and Destinations to write to various namespaces. In Airbyte, the following options are available and are set on each individual connection.

### Destination default
### Custom

When replicating multiple sources into the same destination, you may create table conflicts where tables are overwritten by different syncs. This is where using a custom namespace will ensure data is synced accurately.

For example, a Github source can be replicated into a `github` schema. However, you may have multiple connections writing from different GitHub repositories \(common in multi-tenant scenarios\).

:::tip
To write more than 1 table with the same name to your destination, Airbyte recommends writing the connections to unique namespaces to avoid mixing data from the different GitHub repositories.
:::

You can enter plain text (most common) or additionally add a dynamic parameter `${SOURCE_NAMESPACE}`, which uses the namespace provided by the source if available.

### Destination-defined

All streams will be replicated and stored in the default namespace defined on the destination settings page, which is typically defined when the destination was set up. Depending on your destination, the namespace refers to:

Expand All @@ -45,21 +58,9 @@ All streams will be replicated and stored in the default namespace defined on th
If you prefer to replicate multiple sources into the same namespace, use the `Stream Prefix` configuration to differentiate data from these sources to ensure no streams collide when writing to the destination.
:::

### Mirror source structure
### Source-Defined

Some sources \(such as databases based on JDBC\) provide namespace information from which a stream has been extracted. Whenever a source is able to fill this field in the catalog.json file, the destination will try to write to exactly the same namespace when this configuration is set. For sources or streams where the source namespace is not known, the behavior will fall back to the "Destination default". Most APIs do not provide namespace information.

### Custom format

When replicating multiple sources into the same destination, you may create table conflicts where tables are overwritten by different syncs. This is where using a custom namespace will ensure data is synced accurately.

For example, a Github source can be replicated into a `github` schema. However, you may have multiple connections writing from different GitHub repositories \(common in multi-tenant scenarios\).

:::tip
To write more than 1 table with the same name to your destination, Airbyte recommends writing the connections to unique namespaces to avoid mixing data from the different GitHub repositories.
:::

You can enter plain text (most common) or additionally add a dynamic parameter `${SOURCE_NAMESPACE}`, which uses the namespace provided by the source if available.
Some sources \(such as databases based on JDBC\) provide namespace information from which a stream has been extracted. Whenever a source is able to fill this field in the catalog.json file, the destination will try to write to exactly the same namespace when this configuration is set. For sources or streams where the source namespace is not known, the behavior will fall back to the default namespace defined in the destination configuration. Most APIs do not provide namespace information.

### Examples

Expand Down
8 changes: 4 additions & 4 deletions docs/using-airbyte/core-concepts/readme.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,11 +26,11 @@ A connection is an automated data pipeline that replicates data from a source to

| Concept | Description |
|-----------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------|
| [Replication Frequency](/using-airbyte/core-concepts/sync-schedules.md) | When should a data sync be triggered? |
| [Destination Namespace and Stream Prefix](/using-airbyte/core-concepts/namespaces.md) | Where should the replicated data be written? |
| [Catalog Selection](/cloud/managing-airbyte-cloud/configuring-connections.md#modify-streams-in-your-connection) | What data should be replicated from the source to the destination? |
| [Sync Mode](/using-airbyte/core-concepts/sync-modes/README.md) | How should the streams be replicated (read and written)? |
| [Sync Schedule](/using-airbyte/core-concepts/sync-schedules.md) | When should a data sync be triggered? |
| [Destination Namespace and Stream Prefix](/using-airbyte/core-concepts/namespaces.md) | Where should the replicated data be written? |
| [Schema Propagation](/cloud/managing-airbyte-cloud/manage-schema-changes.md) | How should Airbyte handle schema drift in sources? |
| [Catalog Selection](/cloud/managing-airbyte-cloud/configuring-connections.md#modify-streams-in-your-connection) | What data should be replicated from the source to the destination? |

## Stream

Expand All @@ -51,7 +51,7 @@ Examples of fields:
- A column in the table in a relational database
- A field in an API response

## Sync Schedules
## Sync Schedule

There are three options for scheduling a sync to run:

Expand Down
17 changes: 13 additions & 4 deletions docs/using-airbyte/core-concepts/sync-schedules.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,13 +4,13 @@ products: all

# Sync Schedules

For each connection, you can select between three options that allow a sync to run. The three options for `Replication Frequency` are:
For each connection, you can select between three options that allow a sync to run. The three options for `Schedule Type` are:

- Scheduled (e.g. every 24 hours, every 2 hours)
- Cron scheduling
- Cron
- Manual

## Sync Limitations
## Sync Considerations

* Only one sync per connection can run at a time.
* If a sync is scheduled to run before the previous sync finishes, the scheduled sync will start after the completion of the previous sync.
Expand All @@ -21,6 +21,15 @@ For Scheduled or cron scheduled syncs, Airbyte guarantees syncs will initiate wi
:::

## Scheduled syncs
You can choose between the following scheduled options:
- Every 24 hours (most common)
- Every 12 hours
- Every 8 hours
- Every 6 hours
- Every 3 hours
- Every 2 hours
- Every 1 hour

When a scheduled connection is first created, a sync is executed immediately after creation. After that, a sync is run once the time since the last sync \(whether it was triggered manually or due to a schedule\) has exceeded the schedule interval. For example:

- **October 1st, 2pm**, a user sets up a connection to sync data every 24 hours.
Expand All @@ -30,7 +39,7 @@ When a scheduled connection is first created, a sync is executed immediately aft
- **October 3rd, 2:01pm:** since the last sync was less than 24 hours ago, no sync is run
- **October 3rd, 5:01pm:** It has been more than 24 hours since the last sync, so a sync is run

## Cron Scheduling
## Cron Syncs
If you prefer more precision in scheduling your sync, you can also use CRON scheduling to set a specific time of day or month.

Airbyte uses the CRON scheduler from [Quartz](http://www.quartz-scheduler.org/documentation/quartz-2.3.0/tutorials/crontrigger.html). We recommend reading their [documentation](http://www.quartz-scheduler.org/documentation/quartz-2.3.0/tutorials/crontrigger.html) to understand the required formatting. You can also refer to these examples:
Expand Down
2 changes: 1 addition & 1 deletion docs/using-airbyte/getting-started/add-a-destination.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ Destinations are the data warehouses, data lakes, databases and analytics tools

Once you've signed up for Airbyte Cloud or logged in to your Airbyte Open Source deployment, click on the **Destinations** tab in the navigation bar found on the left side of the dashboard. This will take you to the list of available destinations.

![Destination List](./assets/getting-started-destination-list.png)
![Destination List](./assets/getting-started-destination-catalog.png)

You can use the provided search bar at the top of the page, or scroll down the list to find the destination you want to replicate data from.

Expand Down
3 changes: 1 addition & 2 deletions docs/using-airbyte/getting-started/add-a-source.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,8 +6,7 @@ products: all

Setting up a new source in Airbyte is a quick and simple process! When viewing the Airbyte UI, you'll see the main navigation bar on the left side of your screen. Click the **Sources** tab to bring up a list of all available sources.

<Arcade id="4V0TGOX02P0rwVNwz4MR" title="Getting Started (Source)" />

<Arcade id="WjbChISa931Hc55yx4cb" title="Getting Started (Source)" />

You can use the provided search bar, or simply scroll down the list to find the source you want to replicate data from. Let's use a demo source, Faker, as an example. Clicking on the **Sample Data (Faker)** card will bring us to its setup page.

Expand Down
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file not shown.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file not shown.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file not shown.
Loading

0 comments on commit 1bb83ec

Please sign in to comment.