Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix postgres docs broken links #29317

Merged
merged 3 commits into from
Aug 10, 2023
Merged

fix postgres docs broken links #29317

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
5570ce5
fixes to postgres docs
Hesperide Aug 10, 2023
87b095d
no relative links
Hesperide Aug 10, 2023
94ee285
Update docs/integrations/sources/postgres.md
Hesperide Aug 10, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
53 changes: 22 additions & 31 deletions docs/integrations/sources/postgres.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,16 +1,13 @@
import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';

# Postgres

Airbyte's certified Postgres connector offers the following features:
* Multiple methods of keeping your data fresh, including [Change Data Capture (CDC)](./../../understanding-airbyte/cdc.md) and replication using the [xmin system column](#xmin).
* Multiple methods of keeping your data fresh, including [Change Data Capture (CDC)](https://docs.airbyte.com/understanding-airbyte/cdc) and replication using the [xmin system column](#xmin).
* All available [sync modes](https://docs.airbyte.com/cloud/core-concepts#connection-sync-modes), providing flexibility in how data is delivered to your destination.
* Reliable replication at any table size with [checkpointing](./../../understanding-airbyte/airbyte-protocol.md#state--checkpointing) and chunking of database reads.
* Reliable replication at any table size with [checkpointing](https://docs.airbyte.com/understanding-airbyte/airbyte-protocol/#state--checkpointing) and chunking of database reads.

The contents below include a 'Quick Start' guide, advanced setup steps, and reference information (data type mapping, and changelogs). See [here](./postgres/postgres-troubleshooting.md) to troubleshooting issues with the Postgres connector.
The contents below include a 'Quick Start' guide, advanced setup steps, and reference information (data type mapping, and changelogs). See [here](https://docs.airbyte.com/integrations/sources/postgres/postgres-troubleshooting) to troubleshooting issues with the Postgres connector.

![Airbyte Postgres Connection](./postgres/assets/airbyte_postgres_source.png)
![Airbyte Postgres Connection](https://raw.githubusercontent.com/airbytehq/airbyte/c078e8ed6703020a584d9362efa5665fbe8db77f/docs/integrations/sources/postgres/assets/airbyte_postgres_source.png?raw=true)

## Quick Start

Expand Down Expand Up @@ -43,7 +40,7 @@ ALTER DEFAULT PRIVILEGES IN SCHEMA <schema_name> GRANT SELECT ON TABLES TO <user

From your [Airbyte Cloud](https://cloud.airbyte.com/workspaces) or Airbyte Open Source account, select `Sources` from the left navigation bar, search for `Postgres`, then create a new Postgres source.

![Create an Airbyte source](./postgres/assets/airbyte_source_selection.png)
![Create an Airbyte source](https://github.com/airbytehq/airbyte/blob/c078e8ed6703020a584d9362efa5665fbe8db77f/docs/integrations/sources/postgres/assets/airbyte_source_selection.png?raw=true)

To fill out the required information:
1. Enter the hostname, port number, and name for your Postgres database.
Expand Down Expand Up @@ -78,8 +75,8 @@ Now, click `Set up source` in the Airbyte UI. Airbyte will now test connecting t
### Setup using CDC

Airbyte uses [logical replication](https://www.postgresql.org/docs/10/logical-replication.html) of the Postgres write-ahead log (WAL) to incrementally capture deletes using a replication plugin:
* See [here](./../../understanding-airbyte/cdc.md) to learn more on how Airbyte implements CDC.
* See [here](./postgres/postgres-troubleshooting.md#cdc-requirements) to learn more about Postgres CDC requirements and limitations.
* See [here](https://docs.airbyte.com/understanding-airbyte/cdc) to learn more on how Airbyte implements CDC.
* See [here](https://docs.airbyte.com/integrations/sources/postgres/postgres-troubleshooting#cdc-requirements) to learn more about Postgres CDC requirements and limitations.

We recommend configuring your Postgres source with CDC when:
- You need a record of deletions.
Expand Down Expand Up @@ -108,32 +105,26 @@ ALTER USER <user_name> REPLICATION;

#### Step 3: Enable logical replication on your Postgres database

<Tabs defaultValue="generic">
<TabItem value="generic" label="Generic">
To enable logical replication on bare metal, VMs (EC2/GCE/etc), or Docker, configure the following parameters in the <a href="http://https://www.postgresql.org/docs/current/config-setting.html">postgresql.conf file</a> for your Postgres database:
To enable logical replication on bare metal, VMs (EC2/GCE/etc), or Docker, configure the following parameters in the <a href="http://https://www.postgresql.org/docs/current/config-setting.html">postgresql.conf file</a> for your Postgres database:

| Parameter | Description | Set value to |
| --------------------- | ------------------------------------------------------------------------------ |------------------------------------------------------------------------------------------------------------------------------------|
| wal_level | Type of coding used within the Postgres write-ahead log | `logical ` |
| max_wal_senders | The maximum number of processes used for handling WAL changes | `min: 1` |
| max_replication_slots | The maximum number of replication slots that are allowed to stream WAL changes | `1` (if Airbyte is the only service reading subscribing to WAL changes. More than 1 if other services are also reading from the WAL) |
</TabItem>
<TabItem value="aws" label="Amazon RDS / Aurora">
<p>To enable logical replication on AWS Postgres RDS or Aurora:</p>
<ul>
<li>Go to the Configuration tab for your DB cluster.</li>
<li>Find your cluster parameter group. Either edit the parameters for this group or create a copy of this parameter group to edit. If you create a copy, change your cluster's parameter group before restarting.</li>
<li>Within the parameter group page, search for <code>rds.logical_replication</code>. Select this row and click Edit parameters. Set this value to 1.</li>
<li>Wait for a maintenance window to automatically restart the instance or restart it manually.</li>
</ul>
</TabItem>
<TabItem value="azure" label="Azure Database">
<p>To enable logical replication on Azure Database for Postgres:</p>
<p>Change the replication mode of your Postgres DB on Azure to <code>logical</code> using the <code>Replication</code> menu of your PostgreSQL instance in the Azure Portal. Alternatively, use the Azure CLI to run the following command:</p>
<pre><code>az postgres server configuration set --resource-group group --server-name server --name azure.replication_support --value logical
az postgres server restart --resource-group group --name server</code></pre>
</TabItem>
</Tabs>

To enable logical replication on AWS Postgres RDS or Aurora:
* Go to the Configuration tab for your DB cluster.
* Find your cluster parameter group. Either edit the parameters for this group or create a copy of this parameter group to edit. If you create a copy, change your cluster's parameter group before restarting.
* Within the parameter group page, search for `rds.logical_replication`. Select this row and click Edit parameters. Set this value to 1.
* Wait for a maintenance window to automatically restart the instance or restart it manually.

To enable logical replication on Azure Database for Postgres, change the replication mode of your Postgres DB on Azure to `logical` using the replication menu of your PostgreSQL instance in the Azure Portal. Alternatively, use the Azure CLI to run the following command:

```
az postgres server configuration set --resource-group group --server-name server --name azure.replication_support --value logical
az postgres server restart --resource-group group --name server
```

#### Step 4: Create a replication slot on your Postgres database

Expand Down Expand Up @@ -257,7 +248,7 @@ The command produces the private key in PEM format and the public key remains in

## Limitations & Troubleshooting

To see connector limitations, or troubleshoot your Postgres connector, see more [in our Postgres troubleshooting guide](/integrations/sources/postgres/postgres-troubleshooting.md).
To see connector limitations, or troubleshoot your Postgres connector, see more [in our Postgres troubleshooting guide](https://docs.airbyte.com/integrations/sources/postgres/postgres-troubleshooting).

## Data type mapping

Expand Down