From 5570ce5369acf9940c017169c394020407162c17 Mon Sep 17 00:00:00 2001 From: Alexandre Cuoci Date: Wed, 9 Aug 2023 20:41:09 -0400 Subject: [PATCH 1/3] fixes to postgres docs --- docs/integrations/sources/postgres.md | 41 +++++++++++---------------- 1 file changed, 16 insertions(+), 25 deletions(-) diff --git a/docs/integrations/sources/postgres.md b/docs/integrations/sources/postgres.md index f97a48afb7db..a8a4a9ff7150 100644 --- a/docs/integrations/sources/postgres.md +++ b/docs/integrations/sources/postgres.md @@ -1,6 +1,3 @@ -import Tabs from '@theme/Tabs'; -import TabItem from '@theme/TabItem'; - # Postgres Airbyte's certified Postgres connector offers the following features: @@ -10,7 +7,7 @@ Airbyte's certified Postgres connector offers the following features: The contents below include a 'Quick Start' guide, advanced setup steps, and reference information (data type mapping, and changelogs). See [here](./postgres/postgres-troubleshooting.md) to troubleshooting issues with the Postgres connector. -![Airbyte Postgres Connection](./postgres/assets/airbyte_postgres_source.png) +![Airbyte Postgres Connection](https://raw.githubusercontent.com/airbytehq/airbyte/c078e8ed6703020a584d9362efa5665fbe8db77f/docs/integrations/sources/postgres/assets/airbyte_postgres_source.png) ## Quick Start @@ -43,7 +40,7 @@ ALTER DEFAULT PRIVILEGES IN SCHEMA GRANT SELECT ON TABLES TO REPLICATION; #### Step 3: Enable logical replication on your Postgres database - - - To enable logical replication on bare metal, VMs (EC2/GCE/etc), or Docker, configure the following parameters in the postgresql.conf file for your Postgres database: +To enable logical replication on bare metal, VMs (EC2/GCE/etc), or Docker, configure the following parameters in the postgresql.conf file for your Postgres database: | Parameter | Description | Set value to | | --------------------- | ------------------------------------------------------------------------------ |------------------------------------------------------------------------------------------------------------------------------------| | wal_level | Type of coding used within the Postgres write-ahead log | `logical ` | | max_wal_senders | The maximum number of processes used for handling WAL changes | `min: 1` | | max_replication_slots | The maximum number of replication slots that are allowed to stream WAL changes | `1` (if Airbyte is the only service reading subscribing to WAL changes. More than 1 if other services are also reading from the WAL) | - - -

To enable logical replication on AWS Postgres RDS or Aurora:

-
    -
  • Go to the Configuration tab for your DB cluster.
    • -
  • Find your cluster parameter group. Either edit the parameters for this group or create a copy of this parameter group to edit. If you create a copy, change your cluster's parameter group before restarting.
    • -
  • Within the parameter group page, search for rds.logical_replication. Select this row and click Edit parameters. Set this value to 1.
    • -
  • Wait for a maintenance window to automatically restart the instance or restart it manually.
    • -
- - -

To enable logical replication on Azure Database for Postgres:

-

Change the replication mode of your Postgres DB on Azure to logical using the Replication menu of your PostgreSQL instance in the Azure Portal. Alternatively, use the Azure CLI to run the following command:

- 
az postgres server configuration set --resource-group group --server-name server --name azure.replication_support --value logical
-    az postgres server restart --resource-group group --name server
- - + +To enable logical replication on AWS Postgres RDS or Aurora: +* Go to the Configuration tab for your DB cluster. +* Find your cluster parameter group. Either edit the parameters for this group or create a copy of this parameter group to edit. If you create a copy, change your cluster's parameter group before restarting. +* Within the parameter group page, search for `rds.logical_replication`. Select this row and click Edit parameters. Set this value to 1. +* Wait for a maintenance window to automatically restart the instance or restart it manually. + +To enable logical replication on Azure Database for Postgres, change the replication mode of your Postgres DB on Azure to `logical` using the replication menu of your PostgreSQL instance in the Azure Portal. Alternatively, use the Azure CLI to run the following command: + +``` +az postgres server configuration set --resource-group group --server-name server --name azure.replication_support --value logical +az postgres server restart --resource-group group --name server +``` #### Step 4: Create a replication slot on your Postgres database From 87b095dc71cfce09d07b035555a7fc1fa8ebb58a Mon Sep 17 00:00:00 2001 From: Alexandre Cuoci Date: Wed, 9 Aug 2023 20:50:38 -0400 Subject: [PATCH 2/3] no relative links --- docs/integrations/sources/postgres.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/docs/integrations/sources/postgres.md b/docs/integrations/sources/postgres.md index a8a4a9ff7150..b72bf7374662 100644 --- a/docs/integrations/sources/postgres.md +++ b/docs/integrations/sources/postgres.md @@ -1,11 +1,11 @@ # Postgres Airbyte's certified Postgres connector offers the following features: -* Multiple methods of keeping your data fresh, including [Change Data Capture (CDC)](./../../understanding-airbyte/cdc.md) and replication using the [xmin system column](#xmin). +* Multiple methods of keeping your data fresh, including [Change Data Capture (CDC)](https://docs.airbyte.com/understanding-airbyte/cdc) and replication using the [xmin system column](#xmin). * All available [sync modes](https://docs.airbyte.com/cloud/core-concepts#connection-sync-modes), providing flexibility in how data is delivered to your destination. -* Reliable replication at any table size with [checkpointing](./../../understanding-airbyte/airbyte-protocol.md#state--checkpointing) and chunking of database reads. +* Reliable replication at any table size with [checkpointing](https://docs.airbyte.com/understanding-airbyte/airbyte-protocol/#state--checkpointing) and chunking of database reads. -The contents below include a 'Quick Start' guide, advanced setup steps, and reference information (data type mapping, and changelogs). See [here](./postgres/postgres-troubleshooting.md) to troubleshooting issues with the Postgres connector. +The contents below include a 'Quick Start' guide, advanced setup steps, and reference information (data type mapping, and changelogs). See [here](https://docs.airbyte.com/integrations/sources/postgres/postgres-troubleshooting) to troubleshooting issues with the Postgres connector. ![Airbyte Postgres Connection](https://raw.githubusercontent.com/airbytehq/airbyte/c078e8ed6703020a584d9362efa5665fbe8db77f/docs/integrations/sources/postgres/assets/airbyte_postgres_source.png) @@ -75,8 +75,8 @@ Now, click `Set up source` in the Airbyte UI. Airbyte will now test connecting t ### Setup using CDC Airbyte uses [logical replication](https://www.postgresql.org/docs/10/logical-replication.html) of the Postgres write-ahead log (WAL) to incrementally capture deletes using a replication plugin: -* See [here](./../../understanding-airbyte/cdc.md) to learn more on how Airbyte implements CDC. -* See [here](./postgres/postgres-troubleshooting.md#cdc-requirements) to learn more about Postgres CDC requirements and limitations. +* See [here](https://docs.airbyte.com/understanding-airbyte/cdc) to learn more on how Airbyte implements CDC. +* See [here](https://docs.airbyte.com/integrations/sources/postgres/postgres-troubleshooting#cdc-requirements) to learn more about Postgres CDC requirements and limitations. We recommend configuring your Postgres source with CDC when: - You need a record of deletions. @@ -248,7 +248,7 @@ The command produces the private key in PEM format and the public key remains in ## Limitations & Troubleshooting -To see connector limitations, or troubleshoot your Postgres connector, see more [in our Postgres troubleshooting guide](/integrations/sources/postgres/postgres-troubleshooting.md). +To see connector limitations, or troubleshoot your Postgres connector, see more [in our Postgres troubleshooting guide](https://docs.airbyte.com/integrations/sources/postgres/postgres-troubleshooting). ## Data type mapping From 94ee2851c895c5ebebcb7ff8b7c7e25e77f3f3e9 Mon Sep 17 00:00:00 2001 From: Alexandre Cuoci Date: Wed, 9 Aug 2023 20:52:09 -0400 Subject: [PATCH 3/3] Update docs/integrations/sources/postgres.md --- docs/integrations/sources/postgres.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/integrations/sources/postgres.md b/docs/integrations/sources/postgres.md index b72bf7374662..1af4e013d05e 100644 --- a/docs/integrations/sources/postgres.md +++ b/docs/integrations/sources/postgres.md @@ -7,7 +7,7 @@ Airbyte's certified Postgres connector offers the following features: The contents below include a 'Quick Start' guide, advanced setup steps, and reference information (data type mapping, and changelogs). See [here](https://docs.airbyte.com/integrations/sources/postgres/postgres-troubleshooting) to troubleshooting issues with the Postgres connector. -![Airbyte Postgres Connection](https://raw.githubusercontent.com/airbytehq/airbyte/c078e8ed6703020a584d9362efa5665fbe8db77f/docs/integrations/sources/postgres/assets/airbyte_postgres_source.png) +![Airbyte Postgres Connection](https://raw.githubusercontent.com/airbytehq/airbyte/c078e8ed6703020a584d9362efa5665fbe8db77f/docs/integrations/sources/postgres/assets/airbyte_postgres_source.png?raw=true) ## Quick Start