🐛 Source Postgres: fix CDC OOM issue #5304
Conversation
github-actions
bot
added
area/connectors
irynakruk
changed the title
fixed test
| "type": "string", | ||
| "description": "A pgoutput logical replication slot.", | ||
| "description": "A logical decoding plug-in installed on the PostgreSQL server. Please use `pgoutput` plug-in by default.\nFor more information when to use `wal2json` plug-in read <a href=\"https://docs.airbyte.io/integrations/sources/postgres\">Postgres Source</a> docs.", |
There was a problem hiding this comment.
@irynakruk pgoutput plug-in is used by default.
For the next sentence you can specify a key objections which are in that article so the description wil be selfcontained
There was a problem hiding this comment.
@yaroslav-hrytsaienko changed to:
A logical decoding plug-in installed on the PostgreSQL server.
pgoutputplug-in is used by default.\nIf replication table contains a lot of big jsonb values it is recommended to usewal2jsonplug-in. For more information aboutwal2jsonplug-in read <a href="https://docs.airbyte.io/integrations/sources/postgres\">Postgres Source docs.
|
/test connector=connectors/source-postgres
|
| @@ -155,6 +155,9 @@ protected Properties getDebeziumProperties() { | |||
| props.setProperty("offset.storage", "org.apache.kafka.connect.storage.FileOffsetBackingStore"); | |||
| props.setProperty("offset.storage.file.filename", offsetManager.getOffsetFilePath().toString()); | |||
| props.setProperty("offset.flush.interval.ms", "1000"); // todo: make this longer | |||
| // default values from debezium CommonConnectorConfig | |||
There was a problem hiding this comment.
Why do we need this change?
There was a problem hiding this comment.
The default queue size and batch size of the debezium for MySql and MS Sql is 8192 and 2048 accordingly, so we use queue size = 10000 on our side. However, the default values for Postgres is much higher (20240 and 10240) and when this properties are missing on our side, there is much more chances to get OOM. Since those values are default in CommonConnectorConfig, we can use it as a default on out side too.
|
/test connector=connectors/source-postgres
|
|
/publish connector=connectors/source-postgres
|
What
CDC OOM issue: table with lots of columns and few of them being JSON columns containing big json blobs cause an error.
How
During investigation it was found out that changing to
wal2jsonplugin helps to solve the issue.The solution described below:
wal2jsonduring source configuration.pgoutputstays as a default pluginRecommended reading order
PostgresSource.javaPostgresCdcProperties.javaPostgresUtils.javaspec.jsonpostgres.mdPre-merge Checklist
airbyte_secret./gradlew :airbyte-integrations:connectors:<name>:integrationTest.README.mddocs/integrations/<source or destination>/<name>.mdincluding changelog. See changelog example