Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Source Postgres: Clarify the behaviour of CDC replication with two or more destination #19082

Closed
killthekitten opened this issue Nov 7, 2022 · 1 comment · Fixed by #27325
Labels
area/documentation Improvements or additions to documentation community team/documentation type/enhancement New feature or request

Comments

@killthekitten
Copy link
Contributor

Tell us about the documentation you'd like us to add or update.

Currently, it’s possible to connect more than one destination to a single CDC source (PostgreSQL), which might lead to an inconsistent replication behavior. There's no explicit documentation of what would happen in case two or more connections would run a replication from the same PostgreSQL replication slot, and the UI encourages reusing the same source multiple times.

My assumption is that both destinations would end up with an incomplete set of changes, and perhaps the replication process itself will fail. From PostgreSQL docs:

A logical replication slot knows nothing about the state of the receiver(s). It's even possible to have multiple different receivers using the same slot at different times; they'll just get the changes following on from when the last receiver stopped consuming them. Only one receiver may consume changes from a slot at any given time.

I’d say the desired behaviour would be for Airbyte to prevent setting up a second connection on a CDC source without removing the first one, but as discussed on Slack, I'd start with making the documentation clearer.

If applicable, add links to the relevant docs that should be updated

@killthekitten killthekitten added area/documentation Improvements or additions to documentation type/enhancement New feature or request labels Nov 7, 2022
@sajarin sajarin removed the autoteam label Nov 9, 2022
@subodh1810
Copy link
Contributor

@killthekitten your understanding is true. CDC source should not be used with multiple destinations cause connection would end up interfering with each other. Airbyte should prohibit usage of a postgres source in cdc mode with multiple destinations

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/documentation Improvements or additions to documentation community team/documentation type/enhancement New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants