Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Epic : Postgres CDC LSN commit Behaviour change #19373

Closed
subodh1810 opened this issue Nov 14, 2022 · 0 comments
Closed

Epic : Postgres CDC LSN commit Behaviour change #19373

subodh1810 opened this issue Nov 14, 2022 · 0 comments
Assignees
Labels
Epic team/db-dw-sources Backlog for Database and Data Warehouse Sources team

Comments

@subodh1810
Copy link
Contributor

In postgres CDC, Debezium commits LSN on the source database to allow postgres to clean up WAL logs. In our current implementation, we continue to commit the LSN as we process records from WAL logs. This is nice since it makes sure that WAL logs are not building up in the source database and causing disk issues. But in case we fail to load the processed records successfully in destination, since we have already committed the LSN on source database, those WAL records are lost forever.

This epic is to change the above behaviour so that we only commit when the data has been successfully loaded in the warehouse.

@subodh1810 subodh1810 added Epic team/db-dw-sources Backlog for Database and Data Warehouse Sources team labels Nov 14, 2022
@subodh1810 subodh1810 self-assigned this Dec 7, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Epic team/db-dw-sources Backlog for Database and Data Warehouse Sources team
Projects
None yet
Development

No branches or pull requests

1 participant