You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
In postgres CDC, Debezium commits LSN on the source database to allow postgres to clean up WAL logs. In our current implementation, we continue to commit the LSN as we process records from WAL logs. This is nice since it makes sure that WAL logs are not building up in the source database and causing disk issues. But in case we fail to load the processed records successfully in destination, since we have already committed the LSN on source database, those WAL records are lost forever.
This epic is to change the above behaviour so that we only commit when the data has been successfully loaded in the warehouse.
The text was updated successfully, but these errors were encountered:
In postgres CDC, Debezium commits LSN on the source database to allow postgres to clean up WAL logs. In our current implementation, we continue to commit the LSN as we process records from WAL logs. This is nice since it makes sure that WAL logs are not building up in the source database and causing disk issues. But in case we fail to load the processed records successfully in destination, since we have already committed the LSN on source database, those WAL records are lost forever.
This epic is to change the above behaviour so that we only commit when the data has been successfully loaded in the warehouse.
The text was updated successfully, but these errors were encountered: