CDC implementation for Citus using Logical Replication #6623

rajeshkt78 · 2023-01-13T17:34:50Z

DESCRIPTION: Implementing CDC changes using Logical Replication to avoid re-publishing events multiple times by setting up replication origin session, which will add "DoNotReplicateId" to every WAL entry.

shard splits
shard moves
create distributed table
undistribute table
alter distributed tables (for some cases)
reference table operations

The citus decoder which will be decoding WAL events for CDC clients,
ignores any WAL entry with replication origin that is not zero.
It also maps the shard names to distributed table names.

src/backend/distributed/utils/replication_origin_session_utils.c

codecov · 2023-01-17T03:38:08Z

Codecov Report

Merging #6623 (4f2fede) into main (9bab819) will increase coverage by 0.01%.
The diff coverage is 92.13%.

❗ Current head 4f2fede differs from pull request most recent head 55abb26. Consider uploading reports for the commit 55abb26 to get more accurate results

@@            Coverage Diff             @@
##             main    #6623      +/-   ##
==========================================
+ Coverage   93.14%   93.16%   +0.01%     
==========================================
  Files         260      262       +2     
  Lines       56140    56327     +187     
==========================================
+ Hits        52293    52478     +185     
- Misses       3847     3849       +2

Previously trigger type was pull_request and events were opened, reopened and synchronize. In PR #6623, packaging pipeline was not triggered and PR was blocked . It may be a bug in GH Actions side. To make sure of execution of the pipeline, I changed pipeline execution type into 'branches'

marcocitus

test approach makes sense to me, for the basic test it would be good to also include update/delete

src/backend/distributed/commands/create_distributed_table.c

src/test/regress/expected/citus_cdc_basic_distributed_table.out

src/backend/distributed/utils/replication_origin_session_utils.c

src/backend/distributed/shardsplit/shardsplit_decoder.c

src/backend/distributed/utils/replication_origin_session_utils.c

src/backend/distributed/shared_library_init.c

src/backend/distributed/utils/replication_origin_session_utils.c

src/test/cdc/t/004_cdc_move_shard.pl

src/backend/distributed/shardsplit/cdc_decoder.c

src/backend/distributed/shardsplit/shardsplit_decoder.c

src/backend/distributed/shardsplit/cdc_decoder.c

src/backend/distributed/utils/replication_origin_session_utils.c

src/backend/distributed/transaction/transaction_management.c

incorporated comments.

marcocitus · 2023-03-21T14:43:49Z

I'm getting this error a lot when something fails and presumably the decoder restarts:

2023-03-21 15:38:38.521 CET [14916] ERROR:  could not receive data from WAL stream: ERROR:  could not attach to dynamic shared memory segment corresponding to handle:0
        CONTEXT:  slot "citus_cdc", output plugin "citus", in the change callback, associated LSN 2/74627410
2023-03-21 15:38:38.524 CET [19342] LOG:  background worker "logical replication worker" (PID 14916) exited with exit code 1

this is mainly related to splits, can we avoid it somehow?

src/backend/distributed/shardsplit/cdc_decoder.c

marcocitus · 2023-03-21T14:55:41Z

src/backend/distributed/shardsplit/cdc_decoder.c

+typedef struct
+{
+	Oid shardId;
+	Oid distributedTableId;


Even after combining with #6776 we're no longer seeing inserts after an alter_distributed_table, not sure why, but we can look at it later.

Maybe SetupReplicationOriginLocalSession need to go into ConvertTable itself so that we cover all different conversions?

alter_distribute_table changes the Oid of the distributed table, so the CDC client's mapping is no longer valid and hence any inserts/update/deletes after that, is not able to be applied by the CDC subscriber. The only current option is for the subscriber to drop the subscription and recreate the subscription again. This will be documented as a known issue.

src/backend/distributed/shardsplit/cdc_decoder.c

onderkalaci · 2023-03-22T07:11:52Z

Implemented replication origin session using DoNotReplicateId

I guess we can improve the title of the PR

src/test/cdc/t/002_cdc_create_distributed_table_concurrently.pl

rajeshkt78 · 2023-03-22T07:23:22Z

Implemented replication origin session using DoNotReplicateId

I guess we can improve the title of the PR

I have changed the Title now, please check if it is ok..

src/test/regress/expected/citus_cdc_basic_distributed_table_concurrently.out

src/test/cdc/t/cdctestlib.pm

src/backend/distributed/shardsplit/shardsplit_decoder.c

marcocitus

This looks pretty good to me. Huge leap forward in terms of setting up CDC for Citus!

I think we can do the "unbundling" of the decoder plugin as a separate PR.

src/backend/distributed/shardsplit/cdc_decoder.c

marcocitus · 2023-03-27T13:46:13Z

src/backend/distributed/shardsplit/shardsplit_decoder.c

-																  "_PG_output_plugin_init",
-																  false, NULL);
+		(LogicalOutputPluginInit) (void *)
+		load_external_function("pgoutput",


maybe make it a compile-time constant for now (ideally one that can be overridden via -D option)

Added 3 library dependencies for CDC TAP tests: libdbi-perl libdbd-pg-perl postgresql-${PG_MAJOR}-wal2json This is needed for dependencies added for these PRs: citusdata/citus#6827 citusdata/citus#6623

rajeshkt78 requested review from marcocitus and onderkalaci January 13, 2023 17:34

rajeshkt78 self-assigned this Jan 13, 2023

marcocitus reviewed Jan 15, 2023

View reviewed changes

src/backend/distributed/utils/replication_origin_session_utils.c Outdated Show resolved Hide resolved

marcocitus reviewed Jan 15, 2023

View reviewed changes

src/backend/distributed/utils/replication_origin_session_utils.c Outdated Show resolved Hide resolved

marcocitus reviewed Jan 16, 2023

View reviewed changes

src/backend/distributed/utils/replication_origin_session_utils.c Outdated Show resolved Hide resolved

marcocitus reviewed Jan 16, 2023

View reviewed changes

src/backend/distributed/utils/replication_origin_session_utils.c Outdated Show resolved Hide resolved

gurkanindibay mentioned this pull request Jan 19, 2023

Changes trigger type from 'pr' into 'branch' #6630

Closed

rajeshkt78 added 2 commits January 20, 2023 16:40

CDC changes with DonotReplicateId

2aa5877

fixed style.

a9b2a59

rajeshkt78 force-pushed the cdc_with_logical_replication_new branch from 70a88d8 to a9b2a59 Compare January 20, 2023 14:19

Merge branch 'main' into cdc_with_logical_replication_new

23c7ede

rajeshkt78 mentioned this pull request Jan 20, 2023

CDC changes with DonotReplicateId #6635

Closed

rajeshkt78 added 3 commits January 27, 2023 13:53

Added CDC basic test case for distributed tables event publication.

2831d7b

Fixed style.

ee23f30

Fixed the test cases to avoid output non-producibile data like LSN.

85cde60

marcocitus reviewed Jan 27, 2023

View reviewed changes

src/backend/distributed/commands/create_distributed_table.c Outdated Show resolved Hide resolved

src/test/regress/expected/citus_cdc_basic_distributed_table.out Outdated Show resolved Hide resolved

src/test/regress/expected/citus_cdc_basic_distributed_table.out Outdated Show resolved Hide resolved

rajeshkt78 and others added 8 commits January 27, 2023 19:44

Added check-cdc test schedule to config.yml

4ff6f32

Added test target for check-cdc in config.yml

88aa202

added consts for const values and improved the CDC basic tests

2024e92

fixed style

5df5820

Merge branch 'main' into cdc_with_logical_replication_new

34e762f

Merge branch 'main' into cdc_with_logical_replication_new

558d64b

Added update and delete oprations to basic CDC test

960cb6b

Fixed the CDC basic test cases and added cocurrent test case

7787e29

marcocitus reviewed Feb 3, 2023

View reviewed changes

src/backend/distributed/utils/replication_origin_session_utils.c Outdated Show resolved Hide resolved

marcocitus reviewed Feb 3, 2023

View reviewed changes

src/backend/distributed/utils/replication_origin_session_utils.c Outdated Show resolved Hide resolved

marcocitus reviewed Feb 3, 2023

View reviewed changes

src/backend/distributed/utils/replication_origin_session_utils.c Outdated Show resolved Hide resolved

marcocitus reviewed Mar 17, 2023

View reviewed changes

aykut-bozkurt reviewed Mar 18, 2023

View reviewed changes

rajeshkt78 and others added 3 commits March 20, 2023 21:51

Refactored the schema change handling logic in CDC decoder and

b1d7dcf

incorporated comments.

Merge branch 'main' into cdc_with_logical_replication_new

e8fe49f

Added test cases for update/delete after schema change and fixed bugs.

f2adbc6

marcocitus reviewed Mar 21, 2023

View reviewed changes

src/backend/distributed/shardsplit/cdc_decoder.c Outdated Show resolved Hide resolved

marcocitus reviewed Mar 21, 2023

View reviewed changes

src/backend/distributed/shardsplit/cdc_decoder.c Outdated Show resolved Hide resolved

rajeshkt78 changed the title ~~Implemented replication origin session using DoNotReplicateId~~ CDC implementation for Citus using Logical Replication Mar 22, 2023

onderkalaci reviewed Mar 22, 2023

View reviewed changes

src/test/cdc/t/002_cdc_create_distributed_table_concurrently.pl Outdated Show resolved Hide resolved

onderkalaci reviewed Mar 22, 2023

View reviewed changes

src/test/regress/expected/citus_cdc_basic_distributed_table_concurrently.out Outdated Show resolved Hide resolved

onderkalaci reviewed Mar 22, 2023

View reviewed changes

src/test/cdc/t/cdctestlib.pm Outdated Show resolved Hide resolved

marcocitus reviewed Mar 22, 2023

View reviewed changes

src/backend/distributed/shardsplit/shardsplit_decoder.c Outdated Show resolved Hide resolved

rajeshkt78 added 2 commits March 27, 2023 17:30

Incorporated review comments.

2976988

Updated formatting.

30525c1

marcocitus approved these changes Mar 27, 2023

View reviewed changes

rajeshkt78 and others added 2 commits March 28, 2023 14:22

moved CDC to seperate dir and test fixes.

4f2fede

Merge branch 'main' into cdc_with_logical_replication_new

55abb26

rajeshkt78 enabled auto-merge (squash) March 28, 2023 10:11

Merge branch 'main' into cdc_with_logical_replication_new

b666991

rajeshkt78 merged commit 85b8a2c into main Mar 28, 2023

rajeshkt78 deleted the cdc_with_logical_replication_new branch March 28, 2023 10:30

onderkalaci mentioned this pull request Mar 31, 2023

Intial commit for CDC using Logical Replication #6453

Closed

3 tasks

rajeshkt78 mentioned this pull request Apr 6, 2023

Add dependencies for CDC TAP tests in Perl citusdata/the-process#121

Merged

rajeshkt78 mentioned this pull request Apr 15, 2023

Changelog entries for 11.3.0 #6856

Merged

rajeshkt78 mentioned this pull request Apr 25, 2023

Doc changes for logical replication of distributed tables citusdata/citus_docs#1085

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CDC implementation for Citus using Logical Replication #6623

CDC implementation for Citus using Logical Replication #6623

rajeshkt78 commented Jan 13, 2023 •

edited by onurctirtir

Loading

codecov bot commented Jan 17, 2023 •

edited

Loading

marcocitus left a comment

marcocitus commented Mar 21, 2023

marcocitus Mar 21, 2023 •

edited

Loading

onderkalaci Mar 22, 2023

rajeshkt78 Mar 28, 2023

onderkalaci commented Mar 22, 2023

rajeshkt78 commented Mar 22, 2023

marcocitus left a comment

marcocitus Mar 27, 2023

CDC implementation for Citus using Logical Replication #6623

CDC implementation for Citus using Logical Replication #6623

Conversation

rajeshkt78 commented Jan 13, 2023 • edited by onurctirtir Loading

codecov bot commented Jan 17, 2023 • edited Loading

Codecov Report

marcocitus left a comment

Choose a reason for hiding this comment

marcocitus commented Mar 21, 2023

marcocitus Mar 21, 2023 • edited Loading

Choose a reason for hiding this comment

onderkalaci Mar 22, 2023

Choose a reason for hiding this comment

rajeshkt78 Mar 28, 2023

Choose a reason for hiding this comment

onderkalaci commented Mar 22, 2023

rajeshkt78 commented Mar 22, 2023

marcocitus left a comment

Choose a reason for hiding this comment

marcocitus Mar 27, 2023

Choose a reason for hiding this comment

rajeshkt78 commented Jan 13, 2023 •

edited by onurctirtir

Loading

codecov bot commented Jan 17, 2023 •

edited

Loading

marcocitus Mar 21, 2023 •

edited

Loading