Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CDC implementation for Citus using Logical Replication #6623

Merged
merged 57 commits into from
Mar 28, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
57 commits
Select commit Hold shift + click to select a range
2aa5877
CDC changes with DonotReplicateId
rajeshkt78 Jan 20, 2023
a9b2a59
fixed style.
rajeshkt78 Jan 20, 2023
23c7ede
Merge branch 'main' into cdc_with_logical_replication_new
rajeshkt78 Jan 20, 2023
2831d7b
Added CDC basic test case for distributed tables event publication.
rajeshkt78 Jan 27, 2023
ee23f30
Fixed style.
rajeshkt78 Jan 27, 2023
85cde60
Fixed the test cases to avoid output non-producibile data like LSN.
rajeshkt78 Jan 27, 2023
4ff6f32
Added check-cdc test schedule to config.yml
rajeshkt78 Jan 27, 2023
88aa202
Added test target for check-cdc in config.yml
rajeshkt78 Jan 27, 2023
2024e92
added consts for const values and improved the CDC basic tests
rajeshkt78 Jan 31, 2023
5df5820
fixed style
rajeshkt78 Jan 31, 2023
34e762f
Merge branch 'main' into cdc_with_logical_replication_new
rajeshkt78 Jan 31, 2023
558d64b
Merge branch 'main' into cdc_with_logical_replication_new
rajeshkt78 Jan 31, 2023
960cb6b
Added update and delete oprations to basic CDC test
rajeshkt78 Jan 31, 2023
7787e29
Fixed the CDC basic test cases and added cocurrent test case
rajeshkt78 Feb 2, 2023
54dcf88
Added TAP tests for CDC and handled schema change after move
rajeshkt78 Feb 16, 2023
efa19a1
Fixed a warning and moved the config options to lib
rajeshkt78 Feb 16, 2023
9e04cae
Fixed style for cdctestlib.pm
rajeshkt78 Feb 16, 2023
55a88e1
Resolved conflicts with main branch
rajeshkt78 Feb 17, 2023
9c668a3
Adds all branches trigger to trigger pipeline exec
gurkanindibay Feb 17, 2023
df81910
Rollbacks trigger code
gurkanindibay Feb 17, 2023
e1ee5d9
Fix regression in allowed foreign keys on distributed tables (#6550)
JelteF Jan 24, 2023
bca915d
Merge branch 'main' into cdc_with_logical_replication_new
rajeshkt78 Feb 17, 2023
eaf7a6c
Added check-cdc-custom-schedule
rajeshkt78 Feb 17, 2023
382369e
Merge branch 'main' into cdc_with_logical_replication_new
rajeshkt78 Feb 17, 2023
0536d9e
work around for cache lookup issue in CDC decoder and review comments…
rajeshkt78 Mar 15, 2023
02ab0b2
Removed a log message since some tests failed due to that.
rajeshkt78 Mar 15, 2023
3fa9753
Fixed changes in upgrade_list_citus_objects.out
rajeshkt78 Mar 15, 2023
10e9bac
Merge branch 'main' into cdc_with_logical_replication_new
rajeshkt78 Mar 15, 2023
0ce1812
Fixed chages in multi_extension.out
rajeshkt78 Mar 15, 2023
d5bc002
Fixed style.
rajeshkt78 Mar 15, 2023
a471b84
Fixed style for run_test.py
rajeshkt78 Mar 15, 2023
a823a8c
Fixed style and GUC alphabetical order.
rajeshkt78 Mar 15, 2023
4b1cfd4
Added downgrade script from 11.3.1 to 11.2.1
rajeshkt78 Mar 16, 2023
c5e3089
Update multi_extension.out for latest changes.
rajeshkt78 Mar 16, 2023
9f01399
Fixed upgrade_list_citus_objects
rajeshkt78 Mar 16, 2023
fb0e7fb
Updated upgrade_list_citus_objects.out
rajeshkt78 Mar 16, 2023
2a7cd35
Removed the CDC SQL tests and schedule and keep CDC TAP tests only.
rajeshkt78 Mar 16, 2023
61fdea5
Added a CDC block for running CDC tests.
rajeshkt78 Mar 16, 2023
2424620
Removed CDC changes from run_test.py
rajeshkt78 Mar 16, 2023
aa964b9
Added CDC TAP test in CircleCI
rajeshkt78 Mar 16, 2023
56a4126
Updated .circleci/config.yml
rajeshkt78 Mar 16, 2023
b20f923
Added run command to CDC Tap test.
rajeshkt78 Mar 16, 2023
3cd206d
Updated Run command for installing DBI package.
rajeshkt78 Mar 16, 2023
00f8578
Updated Run command to install DBI module in CircleCI
rajeshkt78 Mar 16, 2023
0f460d5
Added sudo for installing packages in CircleCI
rajeshkt78 Mar 16, 2023
0fd9e9b
Removed sudo command from CircleCI config
rajeshkt78 Mar 16, 2023
dace06e
Added apt-get update
rajeshkt78 Mar 16, 2023
35c6a99
Install libdbd-pg-perl for CDC in CircleCI config.yml
rajeshkt78 Mar 17, 2023
520e038
Updated the pg_regress_multi.pl to enable CDC for tests.
rajeshkt78 Mar 17, 2023
b1d7dcf
Refactored the schema change handling logic in CDC decoder and
rajeshkt78 Mar 20, 2023
e8fe49f
Merge branch 'main' into cdc_with_logical_replication_new
rajeshkt78 Mar 21, 2023
f2adbc6
Added test cases for update/delete after schema change and fixed bugs.
rajeshkt78 Mar 21, 2023
2976988
Incorporated review comments.
rajeshkt78 Mar 27, 2023
30525c1
Updated formatting.
rajeshkt78 Mar 27, 2023
4f2fede
moved CDC to seperate dir and test fixes.
rajeshkt78 Mar 28, 2023
55abb26
Merge branch 'main' into cdc_with_logical_replication_new
rajeshkt78 Mar 28, 2023
b666991
Merge branch 'main' into cdc_with_logical_replication_new
rajeshkt78 Mar 28, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions .circleci/config.yml
Original file line number Diff line number Diff line change
Expand Up @@ -455,6 +455,10 @@ jobs:
pg_major: << parameters.pg_major >>
- configure
- enable_core
- run:
name: 'Install DBI.pm'
command: |
apt-get update && apt-get install libdbi-perl && apt-get install libdbd-pg-perl
- run:
name: 'Run Test'
command: |
Expand Down Expand Up @@ -889,6 +893,10 @@ workflows:
<<: *tap-test-citus-15
name: 'test-15_tap-columnar-freezing'
suite: columnar_freezing
- tap-test-citus:
<<: *tap-test-citus-15
name: 'test-15_tap-cdc'
suite: cdc

- test-arbitrary-configs:
name: 'test-13_check-arbitrary-configs'
Expand Down
2 changes: 1 addition & 1 deletion src/backend/distributed/Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ generated_downgrade_sql_files += $(patsubst %,$(citus_abs_srcdir)/build/sql/%,$(
DATA_built = $(generated_sql_files)

# directories with source files
SUBDIRS = . commands connection ddl deparser executor metadata operations planner progress relay safeclib shardsplit test transaction utils worker clock
SUBDIRS = . commands cdc connection ddl deparser executor metadata operations planner progress relay safeclib shardsplit test transaction utils worker clock
# enterprise modules
SUBDIRS += replication

Expand Down
Loading