tests(ticdc): add lossy_ddl test #8859

Rustin170506 · 2023-04-26T06:45:32Z

What problem does this PR solve?

Issue Number: close #8686

What is changed and how it works?

Add test for lossy DDLs.

Check List

Tests

Integration test

Questions

Will it cause performance regression or break compatibility?

No

Do you need to update user documentation, design documentation or monitoring documentation?

No

Release note

None

ti-chi-bot · 2023-04-26T06:45:34Z

[REVIEW NOTIFICATION]

This pull request has been approved by:

asddongmen
nongfushanquan

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

ti-chi-bot · 2023-04-26T06:45:35Z

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

Rustin170506 · 2023-04-26T06:49:05Z

tests/integration_tests/run_group.sh

@@ -40,7 +40,7 @@ groups=(
 	["G22"]='big_txn csv_storage_basic changefeed_finish sink_hang canal_json_storage_basic'
 	["G23"]='multi_topics new_ci_collation_with_old_value batch_add_table multi_changefeed'
 	["G24"]='consistent_replicate_nfs consistent_replicate_ddl owner_resign api_v2'
-	["G25"]='canal_json_storage_partition_table csv_storage_partition_table csv_storage_multi_tables_ddl'
+	["G25"]='canal_json_storage_partition_table csv_storage_partition_table csv_storage_multi_tables_ddl lossy_ddl'


Only for temporarily testing it. Add a new group before merging it.

Rustin170506 · 2023-04-26T07:02:56Z

/test all

Rustin170506 · 2023-04-26T08:55:10Z

Blocked by #8685

Rustin170506 · 2023-04-26T08:55:47Z

Tested locally:

docker-compose-mysql_integration_test-1  | run task successfully
docker-compose-mysql_integration_test-1  | wait process cdc.test exit for 1-th time...
docker-compose-mysql_integration_test-1  | wait process cdc.test exit for 2-th time...
docker-compose-mysql_integration_test-1  | wait process cdc.test exit for 3-th time...
docker-compose-mysql_integration_test-1  | wait process cdc.test exit for 4-th time...
docker-compose-mysql_integration_test-1  | cdc.test: no process found
docker-compose-mysql_integration_test-1  | wait process cdc.test exit for 5-th time...
docker-compose-mysql_integration_test-1  | process cdc.test already exit
docker-compose-mysql_integration_test-1  | [Wed Apr 26 08:53:10 UTC 2023] <<<<<< run test case lossy_ddl success! >>>>>>
docker-compose-mysql_integration_test-1  | \033[0;36m<<< Run all test success >>>\033[0m

Rustin170506 · 2023-04-27T00:35:50Z

/test all

tests/integration_tests/lossy_ddl/run.sh

CharlesCheung96 · 2023-04-27T04:03:16Z

tests/integration_tests/lossy_ddl/run.sh

+function run() {
+	# Use blackhole sink to check if the DDL is lossy.
+	# So no need to run this test for other sinks.
+	if [ "$SINK_TYPE" != "mysql" ]; then


Better to run this in kafka or cloudstorage pipeline since mysql pipeline is currently under heavy load.

Cool! I will change it.

tests/integration_tests/lossy_ddl/run.sh

Rustin170506 · 2023-04-27T06:58:16Z

/test all

ti-chi-bot · 2023-04-27T08:04:37Z

This pull request has been accepted and is ready to merge.

Commit hash: 1fad4bf

Rustin170506 · 2023-04-27T10:24:31Z

/test all

Rustin170506 · 2023-04-27T12:36:34Z

/merge

ti-chi-bot · 2023-04-27T12:36:37Z

This pull request has been accepted and is ready to merge.

Commit hash: fcffd00

Rustin170506 · 2023-04-27T12:36:42Z

/test all

ti-chi-bot · 2023-04-27T12:36:50Z

@hi-rustin: Your PR was out of date, I have automatically updated it for you.

At the same time I will also trigger all tests for you:

/run-all-tests

trigger some heavy tests which will not run always when PR updated.

If the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

Rustin170506 · 2023-04-27T12:39:33Z

/test all

ti-chi-bot · 2023-05-17T07:31:35Z

In response to a cherrypick label: new pull request created to branch release-6.6: #8984.

ti-chi-bot · 2023-05-17T07:31:35Z

In response to a cherrypick label: new pull request created to branch release-7.1: #8985.

Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>

ti-chi-bot · 2023-05-17T07:32:56Z

In response to a cherrypick label: new pull request created to branch release-6.5: #8986.

Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>

close #8686

close pingcap#8686 Signed-off-by: hi-rustin <rustin.liu@gmail.com>

close #8686

ti-chi-bot bot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. release-note-none Denotes a PR that doesn't merit a release note. size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Apr 26, 2023

Rustin170506 commented Apr 26, 2023

View reviewed changes

ti-chi-bot bot removed the size/M Denotes a PR that changes 30-99 lines, ignoring generated files. label Apr 26, 2023

Rustin170506 marked this pull request as ready for review April 26, 2023 08:54

ti-chi-bot bot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. labels Apr 26, 2023

Rustin170506 requested review from CharlesCheung96 and nongfushanquan April 26, 2023 08:54

CharlesCheung96 reviewed Apr 27, 2023

View reviewed changes

tests/integration_tests/lossy_ddl/run.sh Outdated Show resolved Hide resolved

CharlesCheung96 reviewed Apr 27, 2023

View reviewed changes

tests/integration_tests/lossy_ddl/run.sh Show resolved Hide resolved

ti-chi-bot bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Apr 27, 2023

Rustin170506 added 4 commits April 27, 2023 14:50

tests(ticdc): add lossy_ddl test

9279c97

tests(ticdc): add more tests

bf847d4

tests(ticdc): fmt

a294980

tests(ticdc): address comments

984351b

Rustin170506 force-pushed the rustin-patch-lossy-ddl branch from f400b12 to 984351b Compare April 27, 2023 06:55

ti-chi-bot bot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Apr 27, 2023

tests(ticdc): address comments

1fad4bf

Rustin170506 requested a review from CharlesCheung96 April 27, 2023 06:58

nongfushanquan approved these changes Apr 27, 2023

View reviewed changes

ti-chi-bot bot added status/can-merge Indicates a PR has been approved by a committer. needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. labels Apr 27, 2023

pkg/sink(ticdc): resolve conflicts

fcffd00

ti-chi-bot bot removed status/can-merge Indicates a PR has been approved by a committer. needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. labels Apr 27, 2023

ti-chi-bot bot added the status/can-merge Indicates a PR has been approved by a committer. label Apr 27, 2023

Merge branch 'master' into rustin-patch-lossy-ddl

104c839

ti-chi-bot bot merged commit c5add2f into pingcap:master Apr 27, 2023

Rustin170506 added needs-cherry-pick-release-6.6 Should cherry pick this PR to release-6.6 branch. needs-cherry-pick-release-7.1 Should cherry pick this PR to release-7.1 branch. labels May 17, 2023

This was referenced May 17, 2023

tests(ticdc): add lossy_ddl test (#8859) #8984

Closed

tests(ticdc): add lossy_ddl test (#8859) #8985

Merged

ti-chi-bot pushed a commit to ti-chi-bot/tiflow that referenced this pull request May 17, 2023

This is an automated cherry-pick of pingcap#8859

a4d71be

Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>

Rustin170506 added needs-cherry-pick-release-6.5 Should cherry pick this PR to release-6.5 branch. and removed needs-cherry-pick-release-6.6 Should cherry pick this PR to release-6.6 branch. labels May 17, 2023

ti-chi-bot mentioned this pull request May 17, 2023

tests(ticdc): add lossy_ddl test (#8859) #8986

Merged

ti-chi-bot pushed a commit to ti-chi-bot/tiflow that referenced this pull request May 17, 2023

This is an automated cherry-pick of pingcap#8859

e4b517a

Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>

ti-chi-bot bot pushed a commit that referenced this pull request May 19, 2023

tests(ticdc): add lossy_ddl test (#8859) (#8985)

29afa87

close #8686

Rustin170506 pushed a commit to ti-chi-bot/tiflow that referenced this pull request May 19, 2023

tests(ticdc): add lossy_ddl test (pingcap#8859) (pingcap#8985)

e3e921f

close pingcap#8686 Signed-off-by: hi-rustin <rustin.liu@gmail.com>

ti-chi-bot bot pushed a commit that referenced this pull request May 20, 2023

tests(ticdc): add lossy_ddl test (#8859) (#8986)

1881877

close #8686

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

tests(ticdc): add lossy_ddl test #8859

tests(ticdc): add lossy_ddl test #8859

Rustin170506 commented Apr 26, 2023

ti-chi-bot bot commented Apr 26, 2023 •

edited

Loading

ti-chi-bot bot commented Apr 26, 2023

Rustin170506 Apr 26, 2023

Rustin170506 commented Apr 26, 2023

Rustin170506 commented Apr 26, 2023

Rustin170506 commented Apr 26, 2023

Rustin170506 commented Apr 27, 2023

CharlesCheung96 Apr 27, 2023

Rustin170506 Apr 27, 2023

Rustin170506 commented Apr 27, 2023

ti-chi-bot bot commented Apr 27, 2023

Rustin170506 commented Apr 27, 2023

Rustin170506 commented Apr 27, 2023

ti-chi-bot bot commented Apr 27, 2023

Rustin170506 commented Apr 27, 2023

ti-chi-bot bot commented Apr 27, 2023

Rustin170506 commented Apr 27, 2023

ti-chi-bot commented May 17, 2023

ti-chi-bot commented May 17, 2023

ti-chi-bot commented May 17, 2023

tests(ticdc): add lossy_ddl test #8859

tests(ticdc): add lossy_ddl test #8859

Conversation

Rustin170506 commented Apr 26, 2023

What problem does this PR solve?

What is changed and how it works?

Check List

Tests

Questions

Will it cause performance regression or break compatibility?

Do you need to update user documentation, design documentation or monitoring documentation?

Release note

ti-chi-bot bot commented Apr 26, 2023 • edited Loading

ti-chi-bot bot commented Apr 26, 2023

Rustin170506 Apr 26, 2023

Choose a reason for hiding this comment

Rustin170506 commented Apr 26, 2023

Rustin170506 commented Apr 26, 2023

Rustin170506 commented Apr 26, 2023

Rustin170506 commented Apr 27, 2023

CharlesCheung96 Apr 27, 2023

Choose a reason for hiding this comment

Rustin170506 Apr 27, 2023

Choose a reason for hiding this comment

Rustin170506 commented Apr 27, 2023

ti-chi-bot bot commented Apr 27, 2023

Rustin170506 commented Apr 27, 2023

Rustin170506 commented Apr 27, 2023

ti-chi-bot bot commented Apr 27, 2023

Rustin170506 commented Apr 27, 2023

ti-chi-bot bot commented Apr 27, 2023

Rustin170506 commented Apr 27, 2023

ti-chi-bot commented May 17, 2023

ti-chi-bot commented May 17, 2023

ti-chi-bot commented May 17, 2023

ti-chi-bot bot commented Apr 26, 2023 •

edited

Loading