Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

sink(cdc): handle sink errors more fast and light #8949

Merged
merged 23 commits into from
May 16, 2023
Merged

sink(cdc): handle sink errors more fast and light #8949

merged 23 commits into from
May 16, 2023

Conversation

hicqu
Copy link
Contributor

@hicqu hicqu commented May 11, 2023 •
edited
Loading

What problem does this PR solve?

Issue Number: ref #8657

What is changed and how it works?

Handle sink errors more fast and light, without rebuilding the whole changefeed.

Check List

Tests

  • Unit test
  • Manual test (add detailed scripts or steps below)

Questions

Will it cause performance regression or break compatibility?
Do you need to update user documentation, design documentation or monitoring documentation?

Release note 

None

hicqu added 3 commits May 11, 2023 14:21
@hicqu
init
f0424b2 
Signed-off-by: qupeng <qupeng@pingcap.com>
@hicqu
handle error in api client
c7dd8cd 
Signed-off-by: qupeng <qupeng@pingcap.com>
@hicqu
some updates
212b4fc 
Signed-off-by: qupeng <qupeng@pingcap.com>
@ti-chi-bot
Copy link
Contributor

ti-chi-bot bot commented May 11, 2023 •
edited
Loading

[REVIEW NOTIFICATION]

This pull request has been approved by:

  • hi-rustin
  • nongfushanquan

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

@ti-chi-bot ti-chi-bot bot added release-note-none Denotes a PR that doesn't merit a release note. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels May 11, 2023
@hicqu
fix tests
d6ddf8d 
Signed-off-by: qupeng <qupeng@pingcap.com>
@hicqu hicqu requested review from sdojjy and Rustin170506 May 11, 2023 10:03
@hicqu
make check
12c21d1 
Signed-off-by: qupeng <qupeng@pingcap.com>
@hicqu
Copy link
Contributor Author

hicqu commented May 12, 2023

/test verify

1 similar comment
@hicqu
Copy link
Contributor Author

hicqu commented May 12, 2023

/test verify

hicqu added 2 commits May 12, 2023 15:07
@hicqu
some changes
88598a4 
Signed-off-by: qupeng <qupeng@pingcap.com>
@hicqu
fix tests
e8f215c 
Signed-off-by: qupeng <qupeng@pingcap.com>
Rustin170506
Rustin170506 reviewed May 12, 2023
View reviewed changes
Copy link
Member

@Rustin170506 Rustin170506 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To ensure clarity and accuracy, kindly revise the title of your PR and provide comprehensive information in its description.

@hicqu
fix tests
2d8e70d 
Signed-off-by: qupeng <qupeng@pingcap.com>
@hicqu hicqu changed the title handle sink errors more fast and light sink(cdc): handle sink errors more fast and light May 12, 2023
hicqu added 4 commits May 12, 2023 16:53
@hicqu
address comments
1343073 
Signed-off-by: qupeng <qupeng@pingcap.com>
@hicqu
fix tests
80aa1e7 
Signed-off-by: qupeng <qupeng@pingcap.com>
@hicqu
fix tests
2bce733 
Signed-off-by: qupeng <qupeng@pingcap.com>
@hicqu
make fmt
fc0e86d 
Signed-off-by: qupeng <qupeng@pingcap.com>
Rustin170506
Rustin170506 approved these changes May 15, 2023
View reviewed changes
cdc/owner/changefeed.go
@@ -179,7 +183,10 @@ func newChangefeed4Test(
schemaStorage entry.SchemaStorage,
filter filter.Filter,
) (puller.DDLPuller, error),
newSink func(changefeedID model.ChangeFeedID, info *model.ChangeFeedInfo, reportErr func(err error)) DDLSink,
newSink func(
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe we can give it a type, then we don't need to copy and paste it a couple of times.

@ti-chi-bot ti-chi-bot bot added the status/LGT1 Indicates that a PR has LGTM 1. label May 15, 2023
@ti-chi-bot ti-chi-bot bot added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels May 15, 2023
@hicqu
Copy link
Contributor Author

hicqu commented May 15, 2023

/merge

@ti-chi-bot
Copy link
Contributor

ti-chi-bot bot commented May 15, 2023

This pull request has been accepted and is ready to merge.

Commit hash: fc0e86d

@ti-chi-bot ti-chi-bot bot added the status/can-merge Indicates a PR has been approved by a committer. label May 15, 2023
@ti-chi-bot
Copy link
Contributor

ti-chi-bot bot commented May 16, 2023

This pull request has been accepted and is ready to merge.

Commit hash: 5ac5fe0

@ti-chi-bot ti-chi-bot bot added the status/can-merge Indicates a PR has been approved by a committer. label May 16, 2023
@hicqu hicqu added needs-cherry-pick-release-6.5 Should cherry pick this PR to release-6.5 branch. and removed needs-cherry-pick-release-6.1 Should cherry pick this PR to release-6.1 branch. labels May 16, 2023
@hicqu
Copy link
Contributor Author

hicqu commented May 16, 2023

/test cdc-integration-mysql-test

@ti-chi-bot
Copy link
Contributor

ti-chi-bot bot commented May 16, 2023 •
edited
Loading

@hicqu: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
pull-cdc-integration-kafka-test 5ac5fe0 link unknown /test cdc-integration-kafka-test

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@hicqu
Copy link
Contributor Author

hicqu commented May 16, 2023

/test cdc-integration-kafka-test

@hicqu hicqu merged commit 6594355 into pingcap:master May 16, 2023
@hicqu hicqu deleted the sink-error-split-1 branch May 16, 2023 09:48
@ti-chi-bot ti-chi-bot mentioned this pull request May 16, 2023
Closed
@ti-chi-bot
Copy link
Member

In response to a cherrypick label: new pull request created to branch release-6.5: #8975.

ti-chi-bot pushed a commit to ti-chi-bot/tiflow that referenced this pull request May 16, 2023
@hicqu @ti-chi-bot
This is an automated cherry-pick of pingcap#8949
c4a1b42 
Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>
@ti-chi-bot ti-chi-bot mentioned this pull request May 16, 2023
Merged
@ti-chi-bot
Copy link
Member

In response to a cherrypick label: new pull request created to branch release-7.1: #8976.

ti-chi-bot bot pushed a commit that referenced this pull request May 17, 2023
@ti-chi-bot
sink(cdc): handle sink errors more fast and light (#8949) (#8976)
f2cd956 
ref #8657
ti-chi-bot bot pushed a commit that referenced this pull request May 18, 2023
@hicqu
(sink/cdc): fix stop table dead lock introduced in #8949 (#8989)
e5caa48 
ref #8657
ti-chi-bot bot pushed a commit that referenced this pull request May 19, 2023
@ti-chi-bot
(sink/cdc): fix stop table dead lock introduced in #8949 (#8989) (#8994)
2ea7904 
ref #8657
@hicqu hicqu mentioned this pull request May 21, 2023
Merged
ti-chi-bot bot pushed a commit that referenced this pull request May 22, 2023
@hicqu
(sink/cdc): fix some bugs introduced by #8949 (#9010)
fbb363a 
ref #8657
This was referenced May 22, 2023
Merged
Closed
ti-chi-bot bot pushed a commit that referenced this pull request May 23, 2023
@ti-chi-bot
(sink/cdc): fix some bugs introduced by #8949 (#9010) (#9015)
5bf7c8f 
ref #8657
@hicqu hicqu mentioned this pull request May 29, 2023
Merged
This was referenced May 29, 2023
Closed
Merged
hicqu added a commit to ti-chi-bot/tiflow that referenced this pull request May 30, 2023
@hicqu
This is an automated cherry-pick of pingcap#8949
5ac4c4e 
Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>
@hicqu hicqu removed the needs-cherry-pick-release-6.5 Should cherry pick this PR to release-6.5 branch. label May 30, 2023
hicqu added a commit to ti-chi-bot/tiflow that referenced this pull request May 30, 2023
@hicqu @ti-chi-bot
sink(cdc): handle sink errors more fast and light (pingcap#8949)
31f2792 
Signed-off-by: qupeng <qupeng@pingcap.com>
Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
hicqu added a commit to ti-chi-bot/tiflow that referenced this pull request May 30, 2023
@hicqu
(sink/cdc): fix stop table dead lock introduced in pingcap#8949 (ping…
3eba44a 
…cap#8989)

ref pingcap#8657

Signed-off-by: qupeng <qupeng@pingcap.com>
hicqu added a commit to ti-chi-bot/tiflow that referenced this pull request May 30, 2023
@hicqu
(sink/cdc): fix some bugs introduced by pingcap#8949 (pingcap#9010)
a8d643b 
ref pingcap#8657

Signed-off-by: qupeng <qupeng@pingcap.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@nongfushanquan nongfushanquan nongfushanquan approved these changes

@Rustin170506 Rustin170506 Rustin170506 approved these changes

@sdojjy sdojjy Awaiting requested review from sdojjy

Assignees
No one assigned
Labels
needs-cherry-pick-release-7.1 Should cherry pick this PR to release-7.1 branch. release-note-none Denotes a PR that doesn't merit a release note. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. status/can-merge Indicates a PR has been approved by a committer. status/LGT2 Indicates that a PR has LGTM 2.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@hicqu @nongfushanquan @asddongmen @ti-chi-bot @Rustin170506