Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

sink(ticdc): asynchronously close the mq producers #5186

Merged
merged 15 commits into from
Apr 19, 2022
Merged

sink(ticdc): asynchronously close the mq producers #5186

merged 15 commits into from
Apr 19, 2022

Conversation

3AceShowHand
Copy link
Contributor

@3AceShowHand 3AceShowHand commented Apr 15, 2022 •
edited
Loading

What problem does this PR solve?

Issue Number: close #4241

What is changed and how it works?

as suggested by #4249 (comment), close the sink asynchronously.

close the producer (kafka / pulsar) in an asynchronous way.

Check List

Tests

  • Manual test (add detailed scripts or steps below)

Master branch, close the sink Manager would block a period of time.
Screen Shot 2022-04-18 at 1 58 46 PM

This PR, sink Manager closed immediately, does not block the processor's progress.
Screen Shot 2022-04-18 at 2 11 50 PM

Code changes

Side effects

Related changes

Release note 

None

@ti-chi-bot
Copy link
Member

ti-chi-bot commented Apr 15, 2022 •
edited
Loading

[REVIEW NOTIFICATION]

This pull request has been approved by:

  • hi-rustin
  • sdojjy

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

@ti-chi-bot ti-chi-bot added do-not-merge/needs-linked-issue release-note Denotes a PR that will be considered when it comes time to generate release notes. size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Apr 15, 2022
@3AceShowHand 3AceShowHand mentioned this pull request Apr 15, 2022
Closed
@ti-chi-bot ti-chi-bot added needs-cherry-pick-release-5.4 Should cherry pick this PR to release-5.4 branch. release-note-none Denotes a PR that doesn't merit a release note. needs-cherry-pick-release-6.0 Should cherry pick this PR to release-6.0 branch. and removed do-not-merge/needs-linked-issue release-note Denotes a PR that will be considered when it comes time to generate release notes. labels Apr 18, 2022
@3AceShowHand 3AceShowHand changed the title [DNM] sink(cdc): asynchronously close the mq producers sink(cdc): asynchronously close the mq producers Apr 18, 2022
@3AceShowHand
Copy link
Contributor Author

/run-kafka-integration-test

@3AceShowHand 3AceShowHand requested review from Rustin170506 and overvenus and removed request for Rustin170506 April 18, 2022 06:22
@Rustin170506 Rustin170506 changed the title sink(cdc): asynchronously close the mq producers sink(ticdc): asynchronously close the mq producers Apr 18, 2022
Rustin170506
Rustin170506 reviewed Apr 18, 2022
View reviewed changes
Copy link
Member

@Rustin170506 Rustin170506 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we do some correctness tests? For example, the correctness of shutting down the producer asynchronously after encountering a kafka error and resuming it again?

cdc/processor/pipeline/table_actor.go Outdated Show resolved Hide resolved
@3AceShowHand
Copy link
Contributor Author

/run-kafka-integration-test

@3AceShowHand
Copy link
Contributor Author

Can we do some correctness tests? For example, the correctness of shutting down the producer asynchronously after encountering a kafka error and resuming it again?

It looks already covered by the integration test kafka_sink_error_resume ?

@3AceShowHand
Copy link
Contributor Author

/run-all-tests

@ti-chi-bot ti-chi-bot added the status/LGT2 Indicates that a PR has LGTM 2. label Apr 19, 2022
@3AceShowHand
Copy link
Contributor Author

/merge

@ti-chi-bot
Copy link
Member

This pull request has been accepted and is ready to merge.

Commit hash: 606639b

@ti-chi-bot ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label Apr 19, 2022
@ti-chi-bot ti-chi-bot merged commit f80a34f into pingcap:master Apr 19, 2022
@ti-chi-bot ti-chi-bot mentioned this pull request Apr 19, 2022
Closed
@ti-chi-bot
Copy link
Member

In response to a cherrypick label: new pull request created: #5212.

ti-chi-bot pushed a commit to ti-chi-bot/tiflow that referenced this pull request Apr 19, 2022
@3AceShowHand @ti-chi-bot
This is an automated cherry-pick of pingcap#5186
e3444fa 
Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>
@ti-chi-bot ti-chi-bot mentioned this pull request Apr 19, 2022
Closed
@ti-chi-bot
Copy link
Member

In response to a cherrypick label: new pull request created: #5213.

ti-chi-bot pushed a commit to ti-chi-bot/tiflow that referenced this pull request Apr 19, 2022
@3AceShowHand @ti-chi-bot
This is an automated cherry-pick of pingcap#5186
b2d35d4 
Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>
@ti-chi-bot ti-chi-bot mentioned this pull request Apr 19, 2022
Closed
@ti-chi-bot
Copy link
Member

In response to a cherrypick label: new pull request created: #5214.

ti-chi-bot pushed a commit to ti-chi-bot/tiflow that referenced this pull request Apr 19, 2022
@3AceShowHand @ti-chi-bot
This is an automated cherry-pick of pingcap#5186
bb4b9b7 
Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>
@ti-chi-bot ti-chi-bot mentioned this pull request Apr 19, 2022
Closed
@ti-chi-bot
Copy link
Member

In response to a cherrypick label: new pull request created: #5215.

ti-chi-bot pushed a commit to ti-chi-bot/tiflow that referenced this pull request Apr 19, 2022
@3AceShowHand @ti-chi-bot
This is an automated cherry-pick of pingcap#5186
32c2b78 
Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>
@ti-chi-bot ti-chi-bot mentioned this pull request Apr 19, 2022
Closed
@ti-chi-bot
Copy link
Member

In response to a cherrypick label: new pull request created: #5216.

ti-chi-bot pushed a commit to ti-chi-bot/tiflow that referenced this pull request Apr 19, 2022
@3AceShowHand @ti-chi-bot
This is an automated cherry-pick of pingcap#5186
fb7d5b2 
Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>
@ti-chi-bot ti-chi-bot mentioned this pull request Apr 19, 2022
Merged
@ti-chi-bot
Copy link
Member

In response to a cherrypick label: new pull request created: #5217.

ti-chi-bot pushed a commit to ti-chi-bot/tiflow that referenced this pull request Apr 19, 2022
@3AceShowHand @ti-chi-bot
This is an automated cherry-pick of pingcap#5186
e59ae27 
Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>
@ti-chi-bot ti-chi-bot mentioned this pull request Apr 19, 2022
Merged
@ti-chi-bot
Copy link
Member

In response to a cherrypick label: new pull request created: #5218.

@3AceShowHand 3AceShowHand mentioned this pull request Apr 24, 2022
Closed
ti-chi-bot added a commit that referenced this pull request Apr 28, 2022
@ti-chi-bot
sink(ticdc): asynchronously close the mq producers (#5186) (#5218)
650557d 
close #4241
@nongfushanquan nongfushanquan removed the needs-cherry-pick-release-6.0 Should cherry pick this PR to release-6.0 branch. label May 9, 2022
ti-chi-bot added a commit that referenced this pull request Jun 17, 2022
@ti-chi-bot
sink(ticdc): asynchronously close the mq producers (#5186) (#5217)
352a8e1 
close #4241
@asddongmen asddongmen mentioned this pull request Sep 26, 2022
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@asddongmen asddongmen asddongmen left review comments

@Rustin170506 Rustin170506 Rustin170506 approved these changes

@sdojjy sdojjy sdojjy approved these changes

@overvenus overvenus Awaiting requested review from overvenus

Assignees
No one assigned
Labels
needs-cherry-pick-release-4.0 Should cherry pick this PR to release-4.0 branch. needs-cherry-pick-release-5.0 Should cherry pick this PR to release-5.0 branch. needs-cherry-pick-release-5.1 Should cherry pick this PR to release-5.1 branch. needs-cherry-pick-release-5.2 Should cherry pick this PR to release-5.2 branch. needs-cherry-pick-release-5.3 Should cherry pick this PR to release-5.3 branch. needs-cherry-pick-release-5.4 Should cherry pick this PR to release-5.4 branch. release-note-none Denotes a PR that doesn't merit a release note. size/S Denotes a PR that changes 10-29 lines, ignoring generated files. status/can-merge Indicates a PR has been approved by a committer. status/LGT2 Indicates that a PR has LGTM 2.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

mysql changefeed blocked by abnormal kafka changefeed
6 participants
@3AceShowHand @ti-chi-bot @Rustin170506 @sdojjy @asddongmen @nongfushanquan