Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

sink: add barrier in underlying sink to support accruate stop (#2417) #2494

Merged
merged 14 commits into from
Aug 12, 2021
Merged

sink: add barrier in underlying sink to support accruate stop (#2417) #2494

merged 14 commits into from
Aug 12, 2021

Conversation

ti-chi-bot
Copy link
Member

This is an automated cherry-pick of #2417

What problem does this PR solve?

Fix #2230

This PR introduces a new Barrier API in sink interface, which is a synchronous function to wait all events to be flushed in underlying sink, note once Barrier is called, the resolved ts can't be pushed until the Barrier call returns.

We call Barrier when table sink is closing (via the Close method of table sink), and before Barrier is called, we draw back all existing events in sink buffer.

Note the cherry-pick cannot be merged into release-4.0, release-5.0 until new owner is enabled in these branches.

What is changed and how it works?

  • The Barrier implementation
    • MySQL sink: wait checkpoint-ts to reach max received resolved-ts
    • MQ sink: TODO
  • Drawback mechanism in buffer sink
    • Buffer sink stores rows by table in memory, and emit rows to underlying sink only resolved ts is forwarded.
    • Hook a channel from sink manager to notify buffer sink to draw back rows of a table, and wait callback

Check List

Tests

  • Unit test
  • Integration test

Release note 

Fix a bug that multiple processors could write the same table when this table is re-scheduling

@ti-chi-bot
Copy link
Member Author

ti-chi-bot commented Aug 10, 2021 •
edited
Loading

[REVIEW NOTIFICATION]

This pull request has been approved by:

  • amyangfei

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

@ti-chi-bot ti-chi-bot added the release-note Denotes a PR that will be considered when it comes time to generate release notes. label Aug 10, 2021
@ti-chi-bot ti-chi-bot mentioned this pull request Aug 10, 2021
Merged
@ti-chi-bot ti-chi-bot added release-blocker This issue blocks a release. Please solve it ASAP. require-LGT1 Indicates that the PR requires an LGTM. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. status/LGT1 Indicates that a PR has LGTM 1. type/bugfix This PR fixes a bug. type/cherry-pick-for-release-5.2 This PR is cherry-picked to release-5.2 from a source PR. labels Aug 10, 2021
@amyangfei amyangfei added this to the v5.2.0 milestone Aug 11, 2021
@amyangfei amyangfei removed the require-LGT1 Indicates that the PR requires an LGTM. label Aug 11, 2021
@ti-chi-bot ti-chi-bot added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels Aug 11, 2021
@amyangfei
Copy link
Contributor

/merge

@ti-chi-bot
Copy link
Member Author

This pull request has been accepted and is ready to merge.

Commit hash: fc58f2c

@ti-chi-bot ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label Aug 11, 2021
@amyangfei
Copy link
Contributor

/run-all-tests

1 similar comment
@amyangfei
Copy link
Contributor

/run-all-tests

@nongfushanquan nongfushanquan added the cherry-pick-approved Cherry pick PR approved by release team. label Aug 12, 2021
@amyangfei
Copy link
Contributor

/merge

@ti-chi-bot ti-chi-bot merged commit f466d8c into pingcap:release-5.2 Aug 12, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@liuzix liuzix liuzix approved these changes

@amyangfei amyangfei amyangfei approved these changes

@zier-one zier-one Awaiting requested review from zier-one

Assignees

@amyangfei amyangfei

Labels
cherry-pick-approved Cherry pick PR approved by release team. release-blocker This issue blocks a release. Please solve it ASAP. release-note Denotes a PR that will be considered when it comes time to generate release notes. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. status/can-merge Indicates a PR has been approved by a committer. status/LGT2 Indicates that a PR has LGTM 2. type/bugfix This PR fixes a bug. type/cherry-pick-for-release-5.2 This PR is cherry-picked to release-5.2 from a source PR.
Projects
None yet
Milestone
v5.2.0
Development

Successfully merging this pull request may close these issues.
4 participants
@ti-chi-bot @amyangfei @liuzix @nongfushanquan