Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

*(ticdc): fix some major problems about pull-based-sink #8179

Merged
merged 24 commits into from
Feb 8, 2023
Merged

*(ticdc): fix some major problems about pull-based-sink #8179

merged 24 commits into from
Feb 8, 2023

Conversation

hicqu
Copy link
Contributor

@hicqu hicqu commented Feb 6, 2023 •
edited by 3AceShowHand
Loading

What problem does this PR solve?

Issue Number: close #8103
Issue Number: close #8157
Issue Number: close #8185
Issue Number: close #8174

What is changed and how it works?

  • record mounter internal memory usage into memory quota;
  • limit task upperbound with schema storage resolved timestamp;
  • Make memquota.MemoryQuota more lock-free.

Check List

Tests

  • Unit test
  • Integration test
  • Manual test (add detailed scripts or steps below)
  • No code

Questions

Will it cause performance regression or break compatibility?
Do you need to update user documentation, design documentation or monitoring documentation?

Release note 

None

@hicqu
add mount metrics
b15ebe3 
Signed-off-by: qupeng <qupeng@pingcap.com>
@hicqu
move memquota into a package
c6cd43a 
Signed-off-by: qupeng <qupeng@pingcap.com>
@ti-chi-bot
Copy link
Member

ti-chi-bot commented Feb 6, 2023 •
edited
Loading

[REVIEW NOTIFICATION]

This pull request has been approved by:

  • amyangfei
  • hi-rustin

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

@ti-chi-bot ti-chi-bot added do-not-merge/needs-triage-completed release-note-none Denotes a PR that doesn't merit a release note. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. needs-cherry-pick-release-6.5 Should cherry pick this PR to release-6.5 branch. and removed do-not-merge/needs-triage-completed labels Feb 6, 2023
@hicqu
fix tests
87c2a6b 
Signed-off-by: qupeng <qupeng@pingcap.com>
@hicqu
make fmt
88ddd67 
Signed-off-by: qupeng <qupeng@pingcap.com>
@hicqu
no lock in memquota
51e9533 
Signed-off-by: qupeng <qupeng@pingcap.com>
@ti-chi-bot ti-chi-bot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. and removed size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Feb 6, 2023
@hicqu
some little changes
3352622 
Signed-off-by: qupeng <qupeng@pingcap.com>
@hicqu hicqu added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Feb 6, 2023
@hicqu
revert mount task
7333321 
Signed-off-by: qupeng <qupeng@pingcap.com>
@hicqu
try to limit sink manager task range
4accb06 
Signed-off-by: qupeng <qupeng@pingcap.com>
@hicqu
seperate sink and redo mounter group
672887e 
Signed-off-by: qupeng <qupeng@pingcap.com>
@hicqu
limit task upperbound with schema storage ts
4a6e952 
Signed-off-by: qupeng <qupeng@pingcap.com>
@hicqu
Merge branch 'master' into mounter-memory-quota
0bfc420
@hicqu
some reverts and fix tests
4eab921 
Signed-off-by: qupeng <qupeng@pingcap.com>
@hicqu
Merge branch 'master' into mounter-memory-quota
049768f
@hicqu hicqu changed the title sinkv2(cdc): record mounter internal memory usage into memory quota sinkv2(cdc): fix some major problems about pull-based-sink Feb 7, 2023
@ti-chi-bot ti-chi-bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Feb 7, 2023
@hicqu
some fixes
6b45fde 
Signed-off-by: qupeng <qupeng@pingcap.com>
@hicqu
Copy link
Contributor Author

hicqu commented Feb 7, 2023

/run-all-tests

@hicqu
remove stale things
afe644e 
Signed-off-by: qupeng <qupeng@pingcap.com>
@hicqu
Copy link
Contributor Author

hicqu commented Feb 7, 2023

/run-all-tests

@hicqu
fix verify
197ac32 
Signed-off-by: qupeng <qupeng@pingcap.com>
amyangfei
amyangfei approved these changes Feb 7, 2023
View reviewed changes
Copy link
Contributor

@amyangfei amyangfei left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

rest lgtm

cdc/processor/sinkmanager/manager.go Show resolved Hide resolved
@ti-chi-bot ti-chi-bot added the status/LGT1 Indicates that a PR has LGTM 1. label Feb 7, 2023
@hicqu
address comments
d5d0395 
Signed-off-by: qupeng <qupeng@pingcap.com>
@hicqu
fix tests
a881ed9 
Signed-off-by: qupeng <qupeng@pingcap.com>
Rustin170506
Rustin170506 reviewed Feb 8, 2023
View reviewed changes
Copy link
Member

@Rustin170506 Rustin170506 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

rest LGTM

cdc/entry/mounter_group.go Outdated Show resolved Hide resolved
cdc/processor/memquota/mem_quota.go Outdated Show resolved Hide resolved
cdc/processor/memquota/mem_quota.go Outdated Show resolved Hide resolved
cdc/processor/memquota/mem_quota.go Outdated Show resolved Hide resolved
cdc/processor/sinkmanager/manager.go Show resolved Hide resolved
cdc/processor/sinkmanager/manager.go Show resolved Hide resolved
@Rustin170506
Copy link
Member

/run-all-tests

@Rustin170506 Rustin170506 changed the title sinkv2(cdc): fix some major problems about pull-based-sink *(ticdc): fix some major problems about pull-based-sink Feb 8, 2023
@ti-chi-bot ti-chi-bot added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels Feb 8, 2023
@hicqu
make verify
4490f2f 
Signed-off-by: qupeng <qupeng@pingcap.com>
@hicqu
Merge branch 'mounter-memory-quota' of github.com:hicqu/ticdc into mo…
294997d 
…unter-memory-quota
@hicqu
Copy link
Contributor Author

hicqu commented Feb 8, 2023

/merge

@ti-chi-bot
Copy link
Member

This pull request has been accepted and is ready to merge.

Commit hash: 294997d

@ti-chi-bot ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label Feb 8, 2023
@hicqu
Copy link
Contributor Author

hicqu commented Feb 8, 2023

/run-verify

@hicqu
Copy link
Contributor Author

hicqu commented Feb 8, 2023

/run-all-tests

1 similar comment
@hicqu
Copy link
Contributor Author

hicqu commented Feb 8, 2023

/run-all-tests

@hicqu
Copy link
Contributor Author

hicqu commented Feb 8, 2023

/run-verify

@ti-chi-bot ti-chi-bot merged commit ae12f82 into pingcap:master Feb 8, 2023
@ti-chi-bot ti-chi-bot mentioned this pull request Feb 8, 2023
Merged
@ti-chi-bot
Copy link
Member

In response to a cherrypick label: new pull request created to branch release-6.5: #8197.

ti-chi-bot pushed a commit to ti-chi-bot/tiflow that referenced this pull request Feb 8, 2023
@hicqu @ti-chi-bot
This is an automated cherry-pick of pingcap#8179
c05ad23 
Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>
@hicqu hicqu deleted the mounter-memory-quota branch February 8, 2023 08:36
ti-chi-bot added a commit that referenced this pull request Feb 8, 2023
@ti-chi-bot
*(ticdc): fix some major problems about pull-based-sink (#8179) (#8197)
b5419b6 
close #8103
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Rustin170506 Rustin170506 Rustin170506 approved these changes

@amyangfei amyangfei amyangfei approved these changes

Assignees
No one assigned
Labels
needs-cherry-pick-release-6.5 Should cherry pick this PR to release-6.5 branch. release-note-none Denotes a PR that doesn't merit a release note. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. status/can-merge Indicates a PR has been approved by a committer. status/LGT2 Indicates that a PR has LGTM 2.
Projects
None yet
Milestone
No milestone
4 participants
@hicqu @ti-chi-bot @Rustin170506 @amyangfei