Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

cdc: limit cdc event fetching speed to reduce RocksDB read load #15849

Merged
merged 2 commits into from
Nov 1, 2023
Merged

cdc: limit cdc event fetching speed to reduce RocksDB read load #15849

merged 2 commits into from
Nov 1, 2023

Conversation

hicqu
Copy link
Contributor

@hicqu hicqu commented Oct 26, 2023

What is changed and how it works?

Issue Number: Close #11390

What's Changed:

None

Related changes

  • PR to update pingcap/docs/pingcap/docs-cn:
  • Need to cherry-pick to the release branch

Check List

Tests

  • Unit test

Side effects

Release note 

None

@ti-chi-bot
Copy link
Contributor

ti-chi-bot bot commented Oct 26, 2023 •
edited
Loading

[REVIEW NOTIFICATION]

This pull request has been approved by:

  • pingyu
  • zhangjinpeng1987

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

@ti-chi-bot ti-chi-bot bot added release-note-none Denotes a PR that doesn't merit a release note. size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Oct 26, 2023
@hicqu hicqu requested a review from pingyu October 26, 2023 09:21
zhangjinpeng87
zhangjinpeng87 approved these changes Oct 30, 2023
View reviewed changes
Copy link
Member

@zhangjinpeng87 zhangjinpeng87 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

rest LGTM

@ti-chi-bot ti-chi-bot bot added the status/LGT1 Indicates that a PR has LGTM 1. label Oct 30, 2023
pingyu
pingyu reviewed Oct 31, 2023
View reviewed changes
Copy link
Contributor

@pingyu pingyu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rest LGTM~

components/cdc/src/initializer.rs Outdated Show resolved Hide resolved
@hicqu
init
469d5d0 
Signed-off-by: qupeng <qupeng@pingcap.com>
@ti-chi-bot ti-chi-bot bot added needs-cherry-pick-release-6.5 Should cherry pick this PR to release-6.5 branch. needs-cherry-pick-release-7.1 Should cherry pick this PR to release-7.1 branch. needs-cherry-pick-release-7.5 Should cherry pick this PR to release-7.5 branch. labels Nov 1, 2023
@hicqu hicqu mentioned this pull request Nov 1, 2023
Open
@hicqu
fix
541e512 
Signed-off-by: qupeng <qupeng@pingcap.com>
@ti-chi-bot ti-chi-bot bot added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels Nov 1, 2023
@hicqu
Copy link
Contributor Author

hicqu commented Nov 1, 2023

/test

@hicqu
Copy link
Contributor Author

hicqu commented Nov 1, 2023

/merge

@ti-chi-bot ti-chi-bot
Copy link
Contributor

ti-chi-bot bot commented Nov 1, 2023

@hicqu: It seems you want to merge this PR, I will help you trigger all the tests:

/run-all-tests

You only need to trigger /merge once, and if the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

If you have any questions about the PR merge process, please refer to pr process.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

@ti-chi-bot ti-chi-bot
Copy link
Contributor

ti-chi-bot bot commented Nov 1, 2023

This pull request has been accepted and is ready to merge.

Commit hash: 541e512

@ti-chi-bot ti-chi-bot bot added the status/can-merge Indicates a PR has been approved by a committer. label Nov 1, 2023
@ti-chi-bot ti-chi-bot bot merged commit b230093 into tikv:master Nov 1, 2023
6 of 7 checks passed
@ti-chi-bot ti-chi-bot bot added this to the Pool milestone Nov 1, 2023
@hicqu hicqu deleted the cdc-event-speed branch November 1, 2023 06:54
@ti-chi-bot
Copy link
Member

In response to a cherrypick label: new pull request created to branch release-7.5: #15893.

This was referenced Nov 1, 2023
Merged
Merged
@ti-chi-bot
Copy link
Member

In response to a cherrypick label: new pull request created to branch release-6.5: #15894.

@ti-chi-bot
Copy link
Member

In response to a cherrypick label: new pull request created to branch release-7.1: #15895.

@ti-chi-bot ti-chi-bot mentioned this pull request Nov 1, 2023
Merged
ti-chi-bot bot pushed a commit that referenced this pull request Nov 8, 2023
@ti-chi-bot @hicqu
cdc: limit cdc event fetching speed to reduce RocksDB read load (#15849…
dbfc918 
…) (#15893)

close #11390

None

Signed-off-by: qupeng <qupeng@pingcap.com>

Co-authored-by: qupeng <qupeng@pingcap.com>
hicqu added a commit to hicqu/tikv that referenced this pull request Nov 15, 2023
@hicqu
cdc: limit cdc event fetching speed to reduce RocksDB read load (tikv…
8929171 
…#15849)

close tikv#11390

None

Signed-off-by: qupeng <qupeng@pingcap.com>
@hicqu hicqu mentioned this pull request Nov 15, 2023
Merged
hicqu added a commit that referenced this pull request Nov 15, 2023
@hicqu @pingyu
cdc: limit incremental scan fetching speed (#15989)
5165943 
* cdc: limit cdc event fetching speed to reduce RocksDB read load (#15849)

close #11390

None

Signed-off-by: qupeng <qupeng@pingcap.com>

* cdc: incremental scans acquire snapshots before semaphores to avoid useless queue (#15865)

close #15866

cdc: incremental scans acquire snapshots before semaphores to avoid useless queue

Signed-off-by: qupeng <qupeng@pingcap.com>

* cdc: notify pending tasks if associated regions change (#15947)

close #15910

Signed-off-by: qupeng <qupeng@pingcap.com>
Signed-off-by: qupeng <onlyqupeng@gmail.com>

Co-authored-by: Ping Yu <shui.yu@126.com>
Signed-off-by: qupeng <qupeng@pingcap.com>

---------

Signed-off-by: qupeng <qupeng@pingcap.com>
Co-authored-by: Ping Yu <shui.yu@126.com>
ti-chi-bot bot pushed a commit that referenced this pull request Nov 16, 2023
@ti-chi-bot @hicqu
cdc: limit cdc event fetching speed to reduce RocksDB read load (#15849…
e72f425 
…) (#15894)

close #11390

None

Signed-off-by: qupeng <qupeng@pingcap.com>

Co-authored-by: qupeng <qupeng@pingcap.com>
ti-chi-bot bot pushed a commit that referenced this pull request Nov 16, 2023
@ti-chi-bot @hicqu
cdc: limit cdc event fetching speed to reduce RocksDB read load (#15849…
63a05b8 
…) (#15895)

close #11390

None

Signed-off-by: qupeng <qupeng@pingcap.com>

Co-authored-by: qupeng <qupeng@pingcap.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@zhangjinpeng87 zhangjinpeng87 zhangjinpeng87 approved these changes

@pingyu pingyu pingyu approved these changes

Assignees
No one assigned
Labels
needs-cherry-pick-release-6.5 Should cherry pick this PR to release-6.5 branch. needs-cherry-pick-release-7.1 Should cherry pick this PR to release-7.1 branch. needs-cherry-pick-release-7.5 Should cherry pick this PR to release-7.5 branch. release-note-none Denotes a PR that doesn't merit a release note. size/M Denotes a PR that changes 30-99 lines, ignoring generated files. status/can-merge Indicates a PR has been approved by a committer. status/LGT2 Indicates that a PR has LGTM 2.
Projects
None yet
Milestone
Pool
Development

Successfully merging this pull request may close these issues.

CDC should limit network bandwidth consumption
4 participants
@hicqu @ti-chi-bot @pingyu @zhangjinpeng87