Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

CDC should limit network bandwidth consumption #11390

Closed
overvenus opened this issue Nov 16, 2021 · 0 comments · Fixed by #15849
Closed

CDC should limit network bandwidth consumption #11390

overvenus opened this issue Nov 16, 2021 · 0 comments · Fixed by #15849
Assignees
@hicqu
Labels
affects-6.5 affects-7.1 affects-7.5 component/CDC Component: Change Data Capture severity/minor type/enhancement The issue or PR belongs to an enhancement.

Comments

@overvenus
Copy link
Member

While #11152 corrects cdc incremental scan speed limit, TiKV may consume more network bandwidth than before. It could affect online service.

We should add a configuration that limit cdc network bandwidth consumption.

Before / Now
image
image
@overvenus overvenus added type/enhancement The issue or PR belongs to an enhancement. component/CDC Component: Change Data Capture labels Nov 16, 2021
hicqu added a commit to hicqu/tikv that referenced this issue Nov 26, 2021
@hicqu
close tikv#11390
bccef5c 
Signed-off-by: qupeng <qupeng@pingcap.com>
@hicqu hicqu mentioned this issue Nov 26, 2021
Closed
@hicqu hicqu mentioned this issue Oct 26, 2023
Merged
ti-chi-bot bot pushed a commit that referenced this issue Nov 1, 2023
@hicqu
cdc: limit cdc event fetching speed to reduce RocksDB read load (#15849)
b230093 
close #11390

None

Signed-off-by: qupeng <qupeng@pingcap.com>
This was referenced Nov 1, 2023
Merged
Merged
Merged
ti-chi-bot bot pushed a commit that referenced this issue Nov 8, 2023
@ti-chi-bot @hicqu
cdc: limit cdc event fetching speed to reduce RocksDB read load (#15849…
dbfc918 
…) (#15893)

close #11390

None

Signed-off-by: qupeng <qupeng@pingcap.com>

Co-authored-by: qupeng <qupeng@pingcap.com>
hicqu added a commit to hicqu/tikv that referenced this issue Nov 15, 2023
@hicqu
cdc: limit cdc event fetching speed to reduce RocksDB read load (tikv…
8929171 
…#15849)

close tikv#11390

None

Signed-off-by: qupeng <qupeng@pingcap.com>
hicqu added a commit that referenced this issue Nov 15, 2023
@hicqu @pingyu
cdc: limit incremental scan fetching speed (#15989)
5165943 
* cdc: limit cdc event fetching speed to reduce RocksDB read load (#15849)

close #11390

None

Signed-off-by: qupeng <qupeng@pingcap.com>

* cdc: incremental scans acquire snapshots before semaphores to avoid useless queue (#15865)

close #15866

cdc: incremental scans acquire snapshots before semaphores to avoid useless queue

Signed-off-by: qupeng <qupeng@pingcap.com>

* cdc: notify pending tasks if associated regions change (#15947)

close #15910

Signed-off-by: qupeng <qupeng@pingcap.com>
Signed-off-by: qupeng <onlyqupeng@gmail.com>

Co-authored-by: Ping Yu <shui.yu@126.com>
Signed-off-by: qupeng <qupeng@pingcap.com>

---------

Signed-off-by: qupeng <qupeng@pingcap.com>
Co-authored-by: Ping Yu <shui.yu@126.com>
ti-chi-bot bot pushed a commit that referenced this issue Nov 16, 2023
@ti-chi-bot @hicqu
cdc: limit cdc event fetching speed to reduce RocksDB read load (#15849…
e72f425 
…) (#15894)

close #11390

None

Signed-off-by: qupeng <qupeng@pingcap.com>

Co-authored-by: qupeng <qupeng@pingcap.com>
ti-chi-bot bot pushed a commit that referenced this issue Nov 16, 2023
@ti-chi-bot @hicqu
cdc: limit cdc event fetching speed to reduce RocksDB read load (#15849…
63a05b8 
…) (#15895)

close #11390

None

Signed-off-by: qupeng <qupeng@pingcap.com>

Co-authored-by: qupeng <qupeng@pingcap.com>
@nongfushanquan nongfushanquan mentioned this issue Nov 30, 2023
Merged
16 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@hicqu hicqu

Labels
affects-6.5 affects-7.1 affects-7.5 component/CDC Component: Change Data Capture severity/minor type/enhancement The issue or PR belongs to an enhancement.
Projects
None yet
Milestone
No milestone
2 participants
@overvenus @hicqu