-
Notifications
You must be signed in to change notification settings - Fork 288
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
CDC/etcd_worker: add rate limiter to limit EtcdWorker tick frequency (#3219) #3271
CDC/etcd_worker: add rate limiter to limit EtcdWorker tick frequency (#3219) #3271
Conversation
[REVIEW NOTIFICATION] This pull request has not been approved. To complete the pull request process, please ask the reviewers in the list to review by filling The full list of commands accepted by this bot can be found here. Reviewer can indicate their review by submitting an approval review. |
/merge |
This pull request has been accepted and is ready to merge. Commit hash: af8cbd9
|
Codecov Report
@@ Coverage Diff @@
## release-5.3 #3271 +/- ##
===================================================
- Coverage 56.7517% 56.5181% -0.2337%
===================================================
Files 211 211
Lines 22794 22798 +4
===================================================
- Hits 12936 12885 -51
- Misses 8529 8602 +73
+ Partials 1329 1311 -18 |
This is an automated cherry-pick of #3219
What problem does this PR solve?
#3112
Too frequent etcd worker tick will cause etcd to be overburdened, and the current etcd qps will increase exponentially with the number of tables that need to be replicated.
What is changed and how it works?
Add rate limiter to limit EtcdWorker ticks frequency.
Check List
Tests
Summary: Limiting the tick frequency of
EtcdWoker
can reduce etcd qps by about 50%, but it will reduce the replication speed by about 20%.Test Environment:
Test1 (EtcdWorker ticks limit 10 times / s)
Create 16 changefeeds, synchronize a table for each, and write 50w rows of data to each table in upstream.
Test2 (EtcdWorker ticks limit 10 times / s)
Create 30 changefeeds, synchronize a table for each, and write 50w rows of data to each table in upstream.
Test3 (EtcdWorker ticks without limit )
Create 16 changefeeds, synchronize a table for each, and write 50w rows of data to each table in upstream.
Test4 (EtcdWorker ticks without limit )
Create 30 changefeeds, synchronize a table for each, and write 50w rows of data to each table in upstream.
Test5 (Incremental scan、EtcdWorker ticks limit 10 times / s)
Test6 (Incremental scan、EtcdWorker ticks limit 10 times / s)
When resume changefeed, the owner was down instantly.
Side effects
Related changes
Release note