server: pause scheduling immediately #7809
Conversation
Signed-off-by: Ryan Leung <rleungx@gmail.com>
|
[REVIEW NOTIFICATION] This pull request has been approved by:
To complete the pull request process, please ask the reviewers in the list to review by filling The full list of commands accepted by this bot can be found here. Reviewer can indicate their review by submitting an approval review. |
ti-chi-bot
bot
added
the
release-note-none
ti-chi-bot
bot
added
the
size/XS
Codecov Report
Additional details and impacted files@@ Coverage Diff @@
## master #7809 +/- ##
==========================================
+ Coverage 73.45% 73.49% +0.04%
==========================================
Files 432 432
Lines 47843 47860 +17
==========================================
+ Hits 35142 35174 +32
+ Misses 9663 9641 -22
- Partials 3038 3045 +7
Flags with carried forward coverage won't be shown. Click here to find out more. |
| @@ -262,6 +263,10 @@ func forwardRegionHeartbeatToScheduling(forwardStream schedulingpb.Scheduling_Re | |||
| errCh <- errors.WithStack(err) | |||
| return | |||
| } | |||
| // TODO: find a better way to halt scheduling immediately. | |||
| if rc.GetOpts().IsSchedulingHalted() { | |||
| continue | |||
There was a problem hiding this comment.
One problem is that the operator generated by the scheduling server still exists in memory. Is it possible to cancel all the scheduling server operators after the online recovery is complete so that the scheduling can be resumed as soon as possible?
There was a problem hiding this comment.
It depends. IMO, if those operators still take effect, cancelling all is not the best way. But if the region is changed, we may need to cancel. So I prefer to leave it for now.
ti-chi-bot
bot
added
status/LGT2
|
/merge |
|
@rleungx: It seems you want to merge this PR, I will help you trigger all the tests: /run-all-tests You only need to trigger
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository. |
|
This pull request has been accepted and is ready to merge. Commit hash: 0dd5c4c
|
ti-chi-bot
bot
added
the
status/can-merge
What problem does this PR solve?
Issue Number: Ref #5839.
What is changed and how does it work?
We use a watch mechanism to sync the configuration to the scheduling server, so there might be a delay. In the online recovery scene, we need to pause scheduling immediately. In this PR, we drop all region heartbeat responses from the scheduling server once the
IsSchedulingHaltedis changed to true.Check List
Tests
Release note