Speed up balance leader

[nolouch]

Development Task

Currently, balance leader only have max ops 100 op/s. we want to increase the speed when a big cluster restart (2M regions).

one way like: #4008

[nolouch]

cc @CabinfeverB, Would you like to take a look? ptal @rleungx

[CabinfeverB]

I will take a look

[nolouch]

/assign @CabinfeverB

[CabinfeverB]

Motivation

Currently, the MinScheduleInterval param determines the balance-leader speed. According to MinScheduleInterval equals 10 ms, balance-leader only has max ops 100 op/s.

If there are 100K regions that need to balance leader when a big cluster restart (2M regions), it will take 30 minutes. This is an unacceptable time cost

Detailed Design

Considering that the trigger frequency of the scheduler should not be too fast, we decided to add a batch field in the balance-leader scheduler to speed up balance leader by increasing the number of operators generated every scheduling.

In the TiKV, since we believe that the performance overhead of transferring leader in a raft group is small, the transfer-leader operator does not consume the store limit. This means that regions can be repeatedly selected from a store, so a priority-queue-like idea can be adopted. Operators are extracted from the store which has the highest/lowest leader score and we calculate the influence to adjust this top of 'heap', unless it is really impossible to extract. Then extract the next highest/timer low, and so on.

Usage Desc

Since we think the balance leader is an urgent scheduler, we set the Batch parameter of the balance-leader to 4 by default. But considering the potential scheduler competition scenario, we have added an api for configuration. At the same time, related configuration functions will be added into pd-ctl.

Development Plan

Subtasks

#4652 must be involved

• scheduler: allow balance-leader-scheduler generate multiple operators #4652
• api, scheduler: add balance-leader config handler #4655
• pdctl: support config balance-leader #4656

Test Plan

Under the same cluster size, it should be possible to obtain an approximate linear optimization by testing the time to reach the equilibrium state when the Batch is equal to different values.

In order to test whether the original goal can be achieved, it is best to have a large cluster to do the test.

[CabinfeverB]

cc @mayjiang0203