Performance/health feedback to client and the Unified Health Controller

[MyonKeminta]

Development Task

Currently, there are several ways to evaluate whether a TiKV node is in healthy state. They are:

• SlowScore which is a way to evaluate whether raftstore is in a abnormally slow state. It can be informed to PD which enables PD to trigger the evict-slow-store-scheduler to avoid write requests to be sent to the store.
• SlowTrend, which is improved from SlowScore
• gRPC HealthService, which is a way built-in in gRPC for indicating whether the service is available to serve.

And all these things are managed in PdWorker.

We found the evict-slow-store-scheduler mentioned before is quite helpful to solve the problem that some TiKV nodes encountering slow IO might significantly affect the whole cluster's performance. However, there can still be problem if follower read is being used in the cluster, since it doesn't need to be processed on leader.

To solve the problem, we want the client-go to know whether each TiKV node is abnormal and avoid sending follower read requests to the problematic ones. We are now considering making it able to send some of the health information to the client via kv responses, so that the client has a more efficient and up-to-time information about the TiKV nodes' status and adjust the policy to select replicas.

We are planning to add a component named Unified Health Controller, which will be the unified entrance for managing and accessing the health status of the TiKV node. The SlowScore, SlowTrend and gRPC HealthService mentioned above should be moved to it, and PdWorker will still be responsible for updating them. The component itself should be outside PdWorker, which enables us to access it elsewhere, or add more information to it that are not proper to be updated in PdWorker (e.g., readpool's stats).

TiKV

• Update kvproto to support sending feedback information
  • Add protocol about performance status feeding back via batch command responses pingcap/kvproto#1222
• Add HealthController to TiKV, move SlowScore and, SlowTrend to it
  • *: Add module health_controller and move SlowScore, SlowTrend, HealthService from PdWorker to it #16456
• Send SlowScore as the feedback information to client-go
  • service/kv: Support sending health feedback via BatchCommandResponse #16498

PD

• PDClient support getting StoreStats field in GetStore. It's already included by the RPC protocol but not returned by the golang PDClient.

client-go

• Receive the feedback information and maintain it as part of the store info.
  • Support receving health feedback client-go#1153
• New replica selection implementation that takes the feedback information into consideration.
  • replica selector refactor client-go#1142
• When a request timed out, reload region info quicklier (instead of retrying 10 times as is currently implemented) as the target TiKV might be alive but not serveable.
  • Ref: Retrying sending requests to a stuck TiKV may cost too much time pingcap/tidb#50432
• Support actively requesting health feedback form tikv when it's not receiving it for too long (perhaps due to requests are all evicted from that store so that the connection is idle).
  • server: Support explicitly getting health feedback information by calling RPC #17008

Other related works in client-go repo: tikv/client-go#1104

TiDB

Dependencies of TiDB repo needs to be updated several times, but there isn't any major development task in TiDB repo.
Ref: pingcap/tidb#51412

• store: Update client-go to support receiving feedback information and adapt interface changes pingcap/tidb#51413

Next step

• Including read path status (e.g. read pool busy)
• TBD.