Enhance ServiceDiscovery to improve client high availability and load balance

[CabinfeverB]

Enhancement Task

Currently, the ServiceDiscovery is used in the pd gRPC client, and it is not directly exposed to the users. Now, we have encapsulated an HTTP SDK, which also requires service discovery functionality. Therefore, we need to optimize the abstraction of ServiceDiscovery to be used by both the pd gRPC client and the pd HTTP client.

Additionally, ServiceDiscovery also needs load balancing for active follower-handle and better maintenance of the health status of leaders and followers, among other things. This aspect also requires optimization.

Moreover, after implementing the scheduling microservice, the related requests can be routed directly to the scheduling server through service discovery, instead of going through the API server. This reduces the resource cost on the API server as well.

Task:

• client: Introduce ServiceClient #7489
• client: use ServiceClient for discovery #7611
• replace more funtion in ServiceDiscovery with ServiceClient for tso
• use ServiceDiscovery for http client