grpc: Limit the qps for all GRPC interfaces.

[AndreMouche]

Enhancement Task

Currently, we have implemented rate limiting on some interfaces of grpc：

• Get Members
  

  pd/server/grpc_service.go

  Lines 416 to 417 in a6e855e

  	func (s *GrpcServer) GetMembers(context.Context, *pdpb.GetMembersRequest) (*pdpb.GetMembersResponse, error) {
  	if s.GetServiceMiddlewarePersistOptions().IsGRPCRateLimitEnabled() {

• GetStore
  

  pd/server/grpc_service.go

  Lines 646 to 648 in a6e855e

  	// GetStore implements gRPC PDServer.
  	func (s *GrpcServer) GetStore(ctx context.Context, request *pdpb.GetStoreRequest) (*pdpb.GetStoreResponse, error) {
  	if s.GetServiceMiddlewarePersistOptions().IsGRPCRateLimitEnabled() {

• GetAllStores
  

  pd/server/grpc_service.go

  Lines 749 to 751 in a6e855e

  	// GetAllStores implements gRPC PDServer.
  	func (s *GrpcServer) GetAllStores(ctx context.Context, request *pdpb.GetAllStoresRequest) (*pdpb.GetAllStoresResponse, error) {
  	if s.GetServiceMiddlewarePersistOptions().IsGRPCRateLimitEnabled() {

• StoreHeartbeat
  

  pd/server/grpc_service.go

  Lines 794 to 796 in a6e855e

  	// StoreHeartbeat implements gRPC PDServer.
  	func (s *GrpcServer) StoreHeartbeat(ctx context.Context, request *pdpb.StoreHeartbeatRequest) (*pdpb.StoreHeartbeatResponse, error) {
  	if s.GetServiceMiddlewarePersistOptions().IsGRPCRateLimitEnabled() {

• GetRegion
  https://github.com/tikv/pd/blob/a6e855eef6744adfac232833769219be4f806756/server/grpc_service.go#L1265C4-L1267
• GetPrevRegion
  

  pd/server/grpc_service.go

  Lines 1309 to 1311 in a6e855e

  	// GetPrevRegion implements gRPC PDServer
  	func (s *GrpcServer) GetPrevRegion(ctx context.Context, request *pdpb.GetRegionRequest) (*pdpb.GetRegionResponse, error) {
  	if s.GetServiceMiddlewarePersistOptions().IsGRPCRateLimitEnabled() {

• GetRegionByID
  

  pd/server/grpc_service.go

  Lines 1354 to 1356 in a6e855e

  	// GetRegionByID implements gRPC PDServer.
  	func (s *GrpcServer) GetRegionByID(ctx context.Context, request *pdpb.GetRegionByIDRequest) (*pdpb.GetRegionResponse, error) {
  	if s.GetServiceMiddlewarePersistOptions().IsGRPCRateLimitEnabled() {

• ScanRegions
  

  pd/server/grpc_service.go

  Lines 1398 to 1400 in a6e855e

  	// ScanRegions implements gRPC PDServer.
  	func (s *GrpcServer) ScanRegions(ctx context.Context, request *pdpb.ScanRegionsRequest) (*pdpb.ScanRegionsResponse, error) {
  	if s.GetServiceMiddlewarePersistOptions().IsGRPCRateLimitEnabled() {

While for other apis, we do not support rate limit:

• AskSplit
• AskBatchSplit
• ReportSplit
• ReportBatchSplit
• GetClusterConfig
• PutClusterConfig
• ScatterRegion
• GetGCSafePoint
• SyncRegions
• UpdateGCSafePoint
• UpdateServiceGCSafePoint
• GetOperator

These APIs that are not subject to flow control are also vulnerable to attacks, which can easily overload the PD.
for example , if I run the following SQL on the TiDB:

CREATE TABLE employees (
    id int unsigned NOT NULL,
    fname varchar(30),
    lname varchar(30),
    hired date NOT NULL DEFAULT '1970-01-01',
    separated date DEFAULT '9999-12-31',
    job_code int,
    store_id int NOT NULL
) PARTITION BY RANGE (id)
INTERVAL (100) FIRST PARTITION LESS THAN (1000) LAST PARTITION LESS THAN (100000) MAXVALUE PARTITION;

mysql> split partition table t  between (0) and (100000000) regions 1000;
split partition table t  between (0) and (10000000) regions 1000;

then the qps of AskBatchSplit will super high, and make the leader PD's memory and CPU explode instantly.

there are at least two things we can do:

1. add ratelimit for this API
2. limit the batch size for batch-split

[CabinfeverB]

The the batch size has limit by TiDB which is 1000.
Can u post some metrics like CPU/Memory/Goroutin? @AndreMouche

[rleungx]

It has already been supported, close it.

[rleungx]

tracked by #5739