kv client optimization

[amyangfei]

Feature Request

Is your feature request related to a problem? Please describe:

Currently TiCDC subscribes to the KV changed logs based on region, and allocates an independent goroutine for each region to handle KV changed logs. When the region count is large, there exists huge lock contention in Golang runtime and CPU cost is too high, which will reduce replication performance. Since the stability and performance of kv client have a great impact on TiCDC service, we want to refactor the implement mechanism of kv client in TiCDC, in order to increase stability and performance.

Describe the feature you'd like:

• Implement a better way to handle region’s kv changed logs. kv-client: add stream binding background region worker, not enabled by default #1397, kv-client: use multiple sync.Map to reduce lock contention #1439
• Separate lock resolve logic from each region. kv-client: add stream binding background region worker, not enabled by default #1397, kv-client: add resolved ts heap for quick resolve lock #1426
• Add batch mechanism to region event feeds dispatcher. kv/client: use ephemeral goroutine to handle region range request #1720
• Add more unit tests for kv client, ref: Improve unit tests of kv client #1292
• Add benchmark for kv client. kv-client: add benchmark for kv client #1448
• Real scenario benchmark with new kv client ga matrix
• Support kv client reconnect when resolved ts in TiKV is blocked Support kv client reconnect when resolved ts in TiKV is blocked #1586
  • cherry pick reconnect logic of client v2 to 5.0

New bugs found in 5.0 test

• region range lock blocking issue region not receiving event from tikv for too long time cause the batch resolvedts event is dropped #1599. Both in v1 and v2
• performance bottleneck in error handling and lock resolver Another resolved ts not forward issue caused by slow error handling #1612. Both in v1 and v2
• The CRTs must be greater than the resolvedTs, panic in cdc server: The CRTs must be greater than the resolvedTs #1548 . v2 only
• kv client may raise error when stream.Recv meets io.EOF in some scenarios. Both in v1 and v2. kv/client: fix the error handling for io.EOF may miss region reconnect #1641