kv (ticdc): fix kvClient reconnection downhill loop

[asddongmen]

What problem does this PR solve?

Issue Number: close #10584

What is changed and how it works?

1. Add an ID to the eventFeedStream struct to identify a stream and prevent it from being deleted unexpectedly.
2. Bind the cancel function of a gRPC stream to its eventFeedStream to prevent the stream from being canceled unexpectedly.
3. Reduce the number of calls to s.deleteStream to only once to prevent the stream from being deleted unexpectedly.

Check List

Tests

• Manual test (add detailed scripts or steps below)

1. Add hard code time.sleep(5 * time.second) before calling s.deleteStream in both unfixed and fixed version of code.
2. Deploy a TiDB cluster with 3 TiKV nodes and 1 TiCDC server, create a changefeed.
3. Patch the changed cdc binary, and restart a TiKV node randomly and observe the changefeed's lag.

unfixed cdc:

fixed cdc:

From the above graphs, it is evident that in the unfixed CDC, the lag of resolvedTs can exceed 12 minutes when a TiKV node is restarted. However, in the fixed CDC, the increase in resolvedTs is limited to a maximum of 35 seconds. This demonstrates the effectiveness of the fix.

Moreover, when the hard-coded time.sleep(5 * time.second) is removed and the fixed version of CDC is tested again, the lag becomes even smaller:

Questions

Will it cause performance regression or break compatibility?

Do you need to update user documentation, design documentation or monitoring documentation?

Release note

Fix a bug in kv client that could cause an increase in changefeed lag when TiKV is restarted.

[3AceShowHand]

/test verify

[codecov]

Codecov Report

Merging #10559 (6aad55e) into master (6600096) will increase coverage by 0.0797%.
Report is 6 commits behind head on master.
The diff coverage is 71.2707%.

Additional details and impacted files

Components	Coverage Δ
cdc	61.9979% <71.2707%> (+0.1840%)	⬆️
dm	51.2214% <ø> (+0.0101%)	⬆️
engine	63.3717% <ø> (-0.0566%)	⬇️

Flag	Coverage Δ
unit	57.5745% <71.2707%> (+0.0797%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

@@               Coverage Diff                @@
##             master     #10559        +/-   ##
================================================
+ Coverage   57.4947%   57.5745%   +0.0797%     
================================================
  Files           848        848                
  Lines        126095     125889       -206     
================================================
- Hits          72498      72480        -18     
+ Misses        48136      47982       -154     
+ Partials       5461       5427        -34     

[ti-chi-bot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: 3AceShowHand, hicqu

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [3AceShowHand,hicqu]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[ti-chi-bot]

[LGTM Timeline notifier]

Timeline:

• 2024-01-30 05:03:27.863205456 +0000 UTC m=+1455449.427503161: ☑️ agreed by hicqu.
• 2024-01-30 06:05:08.294276261 +0000 UTC m=+1459149.858573963: ☑️ agreed by 3AceShowHand.

[ti-chi-bot]

In response to a cherrypick label: new pull request created to branch release-6.5: #10570.

[ti-chi-bot]

In response to a cherrypick label: new pull request created to branch release-7.1: #10571.

[ti-chi-bot]

In response to a cherrypick label: new pull request created to branch release-7.5: #10572.