kv/client: fix region loss in single region handler

[amyangfei]

What problem does this PR solve?

Close #3288

Fix a region loss case, we can reproduce as following steps

1. Create a TiDB cluster with 5 TiKV nodes, create a table with 4TB data(or 100K regions)
2. Setup a TiCDC cluster to replicate table in step-1
3. Use TiUP to restart all these TiKVs without transferring leader region, by tiup cluster restart <cluster-name> -R tikv command for example.
4. Observe the replication stucks even all regions are initialized. (Both cached regions and scanning regions are zero)

By comparing the initialized regions in TiCDC and all regions by querying select region_id from information_schema.tikv_region_status where db_name = 'xx' and table_name = 'yy' from TiDB, we can observe some regions are lost.

By querying the lost region id in TiCDC log, we found the region disconnected without reconnect

[2021/11/04 17:22:10.083 +08:00] [INFO] [client.go:777] ["start new request"] [request="{\"header\":{\"cluster_id\":7012827302444215878,\"ticdc_version\":\"5.2.0-master\"},\"region_id\":1122272,\"region_epoch\":{\"conf_ver\":980,\"version\":57145},\"checkpoint_ts\":428872226710487042,\"start_key\":\"dIAAAAAAAAD/L19ygAAAADL/CQwfAAAAAAD6\",\"end_key\":\"dIAAAAAAAAD/L19ygAAAADL/CpK/AAAAAAD6\",\"request_id\":383728,\"extra_op\":1,\"Request\":null}"] [addr=172.16.7.56:20161]
[2021/11/04 17:22:10.086 +08:00] [INFO] [region_worker.go:243] ["single region event feed disconnected"] [regionID=1122272] [requestID=383728] [span="[7480000000000000ff2f5f728000000032ff090c1f0000000000fa, 7480000000000000ff2f5f728000000032ff0a92bf0000000000fa)"] [checkpoint=428872226710487042] [error="[CDC:ErrEventFeedEventError]not_leader:<region_id:1122272 > : not_leader:<region_id:1122272 > "]
[2021/11/04 17:22:10.087 +08:00] [INFO] [region_range_lock.go:370] ["unlocked range"] [lockID=105] [regionID=1122272] [startKey=7480000000000000ff2f5f728000000032ff090c1f0000000000fa] [endKey=7480000000000000ff2f5f728000000032ff0a92bf0000000000fa] [checkpointTs=428872226710487042]

The root cause is kv client must recycle all failed regions, so we should use the root context of a kv client to call onRegionFail

This bug tends to happen when multiple TiKVs crash or forcing restart, and based on existing test, one TiKV crashes or restarts doesn't trigger this bug. And the more regions, the higher probability.

What is changed and how it works?

Use the parent context of region worker to call onRegionFail when processing region failure.

Check List

Tests

• Unit test
• Manual test (add detailed scripts or steps below)
  • As the PR description described
  • The Detail Test Scenario and Report can be found in Replication stability and latency issue in TiCDC, after this PR, TiCDC passes the test of Force restart of all TiKV nodes

Release note

Fix a bug that TiCDC could meet replication interruption when multiple TiKVs crash or forcing restart.

[ti-chi-bot]

[REVIEW NOTIFICATION]

This pull request has been approved by:

• hi-rustin
• overvenus

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

[amyangfei]

/run-all-tests

[amyangfei]

/run-integration-tests

[amyangfei]

/run-leak-tests

[codecov-commenter]

Codecov Report

Merging #3281 (649cc62) into master (de2fca8) will increase coverage by 0.0119%.
The diff coverage is 81.5789%.

@@               Coverage Diff                @@
##             master      #3281        +/-   ##
================================================
+ Coverage   56.7226%   56.7345%   +0.0119%     
================================================
  Files           214        214                
  Lines         22915      22919         +4     
================================================
+ Hits          12998      13003         +5     
+ Misses         8604       8603         -1     
  Partials       1313       1313                

[Rustin170506]

Do we need to add some comments here? I see another place where it is used added a comment why we use parentCtx instead of ctx.

See: https://github.com/pingcap/ticdc/blob/37bac66f0673103892c71e585ab3d9d4658c1f74/cdc/kv/region_worker.go#L792
It looks like the changes here are also for this purpose, sorry I'm not very familiar with this piece of code.

[amyangfei]

yes, the same reason

[amyangfei]

/merge

[ti-chi-bot]

This pull request has been accepted and is ready to merge.

Commit hash: dadd865

[ti-chi-bot]

@amyangfei: Your PR was out of date, I have automatically updated it for you.

At the same time I will also trigger all tests for you:

/run-all-tests

If the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

[ti-chi-bot]

In response to a cherrypick label: new pull request created: #3289.

[ti-chi-bot]

In response to a cherrypick label: new pull request created: #3290.

[ti-chi-bot]

In response to a cherrypick label: new pull request created: #3291.

[ti-chi-bot]

In response to a cherrypick label: new pull request created: #3292.

[ti-chi-bot]

In response to a cherrypick label: new pull request created: #3293.