apiutil/middleware: add retry logic for obtaining PD leader in redirector

[JmPotato]

What problem does this PR solve?

Issue Number: close #8142.

What is changed and how does it work?

Add retry logic to improve PD HTTP request forwarding success rate during PD leader switch.

Check List

Tests

• Unit test
• Integration test

Release note

None.

[ti-chi-bot]

[REVIEW NOTIFICATION]

This pull request has been approved by:

• HuSharp
• rleungx

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 77.33%. Comparing base (dd7f2a7) to head (1b580df).

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #8216      +/-   ##
==========================================
- Coverage   77.40%   77.33%   -0.07%     
==========================================
  Files         471      471              
  Lines       61456    61478      +22     
==========================================
- Hits        47569    47544      -25     
- Misses      10323    10377      +54     
+ Partials     3564     3557       -7     

Flag	Coverage Δ
unittests	77.33% <100.00%> (-0.07%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

[rleungx]

LGTM, but I was thinking that compared with retry in pd, will it be more reasonable to send a request to the leader directly and backoff on the client side?

[JmPotato]

LGTM, but I was thinking that compared with retry in pd, will it be more reasonable to send a request to the leader directly and backoff on the client side?

I agree; client retry should be the most straightforward choice. However, considering that this PR only takes effect during the interval of PD leader re-election, a small internal retry attempt would be lighter and faster than having the client receive an error and then retry. The two are not in conflict.

[JmPotato]

/merge

[ti-chi-bot]

@JmPotato: It seems you want to merge this PR, I will help you trigger all the tests:

/run-all-tests

You only need to trigger /merge once, and if the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

If you have any questions about the PR merge process, please refer to pr process.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

[ti-chi-bot]

This pull request has been accepted and is ready to merge.

Commit hash: 1b580df

[lhy1024]

Does it need to be picked to other branches?

[JmPotato]

Does it need to be picked to other branches?

Let's observe the effect of this change for a period of time before cherry-picking.

[ti-chi-bot]

In response to a cherrypick label: new pull request created to branch release-8.1: #8466.