br: Enable checkpoint advancer to pause tasks lagged too large

[RidRisR]

What problem does this PR solve?

Issue Number: close #50803

Problem Summary:

What changed and how does it work?

Now, a new option called CheckPointLagLimit is added to the advancer config. When it is set, advancer will check if the checkpoint is lagged too large on every tick. If so, advancer would send a warning to PD and unstall the task.
PS: Theoretically, we need a resume signal when the task could be resumed. However, there is no channel to send this signal now. Maybe we will add it in the future.

Check List

Tests

• Unit test
• Integration test
• Manual test (add detailed scripts or steps below)
• No need to test

  • I checked and no code files have been changed.

Side effects

• Performance regression: Consumes more CPU
• Performance regression: Consumes more Memory
• Breaking backward compatibility

Documentation

• Affects user behaviors
• Contains syntax changes
• Contains variable changes
• Contains experimental features
• Changes MySQL compatibility

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

A new option called `CheckPointLagLimit` is added to the advancer config. When it is set, advancer will check if the checkpoint is lagged too large on every tick. If so, advancer would send a warning to PD and unstall the task.

[tiprow]

Hi @RidRisR. Thanks for your PR.

PRs from untrusted users cannot be marked as trusted with /ok-to-test in this repo meaning untrusted PR authors can never trigger tests themselves. Collaborators can still trigger tests on the PR using /test all.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[3pointer]

LGTM. Note: for now it seems this function is disabled by default. For now, the configuration of TiDB cannot be passed to advancer, which means this function won't be usable in a TiDB binary. Should we enable this with a value safe enough, say, 72h or even longer? cc @3pointer @BornChanger

I think we should enable it with a default value, 3 days is safe enough and we could not let log backup block GC to much.

[BornChanger]

/type cherry-pick-for-release-6,5

[ti-chi-bot]

@BornChanger: The label(s) type/cherry-pick-for-release-6,5 cannot be applied, because the repository doesn't have them.

In response to this:

/type cherry-pick-for-release-6,5

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

[BornChanger]

/type cherry-pick-for-release-6.5

[BornChanger]

/type cherry-pick-for-release-7.1

[BornChanger]

/type cherry-pick-for-release-7.5

[ti-chi-bot]

In response to a cherrypick label: new pull request created to branch release-6.5: #52105.

[BornChanger]

/label needs-cherry-pick-release-7.1

[ti-chi-bot]

In response to a cherrypick label: new pull request created to branch release-7.1: #52554.

[ti-chi-bot]

In response to a cherrypick label: new pull request created to branch release-7.5: #52555.