ebs br: add resilience to single TiKV crash #5572
Conversation
BornChanger
changed the title
BornChanger
force-pushed
the
tolerate_single_wal_corruption
branch
from
5a1fe13
to
b34d39f
Compare
Signed-off-by: BornChanger <dawn_catcher@126.com>
Signed-off-by: BornChanger <dawn_catcher@126.com>
BornChanger
force-pushed
the
tolerate_single_wal_corruption
branch
from
b34d39f
to
ade2e19
Compare
BornChanger
changed the title
Signed-off-by: BornChanger <dawn_catcher@126.com>
|
/run-pull-e2e-kind-br |
Signed-off-by: BornChanger <dawn_catcher@126.com>
|
/run-pull-e2e-kind-br |
|
@YuJuncen: adding LGTM is restricted to approvers and reviewers in OWNERS files. In response to this: Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: WangLe1321, YuJuncen The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
[LGTM Timeline notifier]Timeline:
|
|
New changes are detected. LGTM label has been removed. |
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #5572 +/- ##
==========================================
- Coverage 61.52% 61.51% -0.02%
==========================================
Files 235 235
Lines 30314 30314
==========================================
- Hits 18651 18647 -4
- Misses 9796 9798 +2
- Partials 1867 1869 +2
|
|
/cherry-pick release-1.5 |
|
@csuzhangxc: new pull request created to branch In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository. |
|
@csuzhangxc: new pull request could not be created: failed to create pull request against pingcap/tidb-operator#release-1.5 from head ti-chi-bot:cherry-pick-5572-to-release-1.5: status code 422 not one of [201], body: {"message":"Validation Failed","errors":[{"resource":"PullRequest","code":"custom","message":"A pull request already exists for ti-chi-bot:cherry-pick-5572-to-release-1.5."}],"documentation_url":"https://docs.github.com/rest/pulls/pulls#create-a-pull-request"} In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository. |
| if len(tc.Status.TiKV.Stores) != int(tc.Spec.TiKV.Replicas) { | ||
| func (tc *TidbCluster) AllTiKVsAreAvailable(tolerateSingleTiKVOutage bool) bool { | ||
| if (!tolerateSingleTiKVOutage && len(tc.Status.TiKV.Stores) != int(tc.Spec.TiKV.Replicas)) || | ||
| (tolerateSingleTiKVOutage && int(tc.Spec.TiKV.Replicas-1) != len(tc.Status.TiKV.Stores)) { |
There was a problem hiding this comment.
@BornChanger
would this check return false if we enable tolerateSingleTiKVOutage and don't observe a WAL corruption (ie. replicas == stores)? should it be stores >= replicas - 1
There was a problem hiding this comment.
It's not a common fix but to work-around a single TiKV node crashloop. So, the process is that we observed one TiKV crashes due to the wal corruption issue, THEN turn on tolerateSingleTiKVOutage of the restore CR.
What problem does this PR solve?
This PR provides the resilience to a single TiKV crash due to wal corruption at restore
What is changed and how does it work?
Add a new attribute
spec.tolerateSingleTiKVOutageto restore CR. When it's set to true, restore to the tc can tolerate a single TiKV outage due to wal log corruption issue.Code changes
Tests
Side effects
Related changes
Release Notes
Please refer to Release Notes Language Style Guide before writing the release note.