operator: check store status for running operators #4223

disksing · 2021-10-19T08:52:49Z

What problem does this PR solve?

Fix #3353

What is changed and how it works?

For TransferLeader, AddLearner, AddPeer, check if the target store is down and cancel the operator if need.
Other types of OperatorStep may or may not get blocked by down store, we don't restrict them in the PR.
Add tests

Check List

Tests

Unit test

Related changes

Need to cherry-pick to the release branch

Release note

Fix the issue that operator can get blocked due to down store

close tikv#3353 Signed-off-by: disksing <i@disksing.com>

Signed-off-by: disksing <i@disksing.com>

ti-chi-bot · 2021-10-19T08:52:50Z

[REVIEW NOTIFICATION]

This pull request has been approved by:

HunDunDM
nolouch

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

codecov · 2021-10-19T09:01:50Z

Codecov Report

Merging #4223 (46e139b) into master (dbe5e29) will increase coverage by 0.00%.
The diff coverage is 83.33%.

@@           Coverage Diff           @@
##           master    #4223   +/-   ##
=======================================
  Coverage   75.02%   75.02%           
=======================================
  Files         263      263           
  Lines       27307    27317   +10     
=======================================
+ Hits        20486    20494    +8     
+ Misses       5015     5011    -4     
- Partials     1806     1812    +6

Flag	Coverage Δ
unittests	`75.02% <83.33%> (+<0.01%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
server/schedule/operator/step.go	`74.73% <81.81%> (+1.67%)`	⬆️
server/schedule/operator_controller.go	`83.50% <100.00%> (ø)`
pkg/errs/errs.go	`75.00% <0.00%> (-25.00%)`	⬇️
server/region_syncer/client.go	`78.90% <0.00%> (-4.69%)`	⬇️
pkg/dashboard/adapter/manager.go	`79.78% <0.00%> (-3.20%)`	⬇️
server/election/leadership.go	`77.31% <0.00%> (-3.10%)`	⬇️
server/tso/tso.go	`63.63% <0.00%> (-2.28%)`	⬇️
server/core/storage.go	`69.31% <0.00%> (-0.76%)`	⬇️
server/cluster/cluster.go	`82.56% <0.00%> (-0.47%)`	⬇️
server/server.go	`71.70% <0.00%> (-0.30%)`	⬇️
... and 11 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update dbe5e29...46e139b. Read the comment docs.

nolouch · 2021-10-19T11:38:33Z

/run-unit-tests

disksing · 2021-10-19T13:50:53Z

/run-unit-tests

HunDunDM · 2021-11-16T06:22:37Z

server/schedule/operator_controller_test.go

@@ -165,6 +165,25 @@ func (t *testOperatorControllerSuite) TestFastFailOperator(c *C) {
 	c.Assert(oc.GetOperator(region.GetID()), IsNil)
 }

+// Issue 3353
+func (t *testOperatorControllerSuite) TestFastFailOperator2(c *C) {


How about TestFastFailWithUnhealthyStore?

Signed-off-by: disksing <i@disksing.com>

disksing · 2021-11-16T06:59:02Z

/merge

ti-chi-bot · 2021-11-16T06:59:03Z

@disksing: It seems you want to merge this PR, I will help you trigger all the tests:

/run-all-tests

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

ti-chi-bot · 2021-11-16T06:59:04Z

This pull request has been accepted and is ready to merge.

Commit hash: 2c0adbe

nolouch · 2021-11-22T11:57:27Z

/run-unit-tests

nolouch · 2021-11-22T12:05:20Z

/run-unit-tests

nolouch · 2021-11-22T12:38:30Z

/run-unit-tests

disksing · 2021-11-23T08:38:41Z

/run-unit-tests

ti-chi-bot · 2021-11-23T08:52:27Z

In response to a cherrypick label: new pull request created: #4365.

Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>

ti-chi-bot · 2021-11-23T08:53:04Z

In response to a cherrypick label: new pull request created: #4366.

ti-chi-bot · 2021-11-23T08:53:40Z

In response to a cherrypick label: new pull request created: #4367.

ti-chi-bot · 2021-11-23T08:54:15Z

In response to a cherrypick label: new pull request created: #4368.

* operator: check store status for running operators close tikv#3353 Signed-off-by: disksing <i@disksing.com> * add test Signed-off-by: disksing <i@disksing.com> * add tests Signed-off-by: disksing <i@disksing.com> * address comment Signed-off-by: disksing <i@disksing.com>

* operator: check store status for running operators close #3353 Signed-off-by: disksing <i@disksing.com>

* operator: check store status for running operators close #3353 Signed-off-by: disksing <i@disksing.com> * add test Signed-off-by: disksing <i@disksing.com> * add tests Signed-off-by: disksing <i@disksing.com> * address comment Signed-off-by: disksing <i@disksing.com> * fix build Signed-off-by: disksing <i@disksing.com> * fix ci (try) Signed-off-by: disksing <i@disksing.com> Co-authored-by: disksing <i@disksing.com>

disksing added 3 commits October 19, 2021 16:49

operator: check store status for running operators

3dbad10

close tikv#3353 Signed-off-by: disksing <i@disksing.com>

add test

2d59909

Signed-off-by: disksing <i@disksing.com>

add tests

40bc57f

Signed-off-by: disksing <i@disksing.com>

disksing requested review from nolouch and HunDunDM October 19, 2021 08:52

ti-chi-bot added the release-note Denotes a PR that will be considered when it comes time to generate release notes. label Oct 19, 2021

Merge branch 'master' into issue-3353

e3700c3

nolouch approved these changes Oct 21, 2021

View reviewed changes

ti-chi-bot added the status/LGT1 Indicates that a PR has LGTM 1. label Oct 21, 2021

disksing mentioned this pull request Oct 21, 2021

Consider store status for running operators #4232

Open

HunDunDM approved these changes Nov 16, 2021

View reviewed changes

ti-chi-bot added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels Nov 16, 2021

disksing added 2 commits November 16, 2021 14:57

Merge branch 'master' into issue-3353

687286d

address comment

2c0adbe

Signed-off-by: disksing <i@disksing.com>

ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label Nov 16, 2021

Merge branch 'master' into issue-3353

46e139b

ti-chi-bot merged commit fb3d204 into tikv:master Nov 23, 2021

ti-chi-bot mentioned this pull request Nov 23, 2021

operator: check store status for running operators (#4223) #4365

Merged

ti-chi-bot pushed a commit to ti-chi-bot/pd that referenced this pull request Nov 23, 2021

This is an automated cherry-pick of tikv#4223

f87f146

Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>

ti-chi-bot mentioned this pull request Nov 23, 2021

operator: check store status for running operators (#4223) #4366

Closed

ti-chi-bot mentioned this pull request Nov 23, 2021

operator: check store status for running operators (#4223) #4367

Merged

ti-chi-bot mentioned this pull request Nov 23, 2021

operator: check store status for running operators (#4223) #4368

Merged

disksing deleted the issue-3353 branch November 23, 2021 09:04

disksing pushed a commit that referenced this pull request Nov 30, 2021

operator: check store status for running operators (#4223) (#4365)

8ef0b12

* operator: check store status for running operators close #3353 Signed-off-by: disksing <i@disksing.com>

disksing pushed a commit that referenced this pull request Dec 1, 2021

operator: check store status for running operators (#4223) (#4367)

e482eed

* operator: check store status for running operators close #3353 Signed-off-by: disksing <i@disksing.com>

nolouch mentioned this pull request Jan 14, 2022

releases: add TiDB 5.4.0 release notes pingcap/docs-cn#8061

Merged

14 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

operator: check store status for running operators #4223

operator: check store status for running operators #4223

disksing commented Oct 19, 2021 •

edited

Loading

ti-chi-bot commented Oct 19, 2021 •

edited

Loading

codecov bot commented Oct 19, 2021 •

edited

Loading

nolouch commented Oct 19, 2021

disksing commented Oct 19, 2021

HunDunDM Nov 16, 2021

disksing Nov 16, 2021

disksing commented Nov 16, 2021

ti-chi-bot commented Nov 16, 2021

ti-chi-bot commented Nov 16, 2021

nolouch commented Nov 22, 2021

nolouch commented Nov 22, 2021

nolouch commented Nov 22, 2021

disksing commented Nov 23, 2021

ti-chi-bot commented Nov 23, 2021

ti-chi-bot commented Nov 23, 2021

ti-chi-bot commented Nov 23, 2021

ti-chi-bot commented Nov 23, 2021

operator: check store status for running operators #4223

operator: check store status for running operators #4223

Conversation

disksing commented Oct 19, 2021 • edited Loading

What problem does this PR solve?

What is changed and how it works?

Check List

Release note

ti-chi-bot commented Oct 19, 2021 • edited Loading

codecov bot commented Oct 19, 2021 • edited Loading

Codecov Report

nolouch commented Oct 19, 2021

disksing commented Oct 19, 2021

HunDunDM Nov 16, 2021

Choose a reason for hiding this comment

disksing Nov 16, 2021

Choose a reason for hiding this comment

disksing commented Nov 16, 2021

ti-chi-bot commented Nov 16, 2021

ti-chi-bot commented Nov 16, 2021

nolouch commented Nov 22, 2021

nolouch commented Nov 22, 2021

nolouch commented Nov 22, 2021

disksing commented Nov 23, 2021

ti-chi-bot commented Nov 23, 2021

ti-chi-bot commented Nov 23, 2021

ti-chi-bot commented Nov 23, 2021

ti-chi-bot commented Nov 23, 2021

disksing commented Oct 19, 2021 •

edited

Loading

ti-chi-bot commented Oct 19, 2021 •

edited

Loading

codecov bot commented Oct 19, 2021 •

edited

Loading