rule checker: try to remove orphan peer after fix rule peers

[disksing]

Signed-off-by: disksing i@disksing.com

What problem does this PR solve?

Fix #2419

What is changed and how it works?

• always try to remove orphan peers, even if failed to replace down/offline peer
• add restriction to remove orphan peer, it only be checked when all rules are satisfied (to avoid too small replica count)

Check List

Tests

• Unit test
• Manual test (add detailed scripts or steps below)

1. start cluster with 4 tikv (A, B, C, D)
2. enable placement rules, set rules to 3voters+1learner
3. the region becomes voter(A) + voter(B) + voter(C) + learner(D)
4. set C to offline
5. remove the learner rule
6. make sure region becomes voeter(A) + voter(B) + voter(D)
7. check PD's log, operator history is `remove D + replace C with D`

Related changes

• Need to cherry-pick to the release branch

Release note

• fix the placement rules issue that prevent store changing to tombstone

[codecov-commenter]

Codecov Report

Merging #2546 into master will increase coverage by 0.15%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master    #2546      +/-   ##
==========================================
+ Coverage   77.11%   77.26%   +0.15%     
==========================================
  Files         204      204              
  Lines       22067    22071       +4     
==========================================
+ Hits        17016    17053      +37     
+ Misses       3771     3740      -31     
+ Partials     1280     1278       -2     

Impacted Files	Coverage Δ
server/schedule/checker/rule_checker.go	82.14% <100.00%> (+5.92%)	⬆️
pkg/metricutil/metricutil.go	86.11% <0.00%> (-5.56%)	⬇️
server/region_syncer/client.go	79.38% <0.00%> (-5.35%)	⬇️
pkg/etcdutil/etcdutil.go	82.60% <0.00%> (-1.45%)	⬇️
server/core/storage.go	74.38% <0.00%> (-0.83%)	⬇️
server/tso/tso.go	83.21% <0.00%> (-0.73%)	⬇️
server/server.go	76.48% <0.00%> (-0.46%)	⬇️
server/cluster/cluster.go	79.93% <0.00%> (-0.43%)	⬇️
server/grpc_service.go	59.96% <0.00%> (-0.20%)	⬇️
server/schedule/operator_controller.go	78.69% <0.00%> (+0.16%)	⬆️
... and 9 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 33bf660...2eed7ec. Read the comment docs.

[nolouch]

lgtm

[nolouch]

/merge

[ti-srebot]

/run-all-tests

[ti-srebot]

cherry pick to release-3.1 in PR #2553

[ti-srebot]

cherry pick to release-4.0 in PR #2554

[HunDunDM]

LGTM