cluster/spec: workaround if store IDs are not monotonically assigned

[AstroProfundis]

What problem does this PR solve?

Workaround of tikv/pd#3303

A case of the issue is at: https://asktug.com/t/topic/66779

What is changed and how it works?

When we iterate over the store list, check if any store is marked as "tombstone" state, if not, use it as the final state of the "host:port" store, otherwise use the state from store that has the max ID.

Sorting the slice should be not necessary as we are iterating the whole list anyway, and are only finding specific item in.

Check List

Tests

• Manual test (add detailed scripts or steps below)

Code changes

• Has exported function/method change

Side effects

• Increased code complexity

Related changes

• Need to cherry-pick to the release branch

Release notes:

cluster/spec: add workaround to get correct store state when PD is not assigning store ID monotonically

[lucklove]

/lgtm

[codecov-io]

Codecov Report

Merging #1011 (7090aae) into master (d7fed64) will decrease coverage by 6.77%.
The diff coverage is 0.00%.

@@            Coverage Diff             @@
##           master    #1011      +/-   ##
==========================================
- Coverage   55.64%   48.87%   -6.78%     
==========================================
  Files         279      278       -1     
  Lines       19743    19711      -32     
==========================================
- Hits        10986     9633    -1353     
- Misses       7044     8534    +1490     
+ Partials     1713     1544     -169     

Flag	Coverage Δ
cluster	38.40% <0.00%> (-5.17%)	⬇️
dm	?
integrate	44.06% <0.00%> (-5.86%)	⬇️
playground	20.32% <ø> (ø)
tiup	16.50% <0.00%> (+0.02%)	⬆️
unittest	23.01% <0.00%> (-0.01%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
pkg/cluster/spec/tikv.go	57.07% <0.00%> (-0.57%)	⬇️
components/dm/main.go	0.00% <0.00%> (-100.00%)	⬇️
components/dm/spec/bindversion.go	0.00% <0.00%> (-100.00%)	⬇️
components/dm/spec/cluster.go	0.00% <0.00%> (-87.50%)	⬇️
components/dm/spec/logic.go	0.60% <0.00%> (-78.66%)	⬇️
components/cluster/command/check.go	6.27% <0.00%> (-73.27%)	⬇️
pkg/cluster/template/scripts/dm_worker.go	0.00% <0.00%> (-71.43%)	⬇️
pkg/cluster/task/limits.go	0.00% <0.00%> (-68.75%)	⬇️
pkg/cluster/template/scripts/dm_master.go	0.00% <0.00%> (-67.22%)	⬇️
pkg/cluster/task/sysctl.go	0.00% <0.00%> (-66.67%)	⬇️
... and 66 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d7fed64...7090aae. Read the comment docs.

[lucklove]

/merge

[ti-chi-bot]

This pull request has been accepted and is ready to merge.

Commit hash: 7090aae