PD keeps transfering leader to a down store

[BusyJay]

Bug Report

What did you do?

I used nightly build of PD to test joint consensus.

I enabled shuffle region scheduling and set max store down time to 30s. After killing two stores in the same label, a region will stuck and keep 4 replicas until 10min. One example looked like following:

"region_id": 129,
"peers": [
    {
      "id": 180171,
      "store_id": 1
    },
    {
      "id": 181496,
      "store_id": 4
    },
    {
      "id": 181653,
      "store_id": 13
    },
    {
      "id": 181850,
      "store_id": 5,
      "role": 1
    }
  ],
  "leader": {
    "id": 180171,
    "store_id": 1
  },
  "down_peers": [
    {
      "peer": {
        "id": 181496,
        "store_id": 4
      },
      "down_seconds": 360
    }
  ],

, no pending peer was reported. Store 4 was killed and PD's log kept report

[2021/01/12 12:29:23.613 +08:00] [INFO] [operator_controller.go:626] ["send schedule command"] [region-id=129] [step="transfer leader from store 1 to store 4"] [source="active push"]

[HunDunDM]

The current PD uses a timeout of 10 minutes (not configurable) for the entire operator including peer movement. It does not use a separate timeout for each step, which is the main cause of this bug.

[HunDunDM]

Maybe we should increase the judgment of the store state during the scheduling.