add index failed when enable dist task after change the pd member

[aytrack]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

1. deploy with 3pd(tc-pd0, tc-pd1, tc-pd2), 2tidb, 3tikv
2. set global tidb_enable_dist_task = 1;
3. scale-out pd to 6 replicas, add (tc-pd3, tc-pd4, tc-pd5)
4. using pd-ctl to delete the tc-pd0, tc-pd1, tc-pd2
5. do add index

2. What did you expect to see? (Required)

add index should success

3. What did you see instead (Required)

add index failed with get member failed

[18:28:43]TiDB root:test> set global tidb_enable_dist_task = 0;
Query OK, 0 rows affected
Time: 0.049s
-- reconnect
[18:29:13]TiDB root:test> alter table item_xxx add index proxxxx_id(proxxxx_id);
Query OK, 0 rows affected
Time: 246.075s
[18:33:32]TiDB root:test> set global tidb_enable_dist_task = 1;
Query OK, 0 rows affected
Time: 0.048s
-- reconnect
[18:33:55]TiDB root:test> alter table item_xxx add index  item_xx_id(item_xx_id);
(1105, 'get member failed')
[18:35:46]TiDB root:test>

tidb log

2023/11/17 18:34:01.808 +08:00] [INFO] [ddl.go:1189] ["DDL job is failed"] [category=ddl] [jobID=124]
[2023/11/17 18:34:01.809 +08:00] [INFO] [tidb.go:285] ["rollbackTxn called due to ddl/autocommit failure"]
[2023/11/17 18:34:01.809 +08:00] [WARN] [session.go:2283] ["run statement failed"] [conn=2338325182] [session_alias=] [schemaVersion=94] [error="[PD:client:ErrClientGetMember]get member failed"] [session="{\n  \"currDBName\": \"test\",\n  \"id\": 2338325182,\n  \"status\": 2,\n  \"strictMode\": true,\n  \"user\": {\n    \"Username\": \"root\",\n    \"Hostname\": \"10.200.72.158\",\n    \"CurrentUser\": false,\n    \"AuthUsername\": \"root\",\n    \"AuthHostname\": \"%\",\n    \"AuthPlugin\": \"mysql_native_password\"\n  }\n}"]
[2023/11/17 18:34:01.809 +08:00] [INFO] [conn.go:1098] ["command dispatched failed"] [conn=2338325182] [session_alias=] [connInfo="id:2338325182, addr:10.200.72.158:51298 status:10, collation:utf8_general_ci, user:root"] [command=Query] [status="inTxn:0, autocommit:1"] [sql="alter table item_core add index  item_set_id(item_set_id)"] [txn_mode=PESSIMISTIC] [timestamp=445701747058147330] [err="[PD:client:ErrClientGetMember]get member failed\ngithub.com/pingcap/errors.AddStack\n\t/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20221009092201-b66cddb77c32/errors.go:174\ngithub.com/pingcap/errors.Trace\n\t/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20221009092201-b66cddb77c32/juju_adaptor.go:15\ngithub.com/pingcap/tidb/pkg/ddl.(*ddl).DoDDLJob\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/ddl/ddl.go:1190\ngithub.com/pingcap/tidb/pkg/ddl.(*ddl).createIndex\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/ddl/ddl_api.go:7337\ngithub.com/pingcap/tidb/pkg/ddl.(*ddl).AlterTable\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/ddl/ddl_api.go:3711\ngithub.com/pingcap/tidb/pkg/executor.(*DDLExec).executeAlterTable\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/executor/ddl.go:385\ngithub.com/pingcap/tidb/pkg/executor.(*DDLExec).Next\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/executor/ddl.go:149\ngithub.com/pingcap/tidb/pkg/executor/internal/exec.Next\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/executor/internal/exec/executor.go:283\ngithub.com/pingcap/tidb/pkg/executor.(*ExecStmt).next\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/executor/adapter.go:1223\ngithub.com/pingcap/tidb/pkg/executor.(*ExecStmt).handleNoDelayExecutor\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/executor/adapter.go:968\ngithub.com/pingcap/tidb/pkg/executor.(*ExecStmt).handleNoDelay\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/executor/adapter.go:794\ngithub.com/pingcap/tidb/pkg/executor.(*ExecStmt).Exec\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/executor/adapter.go:575\ngithub.com/pingcap/tidb/pkg/session.runStmt\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/session/session.go:2429\ngithub.com/pingcap/tidb/pkg/session.(*session).ExecuteStmt\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/session/session.go:2271\ngithub.com/pingcap/tidb/pkg/server.(*TiDBContext).ExecuteStmt\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/server/driver_tidb.go:292\ngithub.com/pingcap/tidb/pkg/server.(*clientConn).handleStmt\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/server/conn.go:2010\ngithub.com/pingcap/tidb/pkg/server.(*clientConn).handleQuery\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/server/conn.go:1801\ngithub.com/pingcap/tidb/pkg/server.(*clientConn).dispatch\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/server/conn.go:1288\ngithub.com/pingcap/tidb/pkg/server.(*clientConn).Run\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/server/conn.go:1067\ngithub.com/pingcap/tidb/pkg/server.(*Server).onConn\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/server/server.go:700\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1650"]
[2023/11/17 18:34:01.809 +08:00] [WARN] [terror.go:242] ["Unknown error class"] [class=PD]
[2023/11/17 18:34:09.467 +08:00] [INFO] [pd_service_discovery.go:524] ["[pd] cannot update member from this address"] [address=http://tc-pd-3.tc-pd-peer.sadd-index-comp-hchwang-tps-4710030-1-294.svc:2379] [error="[PD:client:ErrClientGetLeader]get leader failed, leader address doesn't exist"]
[2023/11/17 18:34:09.467 +08:00] [INFO] [pd_service_discovery.go:524] ["[pd] cannot update member from this address"] [address=http://tc-pd-4.tc-pd-peer.sadd-index-comp-hchwang-tps-4710030-1-294.svc:2379] [error="[PD:client:ErrClientGetLeader]get leader failed, leader address doesn't exist"]
[2023/11/17 18:35:09.462 +08:00] [INFO] [pd_service_discovery.go:524] ["[pd] cannot update member from this address"] [address=http://tc-pd-3.tc-pd-peer.sadd-index-comp-hchwang-tps-4710030-1-294.svc:2379] [error="[PD:client:ErrClientGetLeader]get leader failed, leader address doesn't exist"]
[2023/11/17 18:35:09.463 +08:00] [INFO] [pd_service_discovery.go:524] ["[pd] cannot update member from this address"] [address=http://tc-pd-4.tc-pd-peer.sadd-index-comp-hchwang-tps-4710030-1-294.svc:2379] [error="[PD:client:ErrClientGetLeader]get leader failed, leader address doesn't exist"]
[2023/11/17 18:36:09.462 +08:00] [INFO] [pd_service_discovery.go:524] ["[pd] cannot update member from this address"] [address=http://tc-pd-3.tc-pd-peer.sadd-index-comp-hchwang-tps-4710030-1-294.svc:2379] [error="[PD:client:ErrClientGetLeader]get leader failed, leader address doesn't exist"]
[2023/11/17 18:36:09.463 +08:00] [INFO] [pd_service_discovery.go:524] ["[pd] cannot update member from this address"] [address=http://tc-pd-4.tc-pd-peer.sadd-index-comp-hchwang-tps-4710030-1-294.svc:2379] [error="[PD:client:ErrClientGetLeader]get leader failed, leader address doesn't exist"]
[2023/11/17 18:36:09.488 +08:00] [INFO] [domain.go:2830] ["refreshServerIDTTL succeed"] [serverID=1115] ["lease id"=54b98bdc56a09151]
[2023/11/17 18:37:09.463 +08:00] [INFO] [pd_service_discovery.go:524] ["[pd] cannot update member from this address"] [address=http://tc-pd-3.tc-pd-peer.sadd-index-comp-hchwang-tps-4710030-1-294.svc:2379] [error="[PD:client:ErrClientGetLeader]get leader failed, leader address doesn't exist"]
[2023/11/17 18:37:09.463 +08:00] [INFO] [pd_service_discovery.go:524] ["[pd] cannot update member from this address"] [address=http://tc-pd-4.tc-pd-peer.sadd-index-comp-hchwang-tps-4710030-1-294.svc:2379] [error="[PD:client:ErrClientGetLeader]get leader failed, leader address doesn't exist"]

pd member

» member
{
  "header": {
    "cluster_id": 7302340897250549827
  },
  "members": [
    {
      "name": "tc-pd-4",
      "member_id": 72055938477896569,
      "peer_urls": [
        "http://tc-pd-4.tc-pd-peer.sadd-index-comp-hchwang-tps-4710030-1-294.svc:2380"
      ],
      "client_urls": [
        "http://tc-pd-4.tc-pd-peer.sadd-index-comp-hchwang-tps-4710030-1-294.svc:2379"
      ],
      "deploy_path": "/",
      "binary_version": "v7.5.0",
      "git_hash": "ef6ba8551e525a700546d6bdb7ad6766115209cc"
    },
    {
      "name": "tc-pd-5",
      "member_id": 2577259733341751556,
      "peer_urls": [
        "http://tc-pd-5.tc-pd-peer.sadd-index-comp-hchwang-tps-4710030-1-294.svc:2380"
      ],
      "client_urls": [
        "http://tc-pd-5.tc-pd-peer.sadd-index-comp-hchwang-tps-4710030-1-294.svc:2379"
      ],
      "deploy_path": "/",
      "binary_version": "v7.5.0",
      "git_hash": "ef6ba8551e525a700546d6bdb7ad6766115209cc"
    },
    {
      "name": "tc-pd-3",
      "member_id": 12488526090564133148,
      "peer_urls": [
        "http://tc-pd-3.tc-pd-peer.sadd-index-comp-hchwang-tps-4710030-1-294.svc:2380"
      ],
      "client_urls": [
        "http://tc-pd-3.tc-pd-peer.sadd-index-comp-hchwang-tps-4710030-1-294.svc:2379"
      ],
      "deploy_path": "/",
      "binary_version": "v7.5.0",
      "git_hash": "ef6ba8551e525a700546d6bdb7ad6766115209cc"
    }
  ],
  "leader": {
    "name": "tc-pd-5",
    "member_id": 2577259733341751556,
    "peer_urls": [
      "http://tc-pd-5.tc-pd-peer.sadd-index-comp-hchwang-tps-4710030-1-294.svc:2380"
    ],
    "client_urls": [
      "http://tc-pd-5.tc-pd-peer.sadd-index-comp-hchwang-tps-4710030-1-294.svc:2379"
    ],
    "deploy_path": "/",
    "binary_version": "v7.5.0",
    "git_hash": "ef6ba8551e525a700546d6bdb7ad6766115209cc"
  },
  "etcd_leader": {
    "name": "tc-pd-5",
    "member_id": 2577259733341751556,
    "peer_urls": [
      "http://tc-pd-5.tc-pd-peer.sadd-index-comp-hchwang-tps-4710030-1-294.svc:2380"
    ],
    "client_urls": [
      "http://tc-pd-5.tc-pd-peer.sadd-index-comp-hchwang-tps-4710030-1-294.svc:2379"
    ],
    "deploy_path": "/",
    "binary_version": "v7.5.0",
    "git_hash": "ef6ba8551e525a700546d6bdb7ad6766115209cc"
  }
}

4. What is your TiDB version? (Required)

v7.5.0

[seiya-annie]

/report customer