Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

PD can't be added to existing cluster. #126

Closed
weekface opened this issue Oct 17, 2018 · 6 comments
Closed

PD can't be added to existing cluster. #126

weekface opened this issue Oct 17, 2018 · 6 comments
Labels
type/bug Something isn't working

Comments

@weekface
Copy link
Contributor

When I scaled the PD cluster, the new PD member may not be added to existing cluster some time.

2018/10/17 07:03:02.799 util.go:62: [info] Welcome to Placement Driver (PD).
2018/10/17 07:03:02.799 util.go:63: [info] Release Version: v2.0.5
2018/10/17 07:03:02.799 util.go:64: [info] Git Commit Hash: b64716707b7279a4ae822be767085ff17b5f3fea
2018/10/17 07:03:02.800 util.go:65: [info] Git Branch: release-2.0
2018/10/17 07:03:02.800 util.go:66: [info] UTC Build Time:  2018-09-07 12:34:46
2018/10/17 07:03:02.800 metricutil.go:83: [info] disable Prometheus push client
2018/10/17 07:03:02.835 main.go:73: [fatal] join error etcdserver: unhealthy cluster
/home/jenkins/workspace/build_pd_2.0/go/src/github.com/pingcap/pd/pkg/etcdutil/etcdutil.go:82:
/home/jenkins/workspace/build_pd_2.0/go/src/github.com/pingcap/pd/server/join.go:119:

There is a no-name member:

{
  "header": {
    "cluster_id": 6612926505300202376
  },
  "members": [
    {
      "member_id": 1429482795436666386,
      "peer_urls": [
        "http://demo-pd-3.demo-pd-peer.tidb.svc:2380"
      ]
    },
    {
      "name": "demo-pd-2",
      "member_id": 3630075772193212308,
      "peer_urls": [
        "http://demo-pd-2.demo-pd-peer.tidb.svc:2380"
      ],
      "client_urls": [
        "http://demo-pd-2.demo-pd-peer.tidb.svc:2379"
      ]
    },
    {
      "name": "demo-pd-0",
      "member_id": 13229517333287924650,
      "peer_urls": [
        "http://demo-pd-0.demo-pd-peer.tidb.svc:2380"
      ],
      "client_urls": [
        "http://demo-pd-0.demo-pd-peer.tidb.svc:2379"
      ]
    },
    {
      "name": "demo-pd-1",
      "member_id": 13812728483101200181,
      "peer_urls": [
        "http://demo-pd-1.demo-pd-peer.tidb.svc:2380"
      ],
      "client_urls": [
        "http://demo-pd-1.demo-pd-peer.tidb.svc:2379"
      ]
    }
  ],
  "leader": {
    "name": "demo-pd-2",
    "member_id": 3630075772193212308,
    "peer_urls": [
      "http://demo-pd-2.demo-pd-peer.tidb.svc:2380"
    ],
    "client_urls": [
      "http://demo-pd-2.demo-pd-peer.tidb.svc:2379"
    ]
  },
  "etcd_leader": {
    "name": "demo-pd-2",
    "member_id": 3630075772193212308,
    "peer_urls": [
      "http://demo-pd-2.demo-pd-peer.tidb.svc:2380"
    ],
    "client_urls": [
      "http://demo-pd-2.demo-pd-peer.tidb.svc:2379"
    ]
  }
}

When i deleted this no-name member, it works:

$ curl -XDELETE 10.103.241.52:2379/pd/api/v1/members/id/1429482795436666386
@weekface
Copy link
Contributor Author

pd start args:

/pd-server --data-dir=/var/lib/pd/data --name=demo-cluster-pd-3 --peer-urls=http://0.0.0.0:2380 --advertise-peer-urls=http://demo-cluster-pd-3.demo-cluster-pd-peer.tidb-cluster-e2e.svc:2380 --client-urls=http://0.0.0.0:2379 --advertise-client-urls=http://demo-cluster-pd-3.demo-cluster-pd-peer.tidb-cluster-e2e.svc:2379 --config=/etc/pd/pd.toml --join=http://demo-cluster-pd-0.demo-cluster-pd-peer.tidb-cluster-e2e.svc:2380,http://demo-cluster-pd-1.demo-cluster-pd-peer.tidb-cluster-e2e.svc:2380,http://demo-cluster-pd-2.demo-cluster-pd-peer.tidb-cluster-e2e.svc:2380

The new error messages:

2018/10/18 03:42:30.070 util.go:62: [info] Welcome to Placement Driver (PD).
2018/10/18 03:42:30.071 util.go:63: [info] Release Version: v2.0.5
2018/10/18 03:42:30.071 util.go:64: [info] Git Commit Hash: b64716707b7279a4ae822be767085ff17b5f3fea
2018/10/18 03:42:30.071 util.go:65: [info] Git Branch: release-2.0
2018/10/18 03:42:30.071 util.go:66: [info] UTC Build Time:  2018-07-06 10:27:51
2018/10/18 03:42:30.071 metricutil.go:83: [info] disable Prometheus push client
2018/10/18 03:42:30.090 server.go:96: [info] PD config - Config({FlagSet:0xc4201f0660 Version:false ClientUrls:http://0.0.0.0:2379 PeerUrls:http://0.0.0.0:2380 AdvertiseClientUrls:h
ttp://demo-cluster-pd-3.demo-cluster-pd-peer.tidb-cluster-e2e.svc:2379 AdvertisePeerUrls:http://demo-cluster-pd-3.demo-cluster-pd-peer.tidb-cluster-e2e.svc:2380 Name:demo-cluster-pd
-3 DataDir:/var/lib/pd/data InitialCluster:demo-cluster-pd-0=http://demo-cluster-pd-0.demo-cluster-pd-peer.tidb-cluster-e2e.svc:2380,demo-cluster-pd-2=http://demo-cluster-pd-2.demo-
cluster-pd-peer.tidb-cluster-e2e.svc:2380,demo-cluster-pd-1=http://demo-cluster-pd-1.demo-cluster-pd-peer.tidb-cluster-e2e.svc:2380,demo-cluster-pd-3=http://demo-cluster-pd-3.demo-c
luster-pd-peer.tidb-cluster-e2e.svc:2380 InitialClusterState:existing Join:http://demo-cluster-pd-0.demo-cluster-pd-peer.tidb-cluster-e2e.svc:2380,http://demo-cluster-pd-1.demo-clus
ter-pd-peer.tidb-cluster-e2e.svc:2380,http://demo-cluster-pd-2.demo-cluster-pd-peer.tidb-cluster-e2e.svc:2380 LeaderLease:3 Log:{Level:info Format:text DisableTimestamp:false File:{
Filename:/var/lib/pd/pd.log LogRotate:true MaxSize:3000 MaxDays:28 MaxBackups:7}} LogFileDeprecated: LogLevelDeprecated: TsoSaveInterval:3s Metric:{PushJob:demo-cluster-pd-3 PushAdd
ress: PushInterval:15s} Schedule:{MaxSnapshotCount:3 MaxPendingPeerCount:16 MaxMergeRegionSize:0 SplitMergeInterval:1h0m0s PatrolRegionInterval:100ms MaxStoreDownTime:1h0m0s LeaderS
cheduleLimit:4 RegionScheduleLimit:4 ReplicaScheduleLimit:8 MergeScheduleLimit:8 TolerantSizeRatio:5 LowSpaceRatio:0.8 HighSpaceRatio:0.6 EnableRaftLearner:false Schedulers:[{Type:b
alance-region Args:[] Disable:false} {Type:balance-leader Args:[] Disable:false} {Type:hot-region Args:[] Disable:false} {Type:label Args:[] Disable:false}]} Replication:{MaxReplica
s:3 LocationLabels:[zone rack host]} Namespace:map[] QuotaBackendBytes:0 AutoCompactionRetention:1 TickInterval:500ms ElectionInterval:3s Security:{CAPath: CertPath: KeyPath:} Label
Property:map[] configFile:/etc/pd/pd.toml WarningMsgs:[] NamespaceClassifier:default nextRetryDelay:1000000000 disableStrictReconfigCheck:false heartbeatStreamBindInterval:{Duration
:60000000000} leaderPriorityCheckInterval:{Duration:60000000000}})
2018/10/18 03:42:30.093 server.go:122: [info] start embed etcd
2018/10/18 03:42:30.093 log.go:86: [info] embed: [listening for peers on  http://0.0.0.0:2380]
2018/10/18 03:42:30.093 log.go:86: [info] embed: [pprof is enabled under /debug/pprof]
2018/10/18 03:42:30.093 log.go:86: [info] embed: [listening for client requests on  0.0.0.0:2379]
2018/10/18 03:42:30.093 systime_mon.go:24: [info] start system time monitor
2018/10/18 03:42:30.156 main.go:92: [fatal] run server failed: error validating peerURLs {ClusterID:e113a4608f4a1b94 Members:[&{ID:12c400c11fde50a1 RaftAttributes:{PeerURLs:[http://
demo-cluster-pd-0.demo-cluster-pd-peer.tidb-cluster-e2e.svc:2380]} Attributes:{Name:demo-cluster-pd-0 ClientURLs:[http://demo-cluster-pd-0.demo-cluster-pd-peer.tidb-cluster-e2e.svc:
2379]}} &{ID:1b330006834e69dd RaftAttributes:{PeerURLs:[http://demo-cluster-pd-2.demo-cluster-pd-peer.tidb-cluster-e2e.svc:2380]} Attributes:{Name:demo-cluster-pd-2 ClientURLs:[http
://demo-cluster-pd-2.demo-cluster-pd-peer.tidb-cluster-e2e.svc:2379]}} &{ID:1cefec7acc79d7dd RaftAttributes:{PeerURLs:[http://demo-cluster-pd-1.demo-cluster-pd-peer.tidb-cluster-e2e
.svc:2380]} Attributes:{Name:demo-cluster-pd-1 ClientURLs:[http://demo-cluster-pd-1.demo-cluster-pd-peer.tidb-cluster-e2e.svc:2379]}}] RemovedMemberIDs:[]}: member count is unequal
/home/jenkins/workspace/build_pd_2.0/go/src/github.com/pingcap/pd/server/server.go:125:
/home/jenkins/workspace/build_pd_2.0/go/src/github.com/pingcap/pd/server/server.go:276:
2018/10/18 03:42:32.585 util.go:62: [info] Welcome to Placement Driver (PD).
2018/10/18 03:42:32.585 util.go:63: [info] Release Version: v2.0.5
2018/10/18 03:42:32.585 util.go:64: [info] Git Commit Hash: b64716707b7279a4ae822be767085ff17b5f3fea
2018/10/18 03:42:32.585 util.go:65: [info] Git Branch: release-2.0
2018/10/18 03:42:32.585 util.go:66: [info] UTC Build Time:  2018-07-06 10:27:51
2018/10/18 03:42:32.585 metricutil.go:83: [info] disable Prometheus push client
2018/10/18 03:42:32.586 server.go:96: [info] PD config - Config({FlagSet:0xc4201f0660 Version:false ClientUrls:http://0.0.0.0:2379 PeerUrls:http://0.0.0.0:2380 AdvertiseClientUrls:h
ttp://demo-cluster-pd-3.demo-cluster-pd-peer.tidb-cluster-e2e.svc:2379 AdvertisePeerUrls:http://demo-cluster-pd-3.demo-cluster-pd-peer.tidb-cluster-e2e.svc:2380 Name:demo-cluster-pd
-3 DataDir:/var/lib/pd/data InitialCluster: InitialClusterState:existing Join:http://demo-cluster-pd-0.demo-cluster-pd-peer.tidb-cluster-e2e.svc:2380,http://demo-cluster-pd-1.demo-c
luster-pd-peer.tidb-cluster-e2e.svc:2380,http://demo-cluster-pd-2.demo-cluster-pd-peer.tidb-cluster-e2e.svc:2380 LeaderLease:3 Log:{Level:info Format:text DisableTimestamp:false Fil
e:{Filename:/var/lib/pd/pd.log LogRotate:true MaxSize:3000 MaxDays:28 MaxBackups:7}} LogFileDeprecated: LogLevelDeprecated: TsoSaveInterval:3s Metric:{PushJob:demo-cluster-pd-3 Push
Address: PushInterval:15s} Schedule:{MaxSnapshotCount:3 MaxPendingPeerCount:16 MaxMergeRegionSize:0 SplitMergeInterval:1h0m0s PatrolRegionInterval:100ms MaxStoreDownTime:1h0m0s Lead
erScheduleLimit:4 RegionScheduleLimit:4 ReplicaScheduleLimit:8 MergeScheduleLimit:8 TolerantSizeRatio:5 LowSpaceRatio:0.8 HighSpaceRatio:0.6 EnableRaftLearner:false Schedulers:[{Typ
e:balance-region Args:[] Disable:false} {Type:balance-leader Args:[] Disable:false} {Type:hot-region Args:[] Disable:false} {Type:label Args:[] Disable:false}]} Replication:{MaxRepl
icas:3 LocationLabels:[zone rack host]} Namespace:map[] QuotaBackendBytes:0 AutoCompactionRetention:1 TickInterval:500ms ElectionInterval:3s Security:{CAPath: CertPath: KeyPath:} La
belProperty:map[] configFile:/etc/pd/pd.toml WarningMsgs:[] NamespaceClassifier:default nextRetryDelay:1000000000 disableStrictReconfigCheck:false heartbeatStreamBindInterval:{Durat
ion:60000000000} leaderPriorityCheckInterval:{Duration:60000000000}})
2018/10/18 03:42:32.587 server.go:122: [info] start embed etcd
2018/10/18 03:42:32.588 log.go:86: [info] embed: [listening for peers on  http://0.0.0.0:2380]
2018/10/18 03:42:32.588 log.go:86: [info] embed: [pprof is enabled under /debug/pprof]
2018/10/18 03:42:32.588 log.go:86: [info] embed: [listening for client requests on  0.0.0.0:2379]
2018/10/18 03:42:32.588 systime_mon.go:24: [info] start system time monitor
2018/10/18 03:42:32.601 main.go:92: [fatal] run server failed: couldn't find local name "demo-cluster-pd-3" in the initial cluster configuration
/home/jenkins/workspace/build_pd_2.0/go/src/github.com/pingcap/pd/server/server.go:125:
/home/jenkins/workspace/build_pd_2.0/go/src/github.com/pingcap/pd/server/server.go:276:
2018/10/18 03:42:49.222 util.go:62: [info] Welcome to Placement Driver (PD).
2018/10/18 03:42:49.222 util.go:63: [info] Release Version: v2.0.5
2018/10/18 03:42:49.222 util.go:64: [info] Git Commit Hash: b64716707b7279a4ae822be767085ff17b5f3fea
2018/10/18 03:42:49.222 util.go:65: [info] Git Branch: release-2.0
2018/10/18 03:42:49.222 util.go:66: [info] UTC Build Time:  2018-07-06 10:27:51
2018/10/18 03:42:49.222 metricutil.go:83: [info] disable Prometheus push client
2018/10/18 03:42:49.223 server.go:96: [info] PD config - Config({FlagSet:0xc4201f0660 Version:false ClientUrls:http://0.0.0.0:2379 PeerUrls:http://0.0.0.0:2380 AdvertiseClientUrls:h
ttp://demo-cluster-pd-3.demo-cluster-pd-peer.tidb-cluster-e2e.svc:2379 AdvertisePeerUrls:http://demo-cluster-pd-3.demo-cluster-pd-peer.tidb-cluster-e2e.svc:2380 Name:demo-cluster-pd
-3 DataDir:/var/lib/pd/data InitialCluster: InitialClusterState:existing Join:http://demo-cluster-pd-0.demo-cluster-pd-peer.tidb-cluster-e2e.svc:2380,http://demo-cluster-pd-1.demo-c
luster-pd-peer.tidb-cluster-e2e.svc:2380,http://demo-cluster-pd-2.demo-cluster-pd-peer.tidb-cluster-e2e.svc:2380 LeaderLease:3 Log:{Level:info Format:text DisableTimestamp:false Fil
e:{Filename:/var/lib/pd/pd.log LogRotate:true MaxSize:3000 MaxDays:28 MaxBackups:7}} LogFileDeprecated: LogLevelDeprecated: TsoSaveInterval:3s Metric:{PushJob:demo-cluster-pd-3 Push
Address: PushInterval:15s} Schedule:{MaxSnapshotCount:3 MaxPendingPeerCount:16 MaxMergeRegionSize:0 SplitMergeInterval:1h0m0s PatrolRegionInterval:100ms MaxStoreDownTime:1h0m0s Lead
erScheduleLimit:4 RegionScheduleLimit:4 ReplicaScheduleLimit:8 MergeScheduleLimit:8 TolerantSizeRatio:5 LowSpaceRatio:0.8 HighSpaceRatio:0.6 EnableRaftLearner:false Schedulers:[{Typ
e:balance-region Args:[] Disable:false} {Type:balance-leader Args:[] Disable:false} {Type:hot-region Args:[] Disable:false} {Type:label Args:[] Disable:false}]} Replication:{MaxRepl
icas:3 LocationLabels:[zone rack host]} Namespace:map[] QuotaBackendBytes:0 AutoCompactionRetention:1 TickInterval:500ms ElectionInterval:3s Security:{CAPath: CertPath: KeyPath:} La
belProperty:map[] configFile:/etc/pd/pd.toml WarningMsgs:[] NamespaceClassifier:default nextRetryDelay:1000000000 disableStrictReconfigCheck:false heartbeatStreamBindInterval:{Durat
ion:60000000000} leaderPriorityCheckInterval:{Duration:60000000000}})
2018/10/18 03:42:49.225 server.go:122: [info] start embed etcd
2018/10/18 03:42:49.225 log.go:86: [info] embed: [listening for peers on  http://0.0.0.0:2380]
2018/10/18 03:42:49.225 log.go:86: [info] embed: [pprof is enabled under /debug/pprof]
2018/10/18 03:42:49.225 log.go:86: [info] embed: [listening for client requests on  0.0.0.0:2379]
2018/10/18 03:42:49.225 systime_mon.go:24: [info] start system time monitor
2018/10/18 03:42:49.228 main.go:92: [fatal] run server failed: couldn't find local name "demo-cluster-pd-3" in the initial cluster configuration
/home/jenkins/workspace/build_pd_2.0/go/src/github.com/pingcap/pd/server/server.go:125:
/home/jenkins/workspace/build_pd_2.0/go/src/github.com/pingcap/pd/server/server.go:276:
2018/10/18 03:43:15.231 util.go:62: [info] Welcome to Placement Driver (PD).
2018/10/18 03:43:15.232 util.go:63: [info] Release Version: v2.0.5
2018/10/18 03:43:15.232 util.go:64: [info] Git Commit Hash: b64716707b7279a4ae822be767085ff17b5f3fea
2018/10/18 03:43:15.232 util.go:65: [info] Git Branch: release-2.0
2018/10/18 03:43:15.232 util.go:66: [info] UTC Build Time:  2018-07-06 10:27:51
2018/10/18 03:43:15.232 metricutil.go:83: [info] disable Prometheus push client
2018/10/18 03:43:15.232 server.go:96: [info] PD config - Config({FlagSet:0xc4201f0660 Version:false ClientUrls:http://0.0.0.0:2379 PeerUrls:http://0.0.0.0:2380 AdvertiseClientUrls:h
ttp://demo-cluster-pd-3.demo-cluster-pd-peer.tidb-cluster-e2e.svc:2379 AdvertisePeerUrls:http://demo-cluster-pd-3.demo-cluster-pd-peer.tidb-cluster-e2e.svc:2380 Name:demo-cluster-pd
-3 DataDir:/var/lib/pd/data InitialCluster: InitialClusterState:existing Join:http://demo-cluster-pd-0.demo-cluster-pd-peer.tidb-cluster-e2e.svc:2380,http://demo-cluster-pd-1.demo-c
luster-pd-peer.tidb-cluster-e2e.svc:2380,http://demo-cluster-pd-2.demo-cluster-pd-peer.tidb-cluster-e2e.svc:2380 LeaderLease:3 Log:{Level:info Format:text DisableTimestamp:false Fil
e:{Filename:/var/lib/pd/pd.log LogRotate:true MaxSize:3000 MaxDays:28 MaxBackups:7}} LogFileDeprecated: LogLevelDeprecated: TsoSaveInterval:3s Metric:{PushJob:demo-cluster-pd-3 Push
Address: PushInterval:15s} Schedule:{MaxSnapshotCount:3 MaxPendingPeerCount:16 MaxMergeRegionSize:0 SplitMergeInterval:1h0m0s PatrolRegionInterval:100ms MaxStoreDownTime:1h0m0s Lead
erScheduleLimit:4 RegionScheduleLimit:4 ReplicaScheduleLimit:8 MergeScheduleLimit:8 TolerantSizeRatio:5 LowSpaceRatio:0.8 HighSpaceRatio:0.6 EnableRaftLearner:false Schedulers:[{Typ
e:balance-region Args:[] Disable:false} {Type:balance-leader Args:[] Disable:false} {Type:hot-region Args:[] Disable:false} {Type:label Args:[] Disable:false}]} Replication:{MaxRepl
icas:3 LocationLabels:[zone rack host]} Namespace:map[] QuotaBackendBytes:0 AutoCompactionRetention:1 TickInterval:500ms ElectionInterval:3s Security:{CAPath: CertPath: KeyPath:} La
belProperty:map[] configFile:/etc/pd/pd.toml WarningMsgs:[] NamespaceClassifier:default nextRetryDelay:1000000000 disableStrictReconfigCheck:false heartbeatStreamBindInterval:{Durat
ion:60000000000} leaderPriorityCheckInterval:{Duration:60000000000}})
2018/10/18 03:43:15.234 server.go:122: [info] start embed etcd
2018/10/18 03:43:15.234 log.go:86: [info] embed: [listening for peers on  http://0.0.0.0:2380]
2018/10/18 03:43:15.234 log.go:86: [info] embed: [pprof is enabled under /debug/pprof]
2018/10/18 03:43:15.234 log.go:86: [info] embed: [listening for client requests on  0.0.0.0:2379]
2018/10/18 03:43:15.234 systime_mon.go:24: [info] start system time monitor
2018/10/18 03:43:15.239 main.go:92: [fatal] run server failed: couldn't find local name "demo-cluster-pd-3" in the initial cluster configuration
/home/jenkins/workspace/build_pd_2.0/go/src/github.com/pingcap/pd/server/server.go:125:
/home/jenkins/workspace/build_pd_2.0/go/src/github.com/pingcap/pd/server/server.go:276:
2018/10/18 03:44:11.269 util.go:62: [info] Welcome to Placement Driver (PD).
2018/10/18 03:44:11.269 util.go:63: [info] Release Version: v2.0.5
2018/10/18 03:44:11.269 util.go:64: [info] Git Commit Hash: b64716707b7279a4ae822be767085ff17b5f3fea
2018/10/18 03:44:11.269 util.go:65: [info] Git Branch: release-2.0
2018/10/18 03:44:11.269 util.go:66: [info] UTC Build Time:  2018-07-06 10:27:51
2018/10/18 03:44:11.269 metricutil.go:83: [info] disable Prometheus push client
2018/10/18 03:44:11.269 server.go:96: [info] PD config - Config({FlagSet:0xc4201f0660 Version:false ClientUrls:http://0.0.0.0:2379 PeerUrls:http://0.0.0.0:2380 AdvertiseClientUrls:h
ttp://demo-cluster-pd-3.demo-cluster-pd-peer.tidb-cluster-e2e.svc:2379 AdvertisePeerUrls:http://demo-cluster-pd-3.demo-cluster-pd-peer.tidb-cluster-e2e.svc:2380 Name:demo-cluster-pd
-3 DataDir:/var/lib/pd/data InitialCluster: InitialClusterState:existing Join:http://demo-cluster-pd-0.demo-cluster-pd-peer.tidb-cluster-e2e.svc:2380,http://demo-cluster-pd-1.demo-c
luster-pd-peer.tidb-cluster-e2e.svc:2380,http://demo-cluster-pd-2.demo-cluster-pd-peer.tidb-cluster-e2e.svc:2380 LeaderLease:3 Log:{Level:info Format:text DisableTimestamp:false Fil
e:{Filename:/var/lib/pd/pd.log LogRotate:true MaxSize:3000 MaxDays:28 MaxBackups:7}} LogFileDeprecated: LogLevelDeprecated: TsoSaveInterval:3s Metric:{PushJob:demo-cluster-pd-3 Push
Address: PushInterval:15s} Schedule:{MaxSnapshotCount:3 MaxPendingPeerCount:16 MaxMergeRegionSize:0 SplitMergeInterval:1h0m0s PatrolRegionInterval:100ms MaxStoreDownTime:1h0m0s Lead
erScheduleLimit:4 RegionScheduleLimit:4 ReplicaScheduleLimit:8 MergeScheduleLimit:8 TolerantSizeRatio:5 LowSpaceRatio:0.8 HighSpaceRatio:0.6 EnableRaftLearner:false Schedulers:[{Typ
e:balance-region Args:[] Disable:false} {Type:balance-leader Args:[] Disable:false} {Type:hot-region Args:[] Disable:false} {Type:label Args:[] Disable:false}]} Replication:{MaxRepl
icas:3 LocationLabels:[zone rack host]} Namespace:map[] QuotaBackendBytes:0 AutoCompactionRetention:1 TickInterval:500ms ElectionInterval:3s Security:{CAPath: CertPath: KeyPath:} La
belProperty:map[] configFile:/etc/pd/pd.toml WarningMsgs:[] NamespaceClassifier:default nextRetryDelay:1000000000 disableStrictReconfigCheck:false heartbeatStreamBindInterval:{Durat
ion:60000000000} leaderPriorityCheckInterval:{Duration:60000000000}})
2018/10/18 03:44:11.271 server.go:122: [info] start embed etcd
2018/10/18 03:44:11.271 log.go:86: [info] embed: [listening for peers on  http://0.0.0.0:2380]
2018/10/18 03:44:11.271 log.go:86: [info] embed: [pprof is enabled under /debug/pprof]
2018/10/18 03:44:11.271 log.go:86: [info] embed: [listening for client requests on  0.0.0.0:2379]
2018/10/18 03:44:11.272 systime_mon.go:24: [info] start system time monitor
2018/10/18 03:44:11.275 main.go:92: [fatal] run server failed: couldn't find local name "demo-cluster-pd-3" in the initial cluster configuration
/home/jenkins/workspace/build_pd_2.0/go/src/github.com/pingcap/pd/server/server.go:125:
/home/jenkins/workspace/build_pd_2.0/go/src/github.com/pingcap/pd/server/server.go:276:

@nolouch
Copy link
Member

nolouch commented Oct 18, 2018 •
edited
Loading

i deleted this no-name member, it works:

$ curl -XDELETE 10.103.241.52:2379/pd/api/v1/members/id/1429482795436666386

can you show me the log after you delete it.

@weekface
Copy link
Contributor Author

When i deleted this no-name member, other pd's logs:

2018/10/18 08:41:11.246 log.go:86: [info] etcdserver/membership: [removed member 312420d38578a47f from cluster e113a4608f4a1b94]
2018/10/18 08:41:11.246 log.go:86: [info] rafthttp: [stopping peer 312420d38578a47f...]
2018/10/18 08:41:11.246 log.go:86: [info] rafthttp: [stopped streaming with peer 312420d38578a47f (writer)]
2018/10/18 08:41:11.246 log.go:86: [info] rafthttp: [stopped streaming with peer 312420d38578a47f (writer)]
2018/10/18 08:41:11.246 log.go:86: [info] rafthttp: [stopped HTTP pipelining with peer 312420d38578a47f]
2018/10/18 08:41:11.247 log.go:86: [info] rafthttp: [stopped streaming with peer 312420d38578a47f (stream MsgApp v2 reader)]
2018/10/18 08:41:11.247 log.go:86: [info] rafthttp: [stopped streaming with peer 312420d38578a47f (stream Message reader)]
2018/10/18 08:41:11.247 log.go:86: [info] rafthttp: [stopped peer 312420d38578a47f]
2018/10/18 08:41:11.247 log.go:86: [info] rafthttp: [removed peer 312420d38578a47f]

@nolouch nolouch mentioned this issue Oct 18, 2018
Merged
@weekface
Copy link
Contributor Author 

2018/10/22 13:21:49.368 util.go:62: [info] Welcome to Placement Driver (PD).
2018/10/22 13:21:49.369 util.go:63: [info] Release Version: v2.0.5-dirty
2018/10/22 13:21:49.369 util.go:64: [info] Git Commit Hash: b64716707b7279a4ae822be767085ff17b5f3fea
2018/10/22 13:21:49.369 util.go:65: [info] Git Branch: HEAD
2018/10/22 13:21:49.369 util.go:66: [info] UTC Build Time:  2018-10-19 07:31:50
2018/10/22 13:21:49.369 metricutil.go:83: [info] disable Prometheus push client
2018/10/22 13:16:02.292 util.go:62: [info] Welcome to Placement Driver (PD).
2018/10/22 13:16:02.293 util.go:63: [info] Release Version: v2.0.5-dirty
2018/10/22 13:16:02.293 util.go:64: [info] Git Commit Hash: b64716707b7279a4ae822be767085ff17b5f3fea
2018/10/22 13:16:02.293 util.go:65: [info] Git Branch: HEAD
2018/10/22 13:16:02.293 util.go:66: [info] UTC Build Time:  2018-10-19 07:31:50
2018/10/22 13:16:02.293 metricutil.go:83: [info] disable Prometheus push client
2018/10/22 13:16:02.347 server.go:96: [info] PD config - Config({FlagSet:0xc4201c8660 Version:false ClientUrls:http://0.0.0.0:2379 PeerUrls:http://0.0.0.0:2380 AdvertiseClientUrls:h
ttp://demo-cluster-pd-3.demo-cluster-pd-peer.tidb-cluster-e2e.svc:2379 AdvertisePeerUrls:http://demo-cluster-pd-3.demo-cluster-pd-peer.tidb-cluster-e2e.svc:2380 Name:demo-cluster-pd
-3 DataDir:/var/lib/pd/data InitialCluster:demo-cluster-pd-0=http://demo-cluster-pd-0.demo-cluster-pd-peer.tidb-cluster-e2e.svc:2380,demo-cluster-pd-1=http://demo-cluster-pd-1.demo-
cluster-pd-peer.tidb-cluster-e2e.svc:2380,demo-cluster-pd-2=http://demo-cluster-pd-2.demo-cluster-pd-peer.tidb-cluster-e2e.svc:2380,demo-cluster-pd-3=http://demo-cluster-pd-3.demo-c
luster-pd-peer.tidb-cluster-e2e.svc:2380 InitialClusterState:existing Join:http://demo-cluster-pd-0.demo-cluster-pd-peer.tidb-cluster-e2e.svc:2380,http://demo-cluster-pd-1.demo-clus
ter-pd-peer.tidb-cluster-e2e.svc:2380,http://demo-cluster-pd-2.demo-cluster-pd-peer.tidb-cluster-e2e.svc:2380 LeaderLease:3 Log:{Level:info Format:text DisableTimestamp:false File:{
Filename:/var/lib/pd/pd.log LogRotate:true MaxSize:3000 MaxDays:28 MaxBackups:7}} LogFileDeprecated: LogLevelDeprecated: TsoSaveInterval:3s Metric:{PushJob:demo-cluster-pd-3 PushAdd
ress: PushInterval:15s} Schedule:{MaxSnapshotCount:3 MaxPendingPeerCount:16 MaxMergeRegionSize:0 SplitMergeInterval:1h0m0s PatrolRegionInterval:100ms MaxStoreDownTime:1h0m0s LeaderS
cheduleLimit:4 RegionScheduleLimit:4 ReplicaScheduleLimit:8 MergeScheduleLimit:8 TolerantSizeRatio:5 LowSpaceRatio:0.8 HighSpaceRatio:0.6 EnableRaftLearner:false Schedulers:[{Type:b
alance-region Args:[] Disable:false} {Type:balance-leader Args:[] Disable:false} {Type:hot-region Args:[] Disable:false} {Type:label Args:[] Disable:false}]} Replication:{MaxReplica
s:3 LocationLabels:[zone rack host]} Namespace:map[] QuotaBackendBytes:0 AutoCompactionRetention:1 TickInterval:500ms ElectionInterval:3s Security:{CAPath: CertPath: KeyPath:} Label
Property:map[] configFile:/etc/pd/pd.toml WarningMsgs:[] NamespaceClassifier:default nextRetryDelay:1000000000 disableStrictReconfigCheck:false heartbeatStreamBindInterval:{Duration
:60000000000} leaderPriorityCheckInterval:{Duration:60000000000}})
2018/10/22 13:16:02.349 server.go:122: [info] start embed etcd
2018/10/22 13:16:02.350 log.go:86: [info] embed: [listening for peers on  http://0.0.0.0:2380]
2018/10/22 13:16:02.350 log.go:86: [info] embed: [pprof is enabled under /debug/pprof]
2018/10/22 13:16:02.350 log.go:86: [info] embed: [listening for client requests on  0.0.0.0:2379]
2018/10/22 13:16:02.350 systime_mon.go:24: [info] start system time monitor
2018/10/22 13:16:02.398 main.go:92: [fatal] run server failed: error validating peerURLs {ClusterID:e113a4608f4a1b94 Members:[&{ID:20b33ad53624ae8f RaftAttributes:{PeerURLs:[http://
demo-cluster-pd-1.demo-cluster-pd-peer.tidb-cluster-e2e.svc:2380]} Attributes:{Name:demo-cluster-pd-1 ClientURLs:[http://demo-cluster-pd-1.demo-cluster-pd-peer.tidb-cluster-e2e.svc:
2379]}} &{ID:a79afaba07c7dc07 RaftAttributes:{PeerURLs:[http://demo-cluster-pd-2.demo-cluster-pd-peer.tidb-cluster-e2e.svc:2380]} Attributes:{Name:demo-cluster-pd-2 ClientURLs:[http
://demo-cluster-pd-2.demo-cluster-pd-peer.tidb-cluster-e2e.svc:2379]}} &{ID:dcd20634bc4bff12 RaftAttributes:{PeerURLs:[http://demo-cluster-pd-3.demo-cluster-pd-peer.tidb-cluster-e2e
.svc:2380]} Attributes:{Name: ClientURLs:[]}} &{ID:12c400c11fde50a1 RaftAttributes:{PeerURLs:[http://demo-cluster-pd-0.demo-cluster-pd-peer.tidb-cluster-e2e.svc:2380]} Attributes:{N
ame:demo-cluster-pd-0 ClientURLs:[http://demo-cluster-pd-0.demo-cluster-pd-peer.tidb-cluster-e2e.svc:2379]}} &{ID:185f4d044cc002c2 RaftAttributes:{PeerURLs:[http://demo-cluster-pd-4
.demo-cluster-pd-peer.tidb-cluster-e2e.svc:2380]} Attributes:{Name: ClientURLs:[]}}] RemovedMemberIDs:[]}: member count is unequal
/local/go/src/github.com/pingcap/pd/server/server.go:125:
/local/go/src/github.com/pingcap/pd/server/server.go:276:

$ cat join
demo-cluster-pd-0=http://demo-cluster-pd-0.demo-cluster-pd-peer.tidb-cluster-e2e.svc:2380,demo-cluster-pd-1=http://demo-cluster-pd-1.demo-cluster-pd-peer.tidb-cluster-e2e.svc:2380,demo-cluster-pd-2=http://demo-cluster-pd-2.demo-cluster-pd-peer.tidb-cluster-e2e.svc:2380,demo-cluster-pd-3=http://demo-cluster-pd-3.demo-cluster-pd-peer.tidb-cluster-e2e.svc:2380

@weekface
Copy link
Contributor Author

demo-cluster-pd-4:

2018/10/22 13:16:02.306 util.go:62: [info] Welcome to Placement Driver (PD).
2018/10/22 13:16:02.307 util.go:63: [info] Release Version: v2.0.5-dirty
2018/10/22 13:16:02.307 util.go:64: [info] Git Commit Hash: b64716707b7279a4ae822be767085ff17b5f3fea
2018/10/22 13:16:02.307 util.go:65: [info] Git Branch: HEAD
2018/10/22 13:16:02.307 util.go:66: [info] UTC Build Time:  2018-10-19 07:31:50
2018/10/22 13:16:02.307 metricutil.go:83: [info] disable Prometheus push client
2018/10/22 13:16:02.392 server.go:96: [info] PD config - Config({FlagSet:0xc4201c8660 Version:false ClientUrls:http://0.0.0.0:2379 PeerUrls:http://0.0.0.0:2380 AdvertiseClientUrls:h
ttp://demo-cluster-pd-4.demo-cluster-pd-peer.tidb-cluster-e2e.svc:2379 AdvertisePeerUrls:http://demo-cluster-pd-4.demo-cluster-pd-peer.tidb-cluster-e2e.svc:2380 Name:demo-cluster-pd
-4 DataDir:/var/lib/pd/data InitialCluster:demo-cluster-pd-0=http://demo-cluster-pd-0.demo-cluster-pd-peer.tidb-cluster-e2e.svc:2380,demo-cluster-pd-4=http://demo-cluster-pd-4.demo-
cluster-pd-peer.tidb-cluster-e2e.svc:2380,demo-cluster-pd-1=http://demo-cluster-pd-1.demo-cluster-pd-peer.tidb-cluster-e2e.svc:2380,demo-cluster-pd-2=http://demo-cluster-pd-2.demo-c
luster-pd-peer.tidb-cluster-e2e.svc:2380,=http://demo-cluster-pd-3.demo-cluster-pd-peer.tidb-cluster-e2e.svc:2380 InitialClusterState:existing Join:http://demo-cluster-pd-0.demo-clu
ster-pd-peer.tidb-cluster-e2e.svc:2380,http://demo-cluster-pd-1.demo-cluster-pd-peer.tidb-cluster-e2e.svc:2380,http://demo-cluster-pd-2.demo-cluster-pd-peer.tidb-cluster-e2e.svc:238
0,http://demo-cluster-pd-3.demo-cluster-pd-peer.tidb-cluster-e2e.svc:2380 LeaderLease:3 Log:{Level:info Format:text DisableTimestamp:false File:{Filename:/var/lib/pd/pd.log LogRotat
e:true MaxSize:3000 MaxDays:28 MaxBackups:7}} LogFileDeprecated: LogLevelDeprecated: TsoSaveInterval:3s Metric:{PushJob:demo-cluster-pd-4 PushAddress: PushInterval:15s} Schedule:{Ma
xSnapshotCount:3 MaxPendingPeerCount:16 MaxMergeRegionSize:0 SplitMergeInterval:1h0m0s PatrolRegionInterval:100ms MaxStoreDownTime:1h0m0s LeaderScheduleLimit:4 RegionScheduleLimit:4
 ReplicaScheduleLimit:8 MergeScheduleLimit:8 TolerantSizeRatio:5 LowSpaceRatio:0.8 HighSpaceRatio:0.6 EnableRaftLearner:false Schedulers:[{Type:balance-region Args:[] Disable:false}
 {Type:balance-leader Args:[] Disable:false} {Type:hot-region Args:[] Disable:false} {Type:label Args:[] Disable:false}]} Replication:{MaxReplicas:3 LocationLabels:[zone rack host]}
 Namespace:map[] QuotaBackendBytes:0 AutoCompactionRetention:1 TickInterval:500ms ElectionInterval:3s Security:{CAPath: CertPath: KeyPath:} LabelProperty:map[] configFile:/etc/pd/pd
.toml WarningMsgs:[] NamespaceClassifier:default nextRetryDelay:1000000000 disableStrictReconfigCheck:false heartbeatStreamBindInterval:{Duration:60000000000} leaderPriorityCheckInt
erval:{Duration:60000000000}})
2018/10/22 13:16:02.395 server.go:122: [info] start embed etcd
2018/10/22 13:16:02.395 log.go:86: [info] embed: [listening for peers on  http://0.0.0.0:2380]
2018/10/22 13:16:02.395 log.go:86: [info] embed: [pprof is enabled under /debug/pprof]
2018/10/22 13:16:02.395 log.go:86: [info] embed: [listening for client requests on  0.0.0.0:2379]
2018/10/22 13:16:02.396 systime_mon.go:24: [info] start system time monitor
2018/10/22 13:16:02.409 log.go:86: [info] pkg/netutil: [resolving demo-cluster-pd-0.demo-cluster-pd-peer.tidb-cluster-e2e.svc:2380 to 10.60.0.189:2380]
2018/10/22 13:16:02.412 log.go:86: [info] pkg/netutil: [resolving demo-cluster-pd-0.demo-cluster-pd-peer.tidb-cluster-e2e.svc:2380 to 10.60.0.189:2380]
2018/10/22 13:16:02.420 log.go:86: [info] pkg/netutil: [resolving demo-cluster-pd-1.demo-cluster-pd-peer.tidb-cluster-e2e.svc:2380 to 10.60.1.147:2380]
2018/10/22 13:16:02.422 log.go:86: [info] pkg/netutil: [resolving demo-cluster-pd-1.demo-cluster-pd-peer.tidb-cluster-e2e.svc:2380 to 10.60.1.147:2380]
2018/10/22 13:16:02.424 log.go:86: [info] pkg/netutil: [resolving demo-cluster-pd-2.demo-cluster-pd-peer.tidb-cluster-e2e.svc:2380 to 10.60.3.45:2380]
2018/10/22 13:16:02.428 log.go:86: [info] pkg/netutil: [resolving demo-cluster-pd-2.demo-cluster-pd-peer.tidb-cluster-e2e.svc:2380 to 10.60.3.45:2380]
2018/10/22 13:16:02.431 log.go:86: [info] pkg/netutil: [resolving demo-cluster-pd-3.demo-cluster-pd-peer.tidb-cluster-e2e.svc:2380 to 10.60.2.215:2380]
2018/10/22 13:16:02.440 log.go:86: [info] pkg/netutil: [resolving demo-cluster-pd-3.demo-cluster-pd-peer.tidb-cluster-e2e.svc:2380 to 10.60.2.215:2380]
2018/10/22 13:16:02.442 log.go:86: [info] pkg/netutil: [resolving demo-cluster-pd-4.demo-cluster-pd-peer.tidb-cluster-e2e.svc:2380 to 10.60.1.149:2380]
2018/10/22 13:16:02.444 log.go:86: [info] pkg/netutil: [resolving demo-cluster-pd-4.demo-cluster-pd-peer.tidb-cluster-e2e.svc:2380 to 10.60.1.149:2380]
2018/10/22 13:16:02.453 log.go:82: [warning] etcdserver: [failed to reach the peerURL(http://demo-cluster-pd-3.demo-cluster-pd-peer.tidb-cluster-e2e.svc:2380) of member dcd20634bc4b
ff12 (Get http://demo-cluster-pd-3.demo-cluster-pd-peer.tidb-cluster-e2e.svc:2380/version: dial tcp 10.60.2.215:2380: getsockopt: connection refused)]
2018/10/22 13:16:02.453 log.go:82: [warning] etcdserver: [cannot get the version of member dcd20634bc4bff12 (Get http://demo-cluster-pd-3.demo-cluster-pd-peer.tidb-cluster-e2e.svc:2
380/version: dial tcp 10.60.2.215:2380: getsockopt: connection refused)]
2018/10/22 13:16:02.453 log.go:86: [info] etcdserver: [name = demo-cluster-pd-4]
2018/10/22 13:16:02.453 log.go:86: [info] etcdserver: [data dir = /var/lib/pd/data]
2018/10/22 13:16:02.453 log.go:86: [info] etcdserver: [member dir = /var/lib/pd/data/member]
2018/10/22 13:16:02.453 log.go:86: [info] etcdserver: [heartbeat = 500ms]
2018/10/22 13:16:02.453 log.go:86: [info] etcdserver: [election = 3000ms]

demo-cluster-pd-3:

2018/10/22 13:16:02.292 util.go:62: [info] Welcome to Placement Driver (PD).
2018/10/22 13:16:02.293 util.go:63: [info] Release Version: v2.0.5-dirty
2018/10/22 13:16:02.293 util.go:64: [info] Git Commit Hash: b64716707b7279a4ae822be767085ff17b5f3fea
2018/10/22 13:16:02.293 util.go:65: [info] Git Branch: HEAD
2018/10/22 13:16:02.293 util.go:66: [info] UTC Build Time:  2018-10-19 07:31:50
2018/10/22 13:16:02.293 metricutil.go:83: [info] disable Prometheus push client
2018/10/22 13:16:02.347 server.go:96: [info] PD config - Config({FlagSet:0xc4201c8660 Version:false ClientUrls:http://0.0.0.0:2379 PeerUrls:http://0.0.0.0:2380 AdvertiseClientUrls:h
ttp://demo-cluster-pd-3.demo-cluster-pd-peer.tidb-cluster-e2e.svc:2379 AdvertisePeerUrls:http://demo-cluster-pd-3.demo-cluster-pd-peer.tidb-cluster-e2e.svc:2380 Name:demo-cluster-pd
-3 DataDir:/var/lib/pd/data InitialCluster:demo-cluster-pd-0=http://demo-cluster-pd-0.demo-cluster-pd-peer.tidb-cluster-e2e.svc:2380,demo-cluster-pd-1=http://demo-cluster-pd-1.demo-
cluster-pd-peer.tidb-cluster-e2e.svc:2380,demo-cluster-pd-2=http://demo-cluster-pd-2.demo-cluster-pd-peer.tidb-cluster-e2e.svc:2380,demo-cluster-pd-3=http://demo-cluster-pd-3.demo-c
luster-pd-peer.tidb-cluster-e2e.svc:2380 InitialClusterState:existing Join:http://demo-cluster-pd-0.demo-cluster-pd-peer.tidb-cluster-e2e.svc:2380,http://demo-cluster-pd-1.demo-clus
ter-pd-peer.tidb-cluster-e2e.svc:2380,http://demo-cluster-pd-2.demo-cluster-pd-peer.tidb-cluster-e2e.svc:2380 LeaderLease:3 Log:{Level:info Format:text DisableTimestamp:false File:{
Filename:/var/lib/pd/pd.log LogRotate:true MaxSize:3000 MaxDays:28 MaxBackups:7}} LogFileDeprecated: LogLevelDeprecated: TsoSaveInterval:3s Metric:{PushJob:demo-cluster-pd-3 PushAdd
ress: PushInterval:15s} Schedule:{MaxSnapshotCount:3 MaxPendingPeerCount:16 MaxMergeRegionSize:0 SplitMergeInterval:1h0m0s PatrolRegionInterval:100ms MaxStoreDownTime:1h0m0s LeaderS
cheduleLimit:4 RegionScheduleLimit:4 ReplicaScheduleLimit:8 MergeScheduleLimit:8 TolerantSizeRatio:5 LowSpaceRatio:0.8 HighSpaceRatio:0.6 EnableRaftLearner:false Schedulers:[{Type:b
alance-region Args:[] Disable:false} {Type:balance-leader Args:[] Disable:false} {Type:hot-region Args:[] Disable:false} {Type:label Args:[] Disable:false}]} Replication:{MaxReplica
s:3 LocationLabels:[zone rack host]} Namespace:map[] QuotaBackendBytes:0 AutoCompactionRetention:1 TickInterval:500ms ElectionInterval:3s Security:{CAPath: CertPath: KeyPath:} Label
Property:map[] configFile:/etc/pd/pd.toml WarningMsgs:[] NamespaceClassifier:default nextRetryDelay:1000000000 disableStrictReconfigCheck:false heartbeatStreamBindInterval:{Duration
:60000000000} leaderPriorityCheckInterval:{Duration:60000000000}})
2018/10/22 13:16:02.349 server.go:122: [info] start embed etcd
2018/10/22 13:16:02.350 log.go:86: [info] embed: [listening for peers on  http://0.0.0.0:2380]
2018/10/22 13:16:02.350 log.go:86: [info] embed: [pprof is enabled under /debug/pprof]
2018/10/22 13:16:02.350 log.go:86: [info] embed: [listening for client requests on  0.0.0.0:2379]
2018/10/22 13:16:02.350 systime_mon.go:24: [info] start system time monitor
2018/10/22 13:16:02.398 main.go:92: [fatal] run server failed: error validating peerURLs {ClusterID:e113a4608f4a1b94 Members:[&{ID:20b33ad53624ae8f RaftAttributes:{PeerURLs:[http://
demo-cluster-pd-1.demo-cluster-pd-peer.tidb-cluster-e2e.svc:2380]} Attributes:{Name:demo-cluster-pd-1 ClientURLs:[http://demo-cluster-pd-1.demo-cluster-pd-peer.tidb-cluster-e2e.svc:
2379]}} &{ID:a79afaba07c7dc07 RaftAttributes:{PeerURLs:[http://demo-cluster-pd-2.demo-cluster-pd-peer.tidb-cluster-e2e.svc:2380]} Attributes:{Name:demo-cluster-pd-2 ClientURLs:[http
://demo-cluster-pd-2.demo-cluster-pd-peer.tidb-cluster-e2e.svc:2379]}} &{ID:dcd20634bc4bff12 RaftAttributes:{PeerURLs:[http://demo-cluster-pd-3.demo-cluster-pd-peer.tidb-cluster-e2e
.svc:2380]} Attributes:{Name: ClientURLs:[]}} &{ID:12c400c11fde50a1 RaftAttributes:{PeerURLs:[http://demo-cluster-pd-0.demo-cluster-pd-peer.tidb-cluster-e2e.svc:2380]} Attributes:{N
ame:demo-cluster-pd-0 ClientURLs:[http://demo-cluster-pd-0.demo-cluster-pd-peer.tidb-cluster-e2e.svc:2379]}} &{ID:185f4d044cc002c2 RaftAttributes:{PeerURLs:[http://demo-cluster-pd-4
.demo-cluster-pd-peer.tidb-cluster-e2e.svc:2380]} Attributes:{Name: ClientURLs:[]}}] RemovedMemberIDs:[]}: member count is unequal
/local/go/src/github.com/pingcap/pd/server/server.go:125:
/local/go/src/github.com/pingcap/pd/server/server.go:276:

pd-4 started before pd-3

@weekface weekface mentioned this issue Oct 23, 2018
Merged
@weekface
Copy link
Contributor Author

weekface commented Oct 29, 2018 •
edited
Loading

#142 and tikv/pd#1279 fixes this issue.

@tennix tennix mentioned this issue Nov 9, 2018
Closed
@tennix tennix added the type/bug Something isn't working label Jan 29, 2019
@weekface weekface mentioned this issue Jun 12, 2019
Closed
fgksgf pushed a commit to fgksgf/tidb-operator that referenced this issue Dec 23, 2024
@csuzhangxc
feat: readiness probe support for PD, TiKV and TiDB (pingcap#126)
13c496c
@anish-db anish-db mentioned this issue Jan 23, 2025
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
type/bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@weekface @nolouch @tennix