FailedScheduling: nodes didn't match node selector #516

jaltabike · 2019-05-24T08:00:14Z

Bug Report

What version of Kubernetes are you using?

Client Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.0", GitCommit:"ddf47ac13c1a9483ea035a79cd7c10005ff21a6d", GitTreeState:"clean", BuildDate:"2018-12-03T21:04:45Z", GoVersion:"go1.11.2", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.5", GitCommit:"51dd616cdd25d6ee22c83a858773b607328a18ec", GitTreeState:"clean", BuildDate:"2019-01-16T18:14:49Z", GoVersion:"go1.10.7", Compiler:"gc", Platform:"linux/amd64"}

What version of TiDB Operator are you using?

TiDB Operator Version: version.Info{TiDBVersion:"2.1.0", GitVersion:"v1.0.0-beta.2", GitCommit:"bc913fc1459118c1d972be37e100d3a70a9e981a", GitTreeState:"clean", BuildDate:"2019-05-10T10:15:56Z", GoVersion:"go1.12", Compiler:"gc", Platform:"linux/amd64"}

What storage classes exist in the Kubernetes cluster and what are used for PD/TiKV pods?

NAME PROVISIONER AGE
local-storage kubernetes.io/no-provisioner 5m40s

What's the status of the TiDB cluster pods?

NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE
demo-discovery-58fb7c6765-wwgbx 1/1 Running 0 3m4s 10.244.1.5 kube-node-3
demo-monitor-5ddb4bf8c-9xv96 2/2 Running 0 3m4s 10.244.2.4 kube-node-1
demo-pd-0 0/1 Pending 0 3m4s
demo-pd-1 0/1 Pending 0 3m4s
demo-pd-2 0/1 Pending 0 3m4s

What did you do?

Following the guide (https://github.com/pingcap/tidb-operator/blob/master/docs/local-dind-tutorial.md) to deploy a TiDB cluster in the DinD Kubernetes cluster.

What did you expect to see?
Success to deploy a TiDB cluster in the DinD Kubernetes cluster

What did you see instead?
pd's status stucks in pending.
When I run "kubectl describe po -n ", I see the following warning:
Warning FailedScheduling 2m45s (x2510 over 7m45s) tidb-scheduler 0/4 nodes are available: 4 node(s) didn't match node selector.

tennix · 2019-05-24T08:15:17Z

@jaltabike Could you show if there are available PVs by kubectl get pv? Also is your local-volume-provisioner running? kubectl get po -n kube-system -lapp=local-volume-provisioner

jaltabike · 2019-05-24T08:49:10Z

@tennix
CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
RWO Delete Available local-storage 52m
RWO Delete Available local-storage 52m
RWO Delete Available local-storage 52m
RWO Delete Available local-storage 52m
RWO Delete Available local-storage 52m
RWO Delete Available local-storage 52m
RWO Delete Available local-storage 52m
RWO Delete Available local-storage 52m
RWO Delete Available local-storage 52m
RWO Delete Available local-storage 52m
RWO Delete Available local-storage 52m
RWO Delete Available local-storage 52m
RWO Delete Available local-storage 52m
RWO Delete Available local-storage 52m
RWO Delete Available local-storage 52m
RWO Delete Available local-storage 52m
RWO Delete Available local-storage 52m
RWO Delete Available local-storage 52m
RWO Delete Available local-storage 52m
RWO Delete Available local-storage 52m
RWO Delete Available local-storage 52m
RWO Delete Available local-storage 52m
RWO Delete Available local-storage 52m
RWO Delete Available local-storage 52m
RWO Delete Available local-storage 52m
RWO Delete Available local-storage 52m
RWO Delete Available local-storage 52m
RWO Delete Available local-storage 52m
RWO Delete Available local-storage 52m
RWO Delete Available local-storage 52m

$ kubectl get po -n kube-system -lapp=local-volume-provisioner
NAME READY STATUS RESTARTS AGE
local-volume-provisioner-b24zr 1/1 Running 0 54m
local-volume-provisioner-g52cx 1/1 Running 0 54m
local-volume-provisioner-mxsn6 1/1 Running 0 53m

weekface · 2019-05-24T09:00:18Z

Please view the TidbCluster object information by:

kubectl get tc -n tidb -oyaml

@jlerche

jaltabike · 2019-05-24T10:41:01Z

@weekface

$ kubectl get tc -n tidb -oyaml
apiVersion: v1
items:
- apiVersion: pingcap.com/v1alpha1
  kind: TidbCluster
  metadata:
    creationTimestamp: "2019-05-24T07:48:47Z"
    generation: 1
    labels:
      app.kubernetes.io/component: tidb-cluster
      app.kubernetes.io/instance: demo
      app.kubernetes.io/managed-by: Tiller
      app.kubernetes.io/name: tidb-cluster
      helm.sh/chart: tidb-cluster-dev
    name: demo
    namespace: tidb
    resourceVersion: "1474"
    selfLink: /apis/pingcap.com/v1alpha1/namespaces/tidb/tidbclusters/demo
    uid: 5c9eb891-7df8-11e9-8409-0242d7093bb5
  spec:
    pd:
      image: pingcap/pd:v2.1.8
      imagePullPolicy: IfNotPresent
      limits: {}
      replicas: 3
      requests:
        storage: 1Gi
      storageClassName: local-storage
    pvReclaimPolicy: Retain
    schedulerName: tidb-scheduler
    services:
    - name: pd
      type: ClusterIP
    tidb:
      image: pingcap/tidb:v2.1.8
      imagePullPolicy: IfNotPresent
      limits: {}
      maxFailoverCount: 3
      replicas: 2
      requests: {}
      slowLogTailer:
        image: busybox:1.26.2
        imagePullPolicy: IfNotPresent
        limits:
          cpu: 100m
          memory: 50Mi
        requests:
          cpu: 20m
          memory: 5Mi
    tikv:
      image: pingcap/tikv:v2.1.8
      imagePullPolicy: IfNotPresent
      limits: {}
      replicas: 3
      requests:
        storage: 10Gi
      storageClassName: local-storage
    tikvPromGateway:
      image: ""
    timezone: UTC
  status:
    pd:
      leader:
        clientURL: ""
        health: false
        id: ""
        lastTransitionTime: null
        name: ""
      phase: Normal
      statefulSet:
        collisionCount: 0
        currentReplicas: 3
        currentRevision: demo-pd-677598c6b9
        observedGeneration: 1
        replicas: 3
        updateRevision: demo-pd-677598c6b9
        updatedReplicas: 3
    tidb: {}
    tikv: {}
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""

xiaojingchen · 2019-05-24T11:48:02Z

@jaltabike Could you give a PD pod's info.

jaltabike · 2019-05-24T12:00:04Z

@xiaojingchen
$ kubectl describe po -n tidb demo-pd-1
Name: demo-pd-1
Namespace: tidb
Priority: 0
PriorityClassName:
Node:
Labels: app.kubernetes.io/component=pd
app.kubernetes.io/instance=demo
app.kubernetes.io/managed-by=tidb-operator
app.kubernetes.io/name=tidb-cluster
controller-revision-hash=demo-pd-677598c6b9
statefulset.kubernetes.io/pod-name=demo-pd-1
Annotations: pingcap.com/last-applied-configuration:
{"volumes":[{"name":"annotations","downwardAPI":{"items":[{"path":"annotations","fieldRef":{"fieldPath":"metadata.annotations"}}]}},{"name...
prometheus.io/path: /metrics
prometheus.io/port: 2379
prometheus.io/scrape: true
Status: Pending
IP:
Controlled By: StatefulSet/demo-pd
Containers:
pd:
Image: pingcap/pd:v2.1.8
Ports: 2380/TCP, 2379/TCP
Host Ports: 0/TCP, 0/TCP
Command:
/bin/sh
/usr/local/bin/pd_start_script.sh
Environment:
NAMESPACE: tidb (v1:metadata.namespace)
PEER_SERVICE_NAME: demo-pd-peer
SERVICE_NAME: demo-pd
SET_NAME: demo-pd
TZ: UTC
Mounts:
/etc/pd from config (ro)
/etc/podinfo from annotations (ro)
/usr/local/bin from startup-script (ro)
/var/lib/pd from pd (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-mscw6 (ro)
Volumes:
pd:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: pd-demo-pd-1
ReadOnly: false
annotations:
Type: DownwardAPI (a volume populated by information about the pod)
Items:
metadata.annotations -> annotations
config:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: demo-pd
Optional: false
startup-script:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: demo-pd
Optional: false
default-token-mscw6:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-mscw6
Optional: false
QoS Class: BestEffort
Node-Selectors:
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message

Warning FailedScheduling 35s (x103928 over 4h10m) tidb-scheduler 0/4 nodes are available: 4 node(s) didn't match node selector.

shuijing198799 · 2019-05-29T06:13:52Z

What version of your operator is used for?

tkanng · 2019-05-29T06:17:18Z

The same problem occurred when I try to deploy TiDB cluster in the DinD Kubernetes cluster. @shuijing198799 My operator's version is :

root@iZhp37kmiszbkwzt5oh9csZ:/k/op# helm get manifest tidb-operator | grep image
        image: pingcap/tidb-operator:v1.0.0-beta.2
        imagePullPolicy: IfNotPresent
        image: pingcap/tidb-operator:v1.0.0-beta.2
        imagePullPolicy: IfNotPresent
        image: mirantis/hypokube:final
        imagePullPolicy: IfNotPresent

weekface · 2019-05-29T06:39:55Z

@jaltabike @tkanng This issue is fixed by #475, can you upgrade to pingcap/tidb-operator:latest and use the latest chart to test it?

By the way, we will release v1.0.0-beta.3 before this weekend which will include this pr: #475.

tkanng · 2019-05-29T06:57:32Z

pingcap/tidb-operator:latest works fine. Thanks!

jaltabike · 2019-06-03T08:42:02Z

@weekface
What does "upgrade to pingcap/tidb-operator:latest" mean?
Does it mean to modify the image version in tidb-cluster/values.yaml and tidb-operator/values.yaml?

weekface · 2019-06-03T08:48:24Z

@jaltabike Yes.

jaltabike · 2019-06-03T09:06:23Z

@weekface
It works now~ Thank you! :)

gregwebs · 2019-06-03T17:34:47Z

@mysticaltech what environment is your K8s running in? on-prem or a cloud provider?

mysticaltech · 2019-06-03T22:02:30Z

@gregwebs It was my fault, now this particular problem is indeed fixed with pingcap/tidb-operator:latest

jaltabike closed this as completed Jun 3, 2019

gregwebs reopened this Jun 3, 2019

gregwebs closed this as completed Jun 3, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

FailedScheduling: nodes didn't match node selector #516

FailedScheduling: nodes didn't match node selector #516

jaltabike commented May 24, 2019

tennix commented May 24, 2019

jaltabike commented May 24, 2019

weekface commented May 24, 2019 •

edited

Loading

jaltabike commented May 24, 2019 •

edited by weekface

Loading

xiaojingchen commented May 24, 2019

jaltabike commented May 24, 2019 •

edited

Loading

shuijing198799 commented May 29, 2019

tkanng commented May 29, 2019

weekface commented May 29, 2019 •

edited

Loading

tkanng commented May 29, 2019

jaltabike commented Jun 3, 2019

weekface commented Jun 3, 2019

jaltabike commented Jun 3, 2019

gregwebs commented Jun 3, 2019

mysticaltech commented Jun 3, 2019

FailedScheduling: nodes didn't match node selector #516

FailedScheduling: nodes didn't match node selector #516

Comments

jaltabike commented May 24, 2019

Bug Report

tennix commented May 24, 2019

jaltabike commented May 24, 2019

weekface commented May 24, 2019 • edited Loading

jaltabike commented May 24, 2019 • edited by weekface Loading

xiaojingchen commented May 24, 2019

jaltabike commented May 24, 2019 • edited Loading

shuijing198799 commented May 29, 2019

tkanng commented May 29, 2019

weekface commented May 29, 2019 • edited Loading

tkanng commented May 29, 2019

jaltabike commented Jun 3, 2019

weekface commented Jun 3, 2019

jaltabike commented Jun 3, 2019

gregwebs commented Jun 3, 2019

mysticaltech commented Jun 3, 2019

weekface commented May 24, 2019 •

edited

Loading

jaltabike commented May 24, 2019 •

edited by weekface

Loading

jaltabike commented May 24, 2019 •

edited

Loading

weekface commented May 29, 2019 •

edited

Loading