Unable to provision PVC with VCP on v1.11.0

[ventris]

/kind bug

What happened:
After setting up the storage provider and storage class, creating a claim results in errors on the controller and the pvc is stuck in status pending.

Errors in Controller manager log:

E0703 12:30:05.239366       1 datacenter.go:78] Unable to find VM by UUID. VM UUID: 
E0703 12:30:05.239392       1 nodemanager.go:414] Error "No VM found" node info for node "hostname.example.com" not found
E0703 12:30:05.239409       1 vsphere_util.go:134] Error while obtaining Kubernetes node nodeVmDetail details. error : No VM found
E0703 12:30:05.239415       1 vsphere.go:1160] Failed to get shared datastore: No VM found
I0703 12:30:05.239676       1 event.go:221] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"pvcsc001", UID:"d012094b-7ebc-11e8-b9bd-005056b38c74", APIVersion:"v1", ResourceVersion:"11416", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' Failed to provision volume with StorageClass "fast": No VM found

storageclass and persistentvolume yamls

kubectl describe sc
Name:            test
IsDefaultClass:  Yes
Annotations:     kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"storage.k8s.io/v1","kind":"StorageClass","metadata":{"annotations":{"storageclass.kubernetes.io/is-default-class":"true"},"name":"test","namespace":""},"parameters":{"datastore":"/Test/Test-DSC1/Test-DS8","diskformat":"zeroedthink"},"provisioner":"kubernetes.io/vsphere-volume"}
,storageclass.kubernetes.io/is-default-class=true
Provisioner:           kubernetes.io/vsphere-volume
Parameters:            datastore=/Test/Test-DSC1/Test-DS8,diskformat=zeroedthink
AllowVolumeExpansion:  <unset>
MountOptions:          <none>
ReclaimPolicy:         Delete
VolumeBindingMode:     Immediate
Events:                <none>

kubectl describe pvc
Name:          test-claim
Namespace:     default
StorageClass:  test
Status:        Pending
Volume:        
Labels:        <none>
Annotations:   volume.beta.kubernetes.io/storage-provisioner=kubernetes.io/vsphere-volume
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      
Access Modes:  
Events:
  Type     Reason              Age               From                         Message
  ----     ------              ----              ----                         -------
  Warning  ProvisioningFailed  9s (x3 over 30s)  persistentvolume-controller  Failed to provision volume with StorageClass "test": No VM found

Using the UUId that kubectl have I can get the vm.info from govc.

kubectl describe node | grep "System UUID"
System UUID: 4233947E-8539-977A-FE40-EF5573B6F3E1

govc vm.info -vm.uuid=4233947e-8539-977a-fe40-ef5573b6f3e1​
Name:
Path:
UUID: 4233947e-8539-977a-fe40-ef5573b6f3e1
Guest name: Red Hat Enterprise Linux 7 (64-bit)
Memory: 12288MB
CPU: 3 vCPU(s)
Power state: poweredOn
Boot time: 2018-07-03 09:45:09.945151 +0000 UTC
IP address: 10.25.25.16
Host:

What you expected to happen:
Able to successfully create dynamic claims.

How to reproduce it (as minimally and precisely as possible):
CentOS

• kubeadm init and create one master, one worker.
• Follow the guide to setup vsphere cloud provider.
  vsphere cloud provider

Anything else we need to know?:

Environment:

• Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.0", GitCommit:"91e7b4fd31fcd3d5f436da26c980becec37ceefe", GitTreeState:"clean", BuildDate:"2018-06-27T20:17:28Z", GoVersion:"go1.10.2", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.0", GitCommit:"91e7b4fd31fcd3d5f436da26c980becec37ceefe", GitTreeState:"clean", BuildDate:"2018-06-27T20:08:34Z", GoVersion:"go1.10.2", Compiler:"gc", Platform:"linux/amd64"}

• Cloud provider or hardware configuration:
  vCenter 6.5, esxi 6.5, redhat 7.5
• OS (e.g. from /etc/os-release):
  Red Hat Enterprise Linux Server 7.5 (Maipo)
• Kernel (e.g. uname -a):
  3.10.0-862.el7.x86_64 WIP: Address lack of support within Kubernetes to consume vSphere managed storage #1 SMP Wed Mar 21 18:14:51 EDT 2018 x86_64 x86_64 x86_64 GNU/Linux
• Install tools:

kubeadm version: &version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.0", GitCommit:"91e7b4fd31fcd3d5f436da26c980becec37ceefe", GitTreeState:"clean", BuildDate:"2018-06-27T20:14:41Z", GoVersion:"go1.10.2", Compiler:"gc", Platform:"linux/amd64"}

• Others:

[divyenpatel]

@ventris Can you check providerID is correctly set on nodes with its systemUUID.

$ kubectl get node kubernetes-node1 -o json | grep providerID
        "providerID": "vsphere://421196b5-3e10-7072-7db2-bd9a26f8755b"

$ kubectl get node kubernetes-node1 -o json | grep UUID
            "systemUUID": "421196B5-3E10-7072-7DB2-BD9A26F8755B"

[ventris]

@divyenpatel None of the servers have providerID set.

There is an UUID but no providerID

"systemUUID": "42336D3A-F604-1636-234A-818777F9C74A"

listing the product_serial gives.

cat /sys/class/dmi/id/product_serial
VMware-42 33 94 7e 85 39 97 7a-fe 40 ef 55 73 b6 f3 e1

Neither externalID or providerID exists under spec in the json.

I have validated against a system that runs debian and have a working vcp. It seems like these are the two fields that are missing.

[ventris]

Inserting the provisionerID as a flag to the kubelet on all nodes makes you able to successfully create persistent volumes on RHEL.

--provider-id='vsphere://<id-number>'

It seems that on CentOS/RHEL you have to supply the --provider-id to the kubelet environment config as the kubelet wont find it by itself. Might be a bug in kublet itself.

[ReSearchITEng]

Here is a script that patches node info with the correct vmware id:
https://github.com/ReSearchITEng/kubeadm-playbook/blob/master/roles/storage/files/vsphere_bug_fix.sh
(alternatively you may want to install the entire cluster using this project)

[franrebo84]

The script works perfectly...! Thanks so much!

[dkirrane]

I believe this is a duplicate
kubernetes#65933