Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

attacher.MountDevice failed to create newCsiDriverClient: driver name smb.csi.k8s.io not found in the list of registered CSI drivers #69

Closed
hetii opened this issue Jul 20, 2020 · 6 comments · Fixed by #110
Closed

attacher.MountDevice failed to create newCsiDriverClient: driver name smb.csi.k8s.io not found in the list of registered CSI drivers #69

hetii opened this issue Jul 20, 2020 · 6 comments · Fixed by #110

Comments

@hetii
Copy link

hetii commented Jul 20, 2020 •
edited
Loading

Hello I deploy v0.2.0 of this driver on kubernetes cluster provisioned by rancher2 on top of openstack.
As you can see below pods from csi are there:

kube-system         csi-smb-controller-864498f867-bldgf                             3/3     Running             17         2d1h
kube-system         csi-smb-controller-864498f867-bqc9v                             3/3     Running             17         2d1h
kube-system         csi-smb-node-9bwrt                                              3/3     Running             0          2d1h
kube-system         csi-smb-node-kb85d                                              3/3     Running             0          2d1h
kube-system         csi-smb-node-mrbfd                                              3/3     Running             0          2d1h
kube-system         csi-smb-node-nhf2d                                              3/3     Running             0          2d1h

But when I try to use this driver to mount CIFS share from another cluster that is made in azure I get such error::

MountVolume.MountDevice failed for volume "pv-smb" : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name smb.csi.k8s.io not found in the list of registered CSI drivers

PV and PVC looks fine:

NAME                 CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM              STORAGECLASS    REASON   AGE
pv-smb               100Gi      RWX            Retain           Bound    default/pvc-smb                             2d1h

NAMESPACE   NAME      STATUS   VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS   AGE
default     pvc-smb   Bound    pv-smb   100Gi      RWX                           2d1h

kubernetes version:

Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.6", GitCommit:"dff82dc0de47299ab66c83c626e08b245ab19037", GitTreeState:"clean", BuildDate:"2020-07-16T14:19:25Z", GoVersion:"go1.13.13", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.3", GitCommit:"2e7996e3e2712684bc73f0dec0200d64eec7fe40", GitTreeState:"clean", BuildDate:"2020-05-20T12:43:34Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}

kubectl logs csi-smb-node-9bwrt -c smb -n kube-system show:

I0718 14:45:30.935428       1 main.go:84] set up prometheus server on [::]:39615
I0718 14:45:30.936100       1 smb.go:58] 
DRIVER INFORMATION:
-------------------
Build Date: "2020-07-12T03:32:01Z"
Compiler: gc
Driver Name: smb.csi.k8s.io
Driver Version: v0.3.0
Git Commit: 84ac39ff780b8db59b29f7f64d6dda2154356c13
Go Version: go1.14.4
Platform: linux/amd64

Streaming logs below:
I0718 14:45:30.936515       1 mount_linux.go:163] Detected OS without systemd
I0718 14:45:30.936887       1 driver.go:93] Enabling controller service capability: CREATE_DELETE_VOLUME
I0718 14:45:30.936959       1 driver.go:112] Enabling volume access mode: SINGLE_NODE_WRITER
I0718 14:45:30.937019       1 driver.go:112] Enabling volume access mode: SINGLE_NODE_READER_ONLY
I0718 14:45:30.937068       1 driver.go:112] Enabling volume access mode: MULTI_NODE_READER_ONLY
I0718 14:45:30.937124       1 driver.go:112] Enabling volume access mode: MULTI_NODE_SINGLE_WRITER
I0718 14:45:30.937177       1 driver.go:112] Enabling volume access mode: MULTI_NODE_MULTI_WRITER
I0718 14:45:30.937228       1 driver.go:103] Enabling node service capability: STAGE_UNSTAGE_VOLUME
I0718 14:45:30.937520       1 server.go:118] Listening for connections on address: &net.UnixAddr{Name:"//csi/csi.sock", Net:"unix"}
I0718 14:45:31.095826       1 utils.go:111] GRPC call: /csi.v1.Identity/GetPluginInfo
I0718 14:45:31.095844       1 utils.go:112] GRPC request: 
I0718 14:45:31.095856       1 identityserver.go:32] Using default GetPluginInfo
I0718 14:45:31.095870       1 utils.go:118] GRPC response: name:"smb.csi.k8s.io" vendor_version:"v0.3.0" 
I0718 14:45:31.905532       1 utils.go:111] GRPC call: /csi.v1.Identity/GetPluginInfo
I0718 14:45:31.905550       1 utils.go:112] GRPC request: 
I0718 14:45:31.905557       1 identityserver.go:32] Using default GetPluginInfo
I0718 14:45:31.905561       1 utils.go:118] GRPC response: name:"smb.csi.k8s.io" vendor_version:"v0.3.0" 
I0718 14:46:27.720230       1 utils.go:111] GRPC call: /csi.v1.Identity/Probe
I0718 14:46:27.720336       1 utils.go:112] GRPC request: 
I0718 14:46:27.720365       1 utils.go:118] GRPC response: ready:<value:true > 
I0718 14:46:57.716983       1 utils.go:111] GRPC call: /csi.v1.Identity/Probe
I0718 14:46:57.717034       1 utils.go:112] GRPC request: 
I0718 14:46:57.717047       1 utils.go:118] GRPC response: ready:<value:true > 
I0718 14:47:27.716511       1 utils.go:111] GRPC call: /csi.v1.Identity/Probe
I0718 14:47:27.716580       1 utils.go:112] GRPC request:

kubectl get CSIDriver -A:

NAME             ATTACHREQUIRED   PODINFOONMOUNT   MODES        AGE
smb.csi.k8s.io   false            true             Persistent   2d2h
@andyzhangx

This comment has been minimized.

Sign in to view
@andyzhangx
Copy link
Member

andyzhangx commented Jul 21, 2020 •
edited
Loading

could you get the registrar logs, e.g.

k logs csi-smb-node-h2pp5 -n kube-system -c node-driver-registrar > /tmp/csi-smb-node-h2pp5

the successful logs could be like following:

I0721 12:34:23.125548       1 main.go:110] Version: v1.2.0-0-g6ef000ae
I0721 12:34:23.125587       1 main.go:120] Attempting to open a gRPC connection with: "/csi/csi.sock"
I0721 12:34:23.125597       1 connection.go:151] Connecting to unix:///csi/csi.sock
I0721 12:34:24.126348       1 main.go:127] Calling CSI driver to discover driver name
I0721 12:34:24.126365       1 connection.go:180] GRPC call: /csi.v1.Identity/GetPluginInfo
I0721 12:34:24.126370       1 connection.go:181] GRPC request: {}
I0721 12:34:24.128574       1 connection.go:183] GRPC response: {"name":"smb.csi.k8s.io","vendor_version":"v0.3.0"}
I0721 12:34:24.129617       1 connection.go:184] GRPC error: <nil>
I0721 12:34:24.129624       1 main.go:137] CSI driver name: "smb.csi.k8s.io"
I0721 12:34:24.129772       1 node_register.go:58] Starting Registration Server at: /registration/smb.csi.k8s.io-reg.sock
I0721 12:34:24.130069       1 node_register.go:67] Registration Server started at: /registration/smb.csi.k8s.io-reg.sock
I0721 12:34:25.864398       1 main.go:77] Received GetInfo call: &InfoRequest{}
I0721 12:34:25.903679       1 main.go:87] Received NotifyRegistrationStatus call: &RegistrationStatus{PluginRegistered:true,Error:,}

And is there sth different with your kubelet path?

csi-driver-smb/deploy/csi-smb-node.yaml

Lines 112 to 122 in 70f3d0f

- hostPath:
path: /var/lib/kubelet/plugins/smb.csi.k8s.io
type: DirectoryOrCreate
name: socket-dir
- hostPath:
path: /var/lib/kubelet/
type: DirectoryOrCreate
name: mountpoint-dir
- hostPath:
path: /var/lib/kubelet/plugins_registry/
type: DirectoryOrCreate

@tallmanbrew
Copy link

tallmanbrew commented Aug 18, 2020 •
edited
Loading

Hello, I'm running into the same issues hetii was.
Here is the log you requested, it looks like the last two lines are not added.

I0818 13:06:09.539575       1 main.go:120] Attempting to open a gRPC connection with: "/csi/csi.sock"
I0818 13:06:09.539595       1 connection.go:151] Connecting to unix:///csi/csi.sock
W0818 13:06:19.539919       1 connection.go:170] Still connecting to unix:///csi/csi.sock
W0818 13:06:29.539952       1 connection.go:170] Still connecting to unix:///csi/csi.sock
W0818 13:06:39.539906       1 connection.go:170] Still connecting to unix:///csi/csi.sock
W0818 13:06:49.539962       1 connection.go:170] Still connecting to unix:///csi/csi.sock
W0818 13:06:59.539927       1 connection.go:170] Still connecting to unix:///csi/csi.sock
W0818 13:07:09.539877       1 connection.go:170] Still connecting to unix:///csi/csi.sock
W0818 13:07:19.540023       1 connection.go:170] Still connecting to unix:///csi/csi.sock
W0818 13:07:29.539890       1 connection.go:170] Still connecting to unix:///csi/csi.sock
W0818 13:07:39.539927       1 connection.go:170] Still connecting to unix:///csi/csi.sock
W0818 13:07:49.540372       1 connection.go:170] Still connecting to unix:///csi/csi.sock
W0818 13:07:59.540408       1 connection.go:170] Still connecting to unix:///csi/csi.sock
W0818 13:08:09.540156       1 connection.go:170] Still connecting to unix:///csi/csi.sock
I0818 13:08:10.305105       1 main.go:127] Calling CSI driver to discover driver name
I0818 13:08:10.305135       1 connection.go:180] GRPC call: /csi.v1.Identity/GetPluginInfo
I0818 13:08:10.305146       1 connection.go:181] GRPC request: {}
I0818 13:08:10.308418       1 connection.go:183] GRPC response: {"name":"smb.csi.k8s.io","vendor_version":"v0.2.0"}
I0818 13:08:10.309792       1 connection.go:184] GRPC error: <nil>
I0818 13:08:10.309908       1 main.go:137] CSI driver name: "smb.csi.k8s.io"
I0818 13:08:10.310133       1 node_register.go:58] Starting Registration Server at: /registration/smb.csi.k8s.io-reg.sock
I0818 13:08:10.310515       1 node_register.go:67] Registration Server started at: /registration/smb.csi.k8s.io-reg.sock```

@hetii
Copy link
Author

hetii commented Aug 18, 2020 •
edited
Loading

Hi again. I set it up from scratch and my logs look like:

I0818 19:07:13.368677       1 main.go:110] Version: v1.2.0-0-g6ef000ae
I0818 19:07:13.368961       1 main.go:120] Attempting to open a gRPC connection with: "/csi/csi.sock"
I0818 19:07:13.369072       1 connection.go:151] Connecting to unix:///csi/csi.sock
W0818 19:07:23.369603       1 connection.go:170] Still connecting to unix:///csi/csi.sock
I0818 19:07:28.095307       1 main.go:127] Calling CSI driver to discover driver name
I0818 19:07:28.095525       1 connection.go:180] GRPC call: /csi.v1.Identity/GetPluginInfo
I0818 19:07:28.095567       1 connection.go:181] GRPC request: {}
I0818 19:07:28.100265       1 connection.go:183] GRPC response: {"name":"smb.csi.k8s.io","vendor_version":"v0.2.0"}
I0818 19:07:28.100963       1 connection.go:184] GRPC error: <nil>
I0818 19:07:28.100972       1 main.go:137] CSI driver name: "smb.csi.k8s.io"
I0818 19:07:28.101245       1 node_register.go:58] Starting Registration Server at: /registration/smb.csi.k8s.io-reg.sock
I0818 19:07:28.101662       1 node_register.go:67] Registration Server started at: /registration/smb.csi.k8s.io-reg.sock

Moreover the paths in rancheros worker node contains such entries:

[root@ru-cluster-standard-4-8-80-w-0 rancher]# ls -al /var/lib/kubelet/
total 20
drwxr-xr-x    5 root     root          4096 Aug 18 19:07 .
drwxr-xr-x    1 root     root          4096 Aug  2 20:15 ..
drwxr-xr-x    3 root     root          4096 Aug 18 19:07 plugins
drwxr-xr-x    2 root     root          4096 Aug 18 19:07 plugins_registry
drwxr-xr-x    2 root     root          4096 Aug  2 20:15 volumeplugins
[root@ru-cluster-standard-4-8-80-w-0 rancher]# ls -al /var/lib/kubelet/plugins
total 12
drwxr-xr-x    3 root     root          4096 Aug 18 19:07 .
drwxr-xr-x    5 root     root          4096 Aug 18 19:07 ..
drwxr-xr-x    2 root     root          4096 Aug 18 19:07 smb.csi.k8s.io
[root@ru-cluster-standard-4-8-80-w-0 rancher]# ls -al /var/lib/kubelet/plugins/smb.csi.k8s.io/
total 8
drwxr-xr-x    2 root     root          4096 Aug 18 19:07 .
drwxr-xr-x    3 root     root          4096 Aug 18 19:07 ..
srwxr-xr-x    1 root     root             0 Aug 18 19:07 csi.sock
[root@ru-cluster-standard-4-8-80-w-0 rancher]# ls -al /var/lib/kubelet/plugins_registry/ 
total 8
drwxr-xr-x    2 root     root          4096 Aug 18 19:07 .
drwxr-xr-x    5 root     root          4096 Aug 18 19:07 ..
srwx------    1 root     root             0 Aug 18 19:07 smb.csi.k8s.io-reg.sock 
[root@ru-cluster-standard-4-8-80-w-0 rancher]# ls -al /var/lib/kubelet/volumeplugins/
total 8
drwxr-xr-x    2 root     root          4096 Aug  2 20:15 .
drwxr-xr-x    5 root     root          4096 Aug 18 19:07 ..
[root@ru-cluster-standard-4-8-80-w-0 rancher]#

@hetii
Copy link
Author

hetii commented Aug 18, 2020

It seams it works for me now.

The main root issue is similar to issues:
rancher/rancher#14675
rancher/rancher#13897

It looks like at least in my case the rancher kublet path is /opt/rke/var/lib/kubelet/ not /var/lib/kubelet/.

After updating csi-smb-node.yaml and redeploying it I was able to mount my azure share.

@hetii hetii mentioned this issue Aug 18, 2020
Closed
@andyzhangx
Copy link
Member

@hetii thanks for the investigation, if use another kubelet path is reasonable requirement, I think we can add this parameter into helm chart installation at least

@boddumanohar boddumanohar mentioned this issue Sep 7, 2020
Merged
4 tasks
@Chup4Chups Chup4Chups mentioned this issue Oct 18, 2021
Closed
andyzhangx pushed a commit to andyzhangx/csi-driver-smb that referenced this issue May 1, 2022
@k8s-ci-robot
Merge pull request kubernetes-csi#69 from pohly/test-driver-config
d717c8c 
prow.sh: generic driver installation
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

make kubelet path configurable from helm values boddumanohar/csi-driver-smb
3 participants
@hetii @andyzhangx @tallmanbrew