CSI driver-smb mount the directory and immediately unmount despite no errors. #880

lanzmlt · 2024-12-05T13:26:17Z

lanzmlt
Dec 5, 2024

Hello,

I can't get csi smb driver to work properly in managed k8s cluster running on CentOS Linux 7 (KERNEL VERSION 3.10.0-1160.92.1.el7.x86_64).
I read a similar discussionCSI driver-smb not mounting, but it's not my case.
I hope someone can help me solve this problem.
I have deployed StorageClass and StatefulSet with following manifests from documentation.

StorageClass:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: smb
provisioner: smb.csi.k8s.io
parameters:
  source: //10.0.0.10/Backup
  # if csi.storage.k8s.io/provisioner-secret is provided, will create a sub directory
  # with PV name under source
  csi.storage.k8s.io/provisioner-secret-name: smbcreds
  csi.storage.k8s.io/provisioner-secret-namespace: default
  csi.storage.k8s.io/node-stage-secret-name: smbcreds
  csi.storage.k8s.io/node-stage-secret-namespace: default
reclaimPolicy: Delete  # available values: Delete, Retain
volumeBindingMode: Immediate
mountOptions:
  - seal
  - vers=3.0
  - nosuid
  - noexec
  - dir_mode=0777
  - file_mode=0777
  - uid=1001
  - gid=1001
  - noperm
  - mfsymlinks
  - cache=strict
  - noserverino

StatefulSet:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: statefulset-smb
  namespace: default
  labels:
    app: nginx
spec:
  serviceName: statefulset-smb
  replicas: 1
  template:
    metadata:
      labels:
        app: nginx
    spec:
      nodeSelector:
        "kubernetes.io/os": linux
      containers:
        - name: statefulset-smb
          image: mcr.microsoft.com/oss/nginx/nginx:1.19.5
          command:
            - "/bin/bash"
            - "-c"
            - set -euo pipefail; while true; do echo $(date) >> /mnt/smb/outfile; sleep 1; done
          volumeMounts:
            - name: persistent-storage
              mountPath: /mnt/smb
              readOnly: false
  updateStrategy:
    type: RollingUpdate
  selector:
    matchLabels:
      app: nginx
  volumeClaimTemplates:
    - metadata:
        name: persistent-storage
      spec:
        storageClassName: smb
        accessModes: ["ReadWriteOnce"]
        resources:
          requests:
            storage: 20Gi

State of pod:

NAME                READY   STATUS    RESTARTS          AGE    IP             NODE             NOMINATED NODE   READINESS GATES
statefulset-smb-0   1/1     Running   0                 15m    192.168.2.214   192.168.242.78    <none>           <none>

After pod in up state, PVC was created automatically and Bound:

NAME                                   STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS        AGE
persistent-storage-statefulset-smb-0   Bound    pvc-8c9943b6-bdcc-4f10-bdf9-6bcc614857d3   20Gi       RWO            smb                 2m55s

But directory is not mounted in pod:

kubectl exec -it statefulset-smb-0 -- df -h                                                                                                    
Filesystem                     Size  Used Avail Use% Mounted on
overlay                         89G   53G   32G  63% /
tmpfs                           64M     0   64M   0% /dev
tmpfs                          3.9G     0  3.9G   0% /sys/fs/cgroup
/dev/mapper/vgpaas-kubernetes  9.8G   55M  9.2G   1% /mnt/smb
/dev/mapper/vgpaas-dockersys    89G   53G   32G  63% /etc/hostname
shm                             64M     0   64M   0% /dev/shm
tmpfs                          7.3G   12K  7.3G   1% /run/secrets/kubernetes.io/serviceaccount
tmpfs                          3.9G     0  3.9G   0% /proc/acpi
tmpfs                          3.9G     0  3.9G   0% /proc/scsi
tmpfs                          3.9G     0  3.9G   0% /sys/firmware

kubectl logs csi-smb-controller-69f9ccfbb6-mpsnc -c smb -n kube-system

I1205 12:49:19.617813       1 utils.go:83] GRPC response: {}
I1205 12:49:56.471186       1 utils.go:76] GRPC call: /csi.v1.Controller/CreateVolume
I1205 12:49:56.471202       1 utils.go:77] GRPC request: {"capacity_range":{"required_bytes":10737418240},"name":"pvc-8c9943b6-bdcc-4f10-bdf9-6bcc614857d3","parameters":{"csi.storage.k8s.io/pv/name":"pvc-8c9943b6-bdcc-4f10-bdf9-6bcc614857d3","csi.storage.k8s.io/pvc/name":"persistent-storage-statefulset-smb-0","csi.storage.k8s.io/pvc/namespace":"default","source":"//10.0.0.10/Backup"},"secrets":"***stripped***","volume_capabilities":[{"AccessType":{"Mount":{"mount_flags":["seal","vers=3.0","nosuid","noexec","dir_mode=0777","file_mode=0777","uid=1001","gid=1001","noperm","mfsymlinks","cache=strict","noserverino"]}},"access_mode":{"mode":7}}]}
I1205 12:49:56.471327       1 controllerserver.go:314] internally mounting //10.0.0.10/Backup at /tmp/pvc-8c9943b6-bdcc-4f10-bdf9-6bcc614857d3
I1205 12:49:56.471413       1 nodeserver.go:209] NodeStageVolume: targetPath(/tmp/pvc-8c9943b6-bdcc-4f10-bdf9-6bcc614857d3) volumeID(10.0.0.10/Backup#pvc-8c9943b6-bdcc-4f10-bdf9-6bcc614857d3##) context(map[source://10.0.0.10/Backup]) mountflags([seal vers=3.0 nosuid noexec dir_mode=0777 file_mode=0777 uid=1001 gid=1001 noperm mfsymlinks cache=strict noserverino]) mountOptions([seal vers=3.0 nosuid noexec dir_mode=0777 file_mode=0777 uid=1001 gid=1001 noperm mfsymlinks cache=strict noserverino])
I1205 12:49:56.471620       1 mount_linux.go:218] Mounting cmd (mount) with arguments (-t cifs -o seal,vers=3.0,nosuid,noexec,dir_mode=0777,file_mode=0777,uid=1001,gid=1001,noperm,mfsymlinks,cache=strict,noserverino,<masked> //10.0.0.10/Backup /tmp/pvc-8c9943b6-bdcc-4f10-bdf9-6bcc614857d3)
I1205 12:49:56.565836       1 nodeserver.go:241] volume(10.0.0.10/Backup#pvc-8c9943b6-bdcc-4f10-bdf9-6bcc614857d3##) mount "//10.0.0.10/Backup" on "/tmp/pvc-8c9943b6-bdcc-4f10-bdf9-6bcc614857d3" succeeded
I1205 12:49:56.593727       1 controllerserver.go:332] internally unmounting /tmp/pvc-8c9943b6-bdcc-4f10-bdf9-6bcc614857d3
I1205 12:49:56.593749       1 nodeserver.go:264] NodeUnstageVolume: CleanupMountPoint on /tmp/pvc-8c9943b6-bdcc-4f10-bdf9-6bcc614857d3 with volume 10.0.0.10/Backup#pvc-8c9943b6-bdcc-4f10-bdf9-6bcc614857d3##
I1205 12:49:56.605737       1 mount_helper_common.go:93] unmounting "/tmp/pvc-8c9943b6-bdcc-4f10-bdf9-6bcc614857d3" (corruptedMount: false, mounterCanSkipMountPointChecks: true)
I1205 12:49:56.605761       1 mount_linux.go:360] Unmounting /tmp/pvc-8c9943b6-bdcc-4f10-bdf9-6bcc614857d3
I1205 12:49:56.632903       1 mount_helper_common.go:150] Deleting path "/tmp/pvc-8c9943b6-bdcc-4f10-bdf9-6bcc614857d3"
I1205 12:49:56.632991       1 nodeserver.go:273] NodeUnstageVolume: unmount volume 10.0.0.10/Backup#pvc-8c9943b6-bdcc-4f10-bdf9-6bcc614857d3## on /tmp/pvc-8c9943b6-bdcc-4f10-bdf9-6bcc614857d3 successfully
I1205 12:49:56.633005       1 utils.go:83] GRPC response: {"volume":{"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-8c9943b6-bdcc-4f10-bdf9-6bcc614857d3","csi.storage.k8s.io/pvc/name":"persistent-storage-statefulset-smb-0","csi.storage.k8s.io/pvc/namespace":"default","source":"//10.0.0.10/Backup","subdir":"pvc-8c9943b6-bdcc-4f10-bdf9-6bcc614857d3"},"volume_id":"10.0.0.10/Backup#pvc-8c9943b6-bdcc-4f10-bdf9-6bcc614857d3##"}}

Login and password is correct, and i can connect to smb from the debug pod on node:

# smbclient //10.0.0.10.local/Backup -U $USER%$PASSWORD -W $WORK_GROUP

Try "help" to get a list of possible commands.
smb: \>

Also i can mount smb directory on node 192.168.242.78 where pod is created after apply statefulset:

# mount -v -t cifs //10.0.0.10/Backup /tmp/test -o vers=3.0,username=$USER,password=$PASSWORD,uid=1001,gid=1001,dir_mode=0777,file_mode=0777,cache=strict,actimeo=30
# ls 
pvc-8c9943b6-bdcc-4f10-bdf9-6bcc614857d3  test

# mount | grep cifs 
//10.0.0.10/Backup on /tmp/test type cifs (rw,relatime,vers=3.0,cache=strict,username=$USER,domain=$DOMAIN_NAME,uid=1001,forceuid,gid=1001,forcegid,addr=10.0.0.10,file_mode=0777,dir_mode=0777,soft,nounix,serverino,mapposix,rsize=1048576,wsize=1048576,echo_interval=60,actimeo=30)

Username, password and domain have been changed for security purposes.
Any help will be greatly appreciated... Thanks

Answered by andyzhangx

Dec 9, 2024

have you --set linux.kubelet="/mnt/paas/kubernetes/kubelet" in the driver install?

View full answer

andyzhangx · 2024-12-09T03:48:52Z

andyzhangx
Dec 9, 2024
Maintainer

@lanzmlt what is your kubelet path? if it's not /var/lib/kubelet, follow guide here: https://github.com/kubernetes-csi/csi-driver-smb/tree/master/charts#tips

0 replies

lanzmlt · 2024-12-09T06:50:55Z

lanzmlt
Dec 9, 2024
Author

@andyzhangx i can't run smb-controller on control plane node, because i have a managed k8s.

In my case kubelet plugins located in:

# ls -l /var/lib/kubelet/plugins/
total 36
drwxr-x--- 2 root root 4096 Nov  9 12:14 disk.csi.everest.io
drwxr-x--- 3 root root 4096 Jun  3  2024 kubernetes.io
drwxr-x--- 2 root root 4096 Nov  9 12:14 local.csi.everest.io
drwxr-x--- 2 root root 4096 Nov  9 12:14 nas.csi.everest.io
drwxr-x--- 2 root root 4096 Nov  9 12:14 obs.csi.everest.io
drwxr-x--- 2 root root 4096 Nov  9 12:14 proxy.csi.everest.io
drwxr-x--- 2 root root 4096 Nov  9 12:14 sfsturbo.csi.everest.io
drwxr-xr-x 2 root root 4096 Nov 14 13:20 smb.csi.io
drwxr-xr-x 2 root root 4096 Dec  5 14:12 smb.csi.k8s.io

So, unfortunately, these tips not my case and can't help me.

6 replies

andyzhangx Dec 9, 2024
Maintainer

can you provide csi driver daemonset pod logs on the node?

lanzmlt Dec 9, 2024
Author

kubectl get po -o wide                                                                                    
NAME                READY   STATUS    RESTARTS          AGE    IP             NODE             NOMINATED NODE   READINESS GATES
statefulset-smb-0   1/1     Running   0                 2m3s   192.168.2.214  192.168.242.78    <none>           <none>

kubectl --namespace=kube-system get pods --selector="app.kubernetes.io/name=csi-driver-smb" -o wide | grep 192.168.242.78
csi-smb-node-qjnsw                    3/3     Running   0              174m   192.168.242.78    192.168.242.78    <none>           <none>

kubectl -n kube-system logs csi-smb-node-qjnsw
Defaulted container "liveness-probe" out of: liveness-probe, node-driver-registrar, smb
I1209 08:56:25.799643       1 main.go:135] "Calling CSI driver to discover driver name"
I1209 08:56:25.804429       1 main.go:143] "CSI driver name" driver="smb.csi.k8s.io"
I1209 08:56:25.804481       1 main.go:172] "ServeMux listening" address="localhost:29643"

kubectl -n kube-system exec csi-smb-node-qjnsw -it -c smb -- mount | grep cifs
//10.0.0.10/Backup/pvc-56e51337-255c-435a-98fa-8f83dcf9b43e on /mnt/paas/kubernetes/kubelet/plugins/kubernetes.io/csi/pv/pvc-56e51337-255c-435a-98fa-8f83dcf9b43e/globalmount type cifs (rw,nosuid,noexec,relatime,vers=3.0,cache=strict,username=backup,domain=smb,uid=1001,forceuid,gid=1001,forcegid,addr=10.0.0.10,file_mode=0777,dir_mode=0777,seal,soft,nounix,mapposix,mfsymlinks,noperm,rsize=1048576,wsize=1048576,echo_interval=60,actimeo=1)
//10.0.0.10/Backup/pvc-56e51337-255c-435a-98fa-8f83dcf9b43e on /mnt/paas/kubernetes/kubelet/pods/a7c5766f-83f2-47d4-8d30-541c73a82d9a/volumes/kubernetes.io~csi/pvc-56e51337-255c-435a-98fa-8f83dcf9b43e/mount type cifs (rw,relatime,vers=3.0,cache=strict,username=backup,domain=smb,uid=1001,forceuid,gid=1001,forcegid,addr=10.0.0.10,file_mode=0777,dir_mode=0777,seal,soft,nounix,mapposix,mfsymlinks,noperm,rsize=1048576,wsize=1048576,echo_interval=60,actimeo=1)

andyzhangx Dec 9, 2024
Maintainer

have you --set linux.kubelet="/mnt/paas/kubernetes/kubelet" in the driver install?

Answer selected by andyzhangx

lanzmlt Dec 9, 2024
Author

No, i use the default path that set in chart values for kubelet - /var/lib/kubelet

andyzhangx Dec 9, 2024
Maintainer

pls use --set linux.kubelet="/mnt/paas/kubernetes/kubelet" in driver install

lanzmlt Dec 9, 2024
Author

This works for me
Thank you @andyzhangx !

RobJohnson-RoJoPro · 2024-12-09T12:08:51Z

RobJohnson-RoJoPro
Dec 9, 2024

I have this problem too with MicroK8s - home lab.
I can confirm I installed the driver with the correct kublet path as per the docs. The SC, PV and PVC all create correctly and become bound. I start a simple nginx pod with PVC. It runs successfully. However, files are not appearing on my local NAS.
I can mount from the node itself successfully. I can see the files appearing in the node /var/snap/microk8s/common/var/lib/kubelet/pods/
from the pod.
kube exec -it csi-smb-node-95z9k -n kube-system -c smb -- mount | grep cifs
Shows the correct paths and directories.
It is very frustrating.

5 replies

andyzhangx Dec 9, 2024
Maintainer

have you --set linux.kubelet="..." in the driver install? follow guide here: https://github.com/kubernetes-csi/csi-driver-smb/tree/master/charts#tips

RobJohnson-RoJoPro Dec 9, 2024

Hi Andy,
This is what I used:
--set kubeletDir=/var/snap/microk8s/common/var/lib/kubelet

As I'm using MicroK8s as per the docs for install. Is this correct?

kube -n kube-system exec csi-smb-node-95z9k -it -c smb -- mount | grep cifs

//192.168.0.210/pv on /var/snap/microk8s/common/var/lib/kubelet/plugins/kubernetes.io/csi/smb.csi.k8s.io/2a97516c354b68848cdbd8f54a226a0a55b21ed138e207ad6c5cbb9c00aa5aea/globalmount type cifs (rw,relatime,vers=2.0,cache=strict,username=*****,uid=0,noforceuid,gid=0,noforcegid,addr=192.168.0.210,file_mode=0777,dir_mode=0777,soft,nounix,serverino,mapposix,reparse=nfs,rsize=65536,wsize=65536,bsize=1048576,retrans=1,echo_interval=60,actimeo=1,closetimeo=1)
//192.168.0.210/pv on /var/snap/microk8s/common/var/lib/kubelet/plugins/kubernetes.io/csi/smb.csi.k8s.io/9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08/globalmount type cifs (rw,relatime,vers=2.0,cache=strict,username=*****,uid=0,noforceuid,gid=0,noforcegid,addr=192.168.0.210,file_mode=0777,dir_mode=0777,soft,nounix,serverino,mapposix,noperm,reparse=nfs,rsize=65536,wsize=65536,bsize=1048576,retrans=1,echo_interval=60,actimeo=1,closetimeo=1)
//192.168.0.210/pv on /var/snap/microk8s/common/var/lib/kubelet/pods/1db63ac6-473a-4495-99f1-5a10cd6771f7/volumes/kubernetes.io~csi/pv-test/mount type cifs (rw,relatime,vers=2.0,cache=strict,username=*****,uid=0,noforceuid,gid=0,noforcegid,addr=192.168.0.210,file_mode=0777,dir_mode=0777,soft,nounix,serverino,mapposix,noperm,reparse=nfs,rsize=65536,wsize=65536,bsize=1048576,retrans=1,echo_interval=60,actimeo=1,closetimeo=1)

andyzhangx Dec 9, 2024
Maintainer

it's --set linux.kubelet="..."

andyzhangx Dec 9, 2024
Maintainer

it should work per #674 (comment)

RobJohnson-RoJoPro Dec 9, 2024

Oh, I can't believe I completely missed that.
Let me go and re-install 🤦

I got that initial command from the MicroK8s docs and never noticed the slight difference.
https://microk8s.io/docs/how-to-nfs#install-the-csi-driver-for-nfs

Thank you @andyzhangx !

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CSI driver-smb mount the directory and immediately unmount despite no errors. #880

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 3 comments 11 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

CSI driver-smb mount the directory and immediately unmount despite no errors. #880

lanzmlt Dec 5, 2024

Replies: 3 comments · 11 replies

andyzhangx Dec 9, 2024 Maintainer

lanzmlt Dec 9, 2024 Author

andyzhangx Dec 9, 2024 Maintainer

lanzmlt Dec 9, 2024 Author

andyzhangx Dec 9, 2024 Maintainer

lanzmlt Dec 9, 2024 Author

andyzhangx Dec 9, 2024 Maintainer

lanzmlt Dec 9, 2024 Author

RobJohnson-RoJoPro Dec 9, 2024

andyzhangx Dec 9, 2024 Maintainer

RobJohnson-RoJoPro Dec 9, 2024

andyzhangx Dec 9, 2024 Maintainer

andyzhangx Dec 9, 2024 Maintainer

RobJohnson-RoJoPro Dec 9, 2024

lanzmlt
Dec 5, 2024

Replies: 3 comments 11 replies

andyzhangx
Dec 9, 2024
Maintainer

lanzmlt
Dec 9, 2024
Author

andyzhangx Dec 9, 2024
Maintainer

lanzmlt Dec 9, 2024
Author

andyzhangx Dec 9, 2024
Maintainer

lanzmlt Dec 9, 2024
Author

andyzhangx Dec 9, 2024
Maintainer

lanzmlt Dec 9, 2024
Author

RobJohnson-RoJoPro
Dec 9, 2024

andyzhangx Dec 9, 2024
Maintainer

andyzhangx Dec 9, 2024
Maintainer

andyzhangx Dec 9, 2024
Maintainer