why sample-busybox-pod.yaml mountPath is hostpath?

[renweijun]

I have seen an issue #14 ，and I haven't got anything 。
sample-busybox-pod mountPath is not seaweedfs buckets，but pv have created 。
(my seaweedfs is 3.32, but seaweedfs-csi-driver is SeaweedFS version 30GB 3.29 linux amd64 )
sample-busybox-pod ：

root@my-csi-app:/# df -h
Filesystem Size Used Avail Use% Mounted on
overlay 432G 19G 414G 5% /
tmpfs 64M 0 64M 0% /dev
tmpfs 126G 0 126G 0% /sys/fs/cgroup
/dev/sda3 432G 19G 414G 5% /data
shm 64M 0 64M 0% /dev/shm
tmpfs 252G 12K 252G 1% /run/secrets/kubernetes.io/serviceaccount
tmpfs 126G 0 126G 0% /proc/acpi
tmpfs 126G 0 126G 0% /proc/scsi
tmpfs 126G 0 126G 0% /sys/firmware

sample-busybox-pod node is ops-k8s-node5：

root@ops-k8s-node5:~# ps -Afw --forest | grep weed
root 114612 116421 0 20:53 pts/0 00:00:00 _ grep --color=auto weed
root 80170 80089 0 13:38 ? 00:00:03 _ /csi-node-driver-registrar --v=5 --csi-address=/csi/csi.sock --kubelet-registration-path=/data/k8s/kubelet/data/plugins/seaweedfs-csi-driver/csi.sock
root 80225 80089 0 13:38 ? 00:00:04 _ /seaweedfs-csi-driver --endpoint=unix:///csi/csi.sock --filer=192.168.1.3:8888 --nodeid=ops-k8s-node5
root 85917 80225 0 13:42 ? 00:00:10 | _ weed -logtostderr=true mount -dirAutoCreate=true -umask=000 -dir=/data/k8s/kubelet/data/plugins/kubernetes.io/csi/pv/pvc-297b69c0-fab5-4f2f-bab6-e2ddfaa330de/globalmount -collection=pvc-297b69c0-fab5-4f2f-bab6-e2ddfaa330de -filer=192.168.1.3:8888 -filer.path=/buckets/pvc-297b69c0-fab5-4f2f-bab6-e2ddfaa330de -cacheCapacityMB=0 -localSocket=/tmp/seaweedfs-mount-24776101.sock -collectionQuotaMB=5120 -cacheDir=/tmp/pvc-297b69c0-fab5-4f2f-bab6-e2ddfaa330de -concurrentWriters=32
root 115019 80225 0 18:09 ? 00:00:04 | _ weed -logtostderr=true mount -dirAutoCreate=true -umask=000 -dir=/data/k8s/kubelet/data/plugins/kubernetes.io/csi/pv/pvc-5592ffea-8bae-49a5-b81a-45dd02e9bbff/globalmount -collection=pvc-5592ffea-8bae-49a5-b81a-45dd02e9bbff -filer=192.168.1.3:8888 -filer.path=/buckets/pvc-5592ffea-8bae-49a5-b81a-45dd02e9bbff -cacheCapacityMB=0 -localSocket=/tmp/seaweedfs-mount-61293285.sock -collectionQuotaMB=10240 -cacheDir=/tmp/pvc-5592ffea-8bae-49a5-b81a-45dd02e9bbff -concurrentWriters=32

seaweedfs mount root path
root@ops-master2:/mnt/seaweedfs/buckets# ll
total 0
drwxrwxrwx 1 root root 0 Oct 21 17:47 ./
drwxr-xr-x 1 root root 0 Oct 22 21:30 ../
drwxrwxrwx 1 root root 0 Oct 22 13:41 pvc-297b69c0-fab5-4f2f-bab6-e2ddfaa330de/
drwxrwxrwx 1 root root 0 Oct 22 18:09 pvc-5592ffea-8bae-49a5-b81a-45dd02e9bbff/
drwxrwxrwx 1 root root 0 Oct 22 17:49 pvc-95274ff1-1b7d-477d-ad90-ae8a01597ebe/
drwxrwxrwx 1 root root 0 Oct 21 17:47 pvc-d29d5259-f8f8-46a0-82a4-4746d9200c49/

[Pod: ]seaweedfs-node container csi-seaweedfs-plugin logs：

I1022 05:38:44.398255 main.go:38 connect to filer 192.168.1.3:8888
I1022 05:38:44.398540 driver.go:50 Driver: seaweedfs-csi-driver version: 1.0.0
I1022 05:38:44.398859 driver.go:99 Enabling volume access mode: MULTI_NODE_MULTI_WRITER
I1022 05:38:44.398867 driver.go:99 Enabling volume access mode: SINGLE_NODE_WRITER
I1022 05:38:44.398871 driver.go:99 Enabling volume access mode: SINGLE_NODE_MULTI_WRITER
I1022 05:38:44.398875 driver.go:99 Enabling volume access mode: SINGLE_NODE_SINGLE_WRITER
I1022 05:38:44.398879 driver.go:110 Enabling controller service capability: CREATE_DELETE_VOLUME
I1022 05:38:44.398885 driver.go:110 Enabling controller service capability: PUBLISH_UNPUBLISH_VOLUME
I1022 05:38:44.398889 driver.go:110 Enabling controller service capability: SINGLE_NODE_MULTI_WRITER
I1022 05:38:44.399206 server.go:92 Listening for connections on address: &net.UnixAddr{Name:"//csi/csi.sock", Net:"unix"}
I1022 05:42:24.159317 nodeserver.go:32 node stage volume pvc-297b69c0-fab5-4f2f-bab6-e2ddfaa330de to /data/k8s/kubelet/data/plugins/kubernetes.io/csi/pv/pvc-297b69c0-fab5-4f2f-bab6-e2ddfaa330de/globalmount
I1022 05:42:24.159716 mounter_seaweedfs.go:64 mounting [192.168.1.3:8888] /buckets/pvc-297b69c0-fab5-4f2f-bab6-e2ddfaa330de to /data/k8s/kubelet/data/plugins/kubernetes.io/csi/pv/pvc-297b69c0-fab5-4f2f-bab6-e2ddfaa330de/globalmount
I1022 05:42:24.159771 mounter.go:53 Mounting fuse with command: weed and args: [-logtostderr=true mount -dirAutoCreate=true -umask=000 -dir=/data/k8s/kubelet/data/plugins/kubernetes.io/csi/pv/pvc-297b69c0-fab5-4f2f-bab6-e2ddfaa330de/globalmount -collection=pvc-297b69c0-fab5-4f2f-bab6-e2ddfaa330de -filer=192.168.1.3:8888 -filer.path=/buckets/pvc-297b69c0-fab5-4f2f-bab6-e2ddfaa330de -cacheCapacityMB=0 -localSocket=/tmp/seaweedfs-mount-24776101.sock -collectionQuotaMB=5120 -cacheDir=/tmp/pvc-297b69c0-fab5-4f2f-bab6-e2ddfaa330de -concurrentWriters=32]
mount point owner uid=0 gid=0 mode=drwxrwxrwx
current uid=0 gid=0
I1022 05:42:24.210486 leveldb_store.go:47 filer store dir: /tmp/pvc-297b69c0-fab5-4f2f-bab6-e2ddfaa330de/79ec0ebe/meta
I1022 05:42:24.210511 file_util.go:23 Folder /tmp/pvc-297b69c0-fab5-4f2f-bab6-e2ddfaa330de/79ec0ebe/meta Permission: -rwxr-xr-x
This is SeaweedFS version 30GB 3.29 linux amd64
I1022 05:42:24.221444 nodeserver.go:83 volume pvc-297b69c0-fab5-4f2f-bab6-e2ddfaa330de successfully staged to /data/k8s/kubelet/data/plugins/kubernetes.io/csi/pv/pvc-297b69c0-fab5-4f2f-bab6-e2ddfaa330de/globalmount
I1022 05:42:24.227871 nodeserver.go:93 node publish volume pvc-297b69c0-fab5-4f2f-bab6-e2ddfaa330de to /data/k8s/kubelet/data/pods/dc580085-d006-48fc-b3dd-f16d130d5857/volumes/kubernetes.iocsi/pvc-297b69c0-fab5-4f2f-bab6-e2ddfaa330de/mount
I1022 05:42:24.229546 nodeserver.go:127 volume pvc-297b69c0-fab5-4f2f-bab6-e2ddfaa330de successfully published to /data/k8s/kubelet/data/pods/dc580085-d006-48fc-b3dd-f16d130d5857/volumes/kubernetes.iocsi/pvc-297b69c0-fab5-4f2f-bab6-e2ddfaa330de/mount
I1022 08:32:16.968660 nodeserver.go:93 node publish volume pvc-297b69c0-fab5-4f2f-bab6-e2ddfaa330de to /data/k8s/kubelet/data/pods/eb143160-d9d5-48d3-af88-a7406a74208e/volumes/kubernetes.iocsi/pvc-297b69c0-fab5-4f2f-bab6-e2ddfaa330de/mount
I1022 08:32:16.970269 nodeserver.go:127 volume pvc-297b69c0-fab5-4f2f-bab6-e2ddfaa330de successfully published to /data/k8s/kubelet/data/pods/eb143160-d9d5-48d3-af88-a7406a74208e/volumes/kubernetes.iocsi/pvc-297b69c0-fab5-4f2f-bab6-e2ddfaa330de/mount
I1022 08:53:23.955290 nodeserver.go:134 node unpublish volume pvc-297b69c0-fab5-4f2f-bab6-e2ddfaa330de from /data/k8s/kubelet/data/pods/eb143160-d9d5-48d3-af88-a7406a74208e/volumes/kubernetes.iocsi/pvc-297b69c0-fab5-4f2f-bab6-e2ddfaa330de/mount
I1022 08:53:24.459111 nodeserver.go:134 node unpublish volume pvc-297b69c0-fab5-4f2f-bab6-e2ddfaa330de from /data/k8s/kubelet/data/pods/eb143160-d9d5-48d3-af88-a7406a74208e/volumes/kubernetes.iocsi/pvc-297b69c0-fab5-4f2f-bab6-e2ddfaa330de/mount
W1022 08:53:24.459212 1 mount_helper_common.go:33] Warning: Unmount skipped because path does not exist: /data/k8s/kubelet/data/pods/eb143160-d9d5-48d3-af88-a7406a74208e/volumes/kubernetes.iocsi/pvc-297b69c0-fab5-4f2f-bab6-e2ddfaa330de/mount
I1022 08:53:25.464827 nodeserver.go:134 node unpublish volume pvc-297b69c0-fab5-4f2f-bab6-e2ddfaa330de from /data/k8s/kubelet/data/pods/eb143160-d9d5-48d3-af88-a7406a74208e/volumes/kubernetes.iocsi/pvc-297b69c0-fab5-4f2f-bab6-e2ddfaa330de/mount

sample-busybox-pod node is ops-k8s-node5：
root@ops-k8s-node5:# ll /data/k8s/kubelet/data/pods/eb143160-d9d5-48d3-af88-a7406a74208e/volumes/kubernetes.iocsi/pvc-297b69c0-fab5-4f2f-bab6-e2ddfaa330de/mount
total 12
drwxr-xr-x 2 root root 4096 Oct 22 16:49 ./
drwxr-x--- 3 root root 4096 Oct 22 16:32 ../
-rw-r--r-- 1 root root 2 Oct 22 16:49 rwj

I don't know why "Warning: Unmount skipped because path does not exist: " Please help guide !

My environment is a binary deployment:

ubuntu18.04
K8s 1.23.6
containerd 1.6.6

[renweijun]

csi-seaweedfs-plugin /usr/bin/weed update 3.32 “ unpublish volume” “Warning: Unmount skipped because path does not exist” disappear ，but sample-busybox-pod mountPath not seaweedfs buckets，is node /dev/sdb1 /data

root@my-csi-app:/# df -h
Filesystem Size Used Avail Use% Mounted on
overlay 4.3T 461G 3.8T 11% /
tmpfs 64M 0 64M 0% /dev
tmpfs 63G 0 63G 0% /sys/fs/cgroup
/dev/sdb1 4.3T 461G 3.8T 11% /data
shm 64M 0 64M 0% /dev/shm
tmpfs 126G 12K 126G 1% /run/secrets/kubernetes.io/serviceaccount
tmpfs 63G 0 63G 0% /proc/acpi
tmpfs 63G 0 63G 0% /proc/scsi
tmpfs 63G 0 63G 0% /sys/firmware
root@my-csi-app:/#

[Pod: ]seaweedfs-node container csi-seaweedfs-plugin logs：

I1022 15:32:12.782443 main.go:38 connect to filer 192.168.1.3:8888
--
Sat, Oct 22 2022 11:32:12 pm | I1022 15:32:12.783116 driver.go:50 Driver: seaweedfs-csi-driver version: 1.0.0
Sat, Oct 22 2022 11:32:12 pm | I1022 15:32:12.783349 driver.go:99 Enabling volume access mode: MULTI_NODE_MULTI_WRITER
Sat, Oct 22 2022 11:32:12 pm | I1022 15:32:12.783355 driver.go:99 Enabling volume access mode: SINGLE_NODE_WRITER
Sat, Oct 22 2022 11:32:12 pm | I1022 15:32:12.783358 driver.go:99 Enabling volume access mode: SINGLE_NODE_MULTI_WRITER
Sat, Oct 22 2022 11:32:12 pm | I1022 15:32:12.783360 driver.go:99 Enabling volume access mode: SINGLE_NODE_SINGLE_WRITER
Sat, Oct 22 2022 11:32:12 pm | I1022 15:32:12.783364 driver.go:110 Enabling controller service capability: CREATE_DELETE_VOLUME
Sat, Oct 22 2022 11:32:12 pm | I1022 15:32:12.783369 driver.go:110 Enabling controller service capability: PUBLISH_UNPUBLISH_VOLUME
Sat, Oct 22 2022 11:32:12 pm | I1022 15:32:12.783384 driver.go:110 Enabling controller service capability: SINGLE_NODE_MULTI_WRITER
Sat, Oct 22 2022 11:32:12 pm | I1022 15:32:12.783669 server.go:92 Listening for connections on address: &net.UnixAddr{Name:"//csi/csi.sock", Net:"unix"}
Sat, Oct 22 2022 11:33:47 pm | I1022 15:33:47.234582 nodeserver.go:32 node stage volume pvc-dc47cf3b-a37b-47aa-b4e8-fe8b0fe5e845 to /data/k8s/kubelet/data/plugins/kubernetes.io/csi/pv/pvc-dc47cf3b-a37b-47aa-b4e8-fe8b0fe5e845/globalmount
Sat, Oct 22 2022 11:33:47 pm | I1022 15:33:47.234983 mounter_seaweedfs.go:64 mounting [192.168.1.3:8888] /buckets/pvc-dc47cf3b-a37b-47aa-b4e8-fe8b0fe5e845 to /data/k8s/kubelet/data/plugins/kubernetes.io/csi/pv/pvc-dc47cf3b-a37b-47aa-b4e8-fe8b0fe5e845/globalmount
Sat, Oct 22 2022 11:33:47 pm | I1022 15:33:47.235031 mounter.go:53 Mounting fuse with command: weed and args: [-logtostderr=true mount -dirAutoCreate=true -umask=000 -dir=/data/k8s/kubelet/data/plugins/kubernetes.io/csi/pv/pvc-dc47cf3b-a37b-47aa-b4e8-fe8b0fe5e845/globalmount -collection=pvc-dc47cf3b-a37b-47aa-b4e8-fe8b0fe5e845 -filer=192.168.1.3:8888 -filer.path=/buckets/pvc-dc47cf3b-a37b-47aa-b4e8-fe8b0fe5e845 -cacheCapacityMB=0 -localSocket=/tmp/seaweedfs-mount-352286451.sock -collectionQuotaMB=5120 -cacheDir=/tmp/pvc-dc47cf3b-a37b-47aa-b4e8-fe8b0fe5e845 -concurrentWriters=32]
Sat, Oct 22 2022 11:33:47 pm | mount point owner uid=0 gid=0 mode=drwxrwxrwx
Sat, Oct 22 2022 11:33:47 pm | current uid=0 gid=0
Sat, Oct 22 2022 11:33:47 pm | I1022 15:33:47.289089 leveldb_store.go:47 filer store dir: /tmp/pvc-dc47cf3b-a37b-47aa-b4e8-fe8b0fe5e845/a8254986/meta
Sat, Oct 22 2022 11:33:47 pm | I1022 15:33:47.289122 file_util.go:23 Folder /tmp/pvc-dc47cf3b-a37b-47aa-b4e8-fe8b0fe5e845/a8254986/meta Permission: -rwxr-xr-x
Sat, Oct 22 2022 11:33:47 pm | This is SeaweedFS version 8000GB 3.32 758d70bdc09834066cfd835d0041f38ac77a1d42 linux amd64
Sat, Oct 22 2022 11:33:47 pm | I1022 15:33:47.296466 nodeserver.go:83 volume pvc-dc47cf3b-a37b-47aa-b4e8-fe8b0fe5e845 successfully staged to /data/k8s/kubelet/data/plugins/kubernetes.io/csi/pv/pvc-dc47cf3b-a37b-47aa-b4e8-fe8b0fe5e845/globalmount
Sat, Oct 22 2022 11:33:47 pm | I1022 15:33:47.302415 nodeserver.go:93 node publish volume pvc-dc47cf3b-a37b-47aa-b4e8-fe8b0fe5e845 to /data/k8s/kubelet/data/pods/cc978f8d-76d1-468a-abc6-04e6b3508104/volumes/kubernetes.io~csi/pvc-dc47cf3b-a37b-47aa-b4e8-fe8b0fe5e845/mount
Sat, Oct 22 2022 11:33:47 pm | I1022 15:33:47.303739 nodeserver.go:127 volume pvc-dc47cf3b-a37b-47aa-b4e8-fe8b0fe5e845 successfully published to /data/k8s/kubelet/d

[renweijun]

use helm install is ok ! I compared the differences

- name: pods-mount-dir
     mountPath: /var/lib/kubelet/pods
     mountPropagation: "Bidirectional"
- name: pods-mount-dir
     hostPath:
        path: /data/k8s/kubelet/data/pods/
        type: Directory

modify

 - name: pods-mount-dir
       mountPath:  /data/k8s/kubelet/data/pods/
       mountPropagation: "Bidirectional"

Bidirectional - This volume mount behaves the same the HostToContainer mount. In addition, all volume mounts created by the container will be propagated back to the host and to all containers of all pods that use the same volume.

[chrislusf]

thanks for debugging! I am not familiar with this part. Please create a PR if you find the bug.

[renweijun]

@chrislusf
I don't understand this parameter very well. This problem occurs（mountPropagation: "Bidirectional"）
You can prompt in readme deploy/kubernetes/seaweedfs-csi.yaml

          volumeMounts:
            - name: pods-mount-dir
              mountPath: {{ .Values.node.volumes.pods_mount_dir }}
              mountPropagation: "Bidirectional"

OR deploy/helm/seaweedfs-csi-driver/values.yaml change first and then To generate an up to date manifest from the helm chart
helm template seaweedfs ./deploy/helm/seaweedfs-csi-driver > deploy/kubernetes/seaweedfs-csi.yaml

[chrislusf]

I am not sure about the exact change. Create a PR?

[renweijun]

done