PVC volumeMode Block Failure

[ElectroshockGuy]

• To reproduce:
  • Create PVC Select volumeDevices as Block
  • Pod uses volumeDevices to mount the PVC.
• What is happening:
  • Pod cannot successfully mount the PVC
• Additional info:

1. FailedMapVolume:

MapVolume.MapBlockVolume failed for volume "pvc-02ad6940-8a09-4066-85da-71858d6caf55" : blkUtil.MapDevice failed. devicePath: /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/publish/pvc-02ad6940-8a09-4066-85da-71858d6caf55/a19e07f8-bf53-4125-8b84-f3868da363c5, globalMapPath:/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/pvc-02ad6940-8a09-4066-85da-71858d6caf55/dev, podUID: a19e07f8-bf53-4125-8b84-f3868da363c5, bindMount: true: failed to bind mount devicePath: /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/publish/pvc-02ad6940-8a09-4066-85da-71858d6caf55/a19e07f8-bf53-4125-8b84-f3868da363c5 to linkPath /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/pvc-02ad6940-8a09-4066-85da-71858d6caf55/dev/a19e07f8-bf53-4125-8b84-f3868da363c5: mount failed: exit status 32 Mounting command: mount Mounting arguments: -o bind /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/publish/pvc-02ad6940-8a09-4066-85da-71858d6caf55/a19e07f8-bf53-4125-8b84-f3868da363c5 /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/pvc-02ad6940-8a09-4066-85da-71858d6caf55/dev/a19e07f8-bf53-4125-8b84-f3868da363c5 Output: mount: /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/pvc-02ad6940-8a09-4066-85da-71858d6caf55/dev/a19e07f8-bf53-4125-8b84-f3868da363c5: mount point is not a directory. dmesg(1) may have more information after failed mount system call.

2. FailedMount:

Unable to attach or mount volumes: unmounted volumes=[seaweedfs-block-1], unattached volumes=[seaweedfs-block-1 seaweedfs-fs kube-api-access-5x7d7]: timed out waiting for the condition

pvc:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: seaweedfs-block-1
  namespace: seaweedfs
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 15Gi
  storageClassName: seaweedfs-storage
  volumeMode: Block

pod:

apiVersion: v1
kind: Pod
metadata:
  name: test-79676d6bf7-bjk96
  namespace: seaweedfs
spec:
  containers:
    - image: docker.io/rancher/mirrored-library-nginx:1.21.1-alpine
      imagePullPolicy: IfNotPresent
      name: container-0
      resources: {}
      securityContext:
        allowPrivilegeEscalation: true
        capabilities: {}
        privileged: true
        readOnlyRootFilesystem: false
        runAsNonRoot: false
      terminationMessagePath: /dev/termination-log
      terminationMessagePolicy: File
      volumeDevices:
        - devicePath: /dev/block1
          name: seaweedfs-block-1
  restartPolicy: Always
  volumes:
    - name: seaweedfs-block-1
      persistentVolumeClaim:
        claimName: seaweedfs-block-1

Weed version:

version 30GB 3.63 linux arm64

Images used:

image: registry.k8s.io/sig-storage/csi-provisioner:v3.5.0
image: registry.k8s.io/sig-storage/csi-resizer:v1.8.0
image: registry.k8s.io/sig-storage/csi-attacher:v4.3.0
image: registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.8.0
image: registry.k8s.io/sig-storage/livenessprobe:v2.10.0
image: chrislusf/seaweedfs-csi-driver:latest [sha256:aaf0ffabaf51e014469a5f0794d077eace49ba3872485442bde6116e9c245e28 arm64]

[chrislusf]

I do not see PersistentVolume defined.

[ElectroshockGuy]

Sorry, I forgot this important information, this is PV

apiVersion: v1
kind: PersistentVolume
metadata:
  annotations:
    pv.kubernetes.io/provisioned-by: seaweedfs-csi-driver
    volume.kubernetes.io/provisioner-deletion-secret-name: ''
    volume.kubernetes.io/provisioner-deletion-secret-namespace: ''
  creationTimestamp: '2024-03-12T07:35:56Z'
  finalizers:
    - kubernetes.io/pv-protection
    - external-attacher/seaweedfs-csi-driver
  name: pvc-02ad6940-8a09-4066-85da-71858d6caf55
  resourceVersion: '6273947'
  uid: 686635ad-090b-40fe-b4ea-a912215dd1b9
spec:
  accessModes:
    - ReadWriteOnce
  capacity:
    storage: 15Gi
  claimRef:
    apiVersion: v1
    kind: PersistentVolumeClaim
    name: seaweedfs-block-1
    namespace: seaweedfs
    resourceVersion: '6273452'
    uid: 02ad6940-8a09-4066-85da-71858d6caf55
  csi:
    driver: seaweedfs-csi-driver
    volumeAttributes:
      storage.kubernetes.io/csiProvisionerIdentity: 1710228321411-4743-seaweedfs-csi-driver
    volumeHandle: pvc-02ad6940-8a09-4066-85da-71858d6caf55
  persistentVolumeReclaimPolicy: Delete
  storageClassName: seaweedfs-storage
  volumeMode: Block
status:
  phase: Bound

[chrislusf]

I saw you already did a lot of work on this. But the yaml files are beyond my knowledge. The root cause may be in some yaml files not shown here.

[ElectroshockGuy]

Hi, I would like to confirm if seaweedfs-csi-driver supports block pvc?

[chrislusf]

Not supporting block storage.