Pod cannot be deleted if the S3 PVC fails to mount

[geniass]

/kind bug

NOTE: If this is a filesystem related bug, please take a look at the Mountpoint repo to submit a bug report

What happened?
Mounting an s3 volume into a pod fails (FailedMount k8s event). Then the kubelet is unable to delete the pod because the vol_data.json file is missing. The pod hangs in the Terminating state forever.

Kubelet logs get flooded with:

Dec 05 10:46:44 gin k3s[3936577]: E1205 10:46:44.663100 3936577 reconciler_common.go:158] "operationExecutor.UnmountVolume failed (controllerAttachDetachEnabled true) for volume \"s3-mybucket-storage\" (UniqueName: \"kubernetes.io/csi/s3.csi.aws.com^s3-csi-driver-volume-mybucket-storage\") pod \"ee13ebc2-e424-47da-aa81-c8f627f69782\" (UID: \"ee13ebc2-e424-47da-aa81-c8f627f69782\") : UnmountVolume.NewUnmounter failed for volume \"s3-mybucket-storage\" (UniqueName: \"kubernetes.io/csi/s3.csi.aws.com^s3-csi-driver-volume-mybucket-storage\") pod \"ee13ebc2-e424-47da-aa81-c8f627f69782\" (UID: \"ee13ebc2-e424-47da-aa81-c8f627f69782\") : kubernetes.io/csi: unmounter failed to load volume data file [/var/lib/kubelet/pods/ee13ebc2-e424-47da-aa81-c8f627f69782/volumes/kubernetes.io~csi/s3-mybucket-storage/mount]: kubernetes.io/csi: failed to open volume data file [/var/lib/kubelet/pods/ee13ebc2-e424-47da-aa81-c8f627f69782/volumes/kubernetes.io~csi/s3-mybucket-storage/vol_data.json]: open /var/lib/kubelet/pods/ee13ebc2-e424-47da-aa81-c8f627f69782/volumes/kubernetes.io~csi/s3-mybucket-storage/vol_data.json: no such file or directory" err="UnmountVolume.NewUnmounter failed for volume \"s3-mybucket-storage\" (UniqueName: \"kubernetes.io/csi/s3.csi.aws.com^s3-csi-driver-volume-mybucket-storage\") pod \"ee13ebc2-e424-47da-aa81-c8f627f69782\" (UID: \"ee13ebc2-e424-47da-aa81-c8f627f69782\") : kubernetes.io/csi: unmounter failed to load volume data file [/var/lib/kubelet/pods/ee13ebc2-e424-47da-aa81-c8f627f69782/volumes/kubernetes.io~csi/s3-mybucket-storage/mount]: kubernetes.io/csi: failed to open volume data file [/var/lib/kubelet/pods/ee13ebc2-e424-47da-aa81-c8f627f69782/volumes/kubernetes.io~csi/s3-mybucket-storage/vol_data.json]: open /var/lib/kubelet/pods/ee13ebc2-e424-47da-aa81-c8f627f69782/volumes/kubernetes.io~csi/s3-mybucket-storage/vol_data.json: no such file or directory"

What you expected to happen?
S3 volume should be unmounted and the pod should terminate normally.

How to reproduce it (as minimally and precisely as possible)?
S3 PV and PVC:

apiVersion: v1
kind: PersistentVolume
metadata:
  name: s3-mybucket-storage
spec:
  accessModes:
  - ReadOnlyMany
  capacity:
    storage: 1200Gi
  claimRef:
    apiVersion: v1
    kind: PersistentVolumeClaim
    name: s3-mybucket-storage
    namespace: coder
  csi:
    driver: s3.csi.aws.com
    volumeAttributes:
      bucketName: mybucket-af-south-1-storage
    volumeHandle: s3-csi-driver-volume-mybucket-storage
  mountOptions:
  - region af-south-1
  - uid=1000
  - gid=1000
  - allow-other
  - read-only
  persistentVolumeReclaimPolicy: Retain
  volumeMode: Filesystem

---

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: s3-mybucket-storage
  namespace: coder
spec:
  accessModes:
  - ReadOnlyMany
  resources:
    requests:
      storage: 1200Gi
  storageClassName: ""
  volumeMode: Filesystem
  volumeName: s3-mybucket-storage

Create a pod that mounts an s3 pvc:

apiVersion: v1
kind: Pod
metadata:
  name: s3-mybucket-storage-pod
  namespace: coder
spec:
  containers:
  - name: s3-mybucket-storage-container
    image: busybox
    command: ["sh", "-c", "while true; do sleep 3600; done"]
    volumeMounts:
    - name: s3-mybucket-storage
      mountPath: /mnt/s3
  volumes:
  - name: s3-mybucket-storage
    persistentVolumeClaim:
      claimName: s3-mybucket-storage

kubectl -n coder describe pod s3-mybucket-storage-pod

There will be a Warning event that the s3 PVC could not be mounted:

   Warning  FailedMount  6s    kubelet            MountVolume.SetUp failed for volume "s3-teneo-storage" : rpc error: code = Internal desc = Could not mount "isazi-hudson-teneo-af-south-1-storage" at "/var/lib/kubelet/pods/3dc25df6-2e3e-4c79-800d-ec4ad50ca70c/volumes/kubernetes.io~csi/s3-teneo-storage/mount": Mount failed: Failed to start service output: Error: mount point /var/lib/kubelet/pods/3dc25df6-2e3e-4c79-800d-ec4ad50ca70c/volumes/kubernetes.io~csi/s3-teneo-storage/mount is already mounted Error: Failed to create mount process

Anything else we need to know?:

Environment

• Kubernetes version (use kubectl version):

Client Version: v1.30.6+k3s1
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.30.6+k3s1

• Driver version: v1.10.0