Problems with XFS volumes on some provisioners

[vadimgusev-codefresh]

Hi, initially I've encountered this problem with our proprietary storage provisioner after switching from AL2 AMIs to Bottlerocket, but I was able to replicate the issue on the mainstream EKS provisioners.

Those are 2 StorageClasses I've defined:

# WORKS
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: ebs-csi-xfs
parameters:
  type: gp3
  fsType: xfs
provisioner: ebs.csi.aws.com
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer
---
# DOESN'T WORK
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: gp2-xfs
parameters:
  type: gp2
  fsType: xfs
provisioner: kubernetes.io/aws-ebs
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer

When I spin up pod+pvc referring those SCs, the ebs-csi-xfs works just fine while gp2-xfs throws me an error in pod events:

Warning  FailedMount             2s (x3 over 5s)    kubelet 
MountVolume.MountDevice failed for volume "pvc-50cf42d9-2bec-4d95-acde-950400d044ba" : format of disk "/dev/nvme2n1" failed: type:("xfs") target:("/var/lib/kubelet/plugins/kubernetes.io/aws-ebs/mounts/aws/us-east-1a/vol-0636ee72044526891") options:("defaults") errcode:(executable file not found in $PATH) output:()

as if it expects host machine to have mkfs.xfs or similar tool to format the partition.

What can I do to further investigate and hopefully resolve the issue? Thanks!

[bcressey]

This is an expected difference between out-of-tree CSI storage drivers (first example) and in-tree storage drivers (second example). Bottlerocket doesn't include xfsprogs, so any attempt to use the in-tree drivers will fail when kubelet tries to run mkfs.xfs.

Given the CSI migration effort in the wider ecosystem - do you actually need the in-tree approach to work? Long-term I'd like to see everything move to CSI.

In #1147 I added e2fsprogs to make the local volume static provisioner work, which had the side effect of enabling the in-tree storage driver to work for ext4 filesystems. So to be clear, I'm not opposed to adding XFS support in the base OS. It's the default filesystem for AL2, and it's reasonable to support it. I just want to be sure that there's not an easy "just use CSI" path to take instead.

[vadimgusev-codefresh]

Thanks for your clarification, things now make more sense!

As I mentioned in the beginning, I've first noticed this issue on our proprietary volume provisioner. Turns out it's just a proxy with custom volume reuse logic and the backend that implements actual volume operations. I was able to switch the backend to ebs.csi.aws.com and follow your "just use CSI" advice :)

So it's up to maintainers to decide whether to include xfsprogs into the distribution or not, as for my case it's no longer needed.