cephfs: use userid and keys for provisioning

[black-dragon74]

This patch modifies the code to use userID and
userKey for provisioning of both static and dynamic PVs.

In case user credentials are not found admin credentials are used as a fallback and for backwards compatibility.

Fixes: #4935

[black-dragon74]

Cluster Config

Ceph user

$ ceph auth get client.nick2
[client.nick2]
        key = AQCJHUdnHeDrGBAAd9/9Qc1orCwKwlRZLgsDeQ==
        caps mds = "allow r fsname=myfs path=/volumes, allow rws fsname=myfs path=/volumes/csi"
        caps mgr = "allow rw"
        caps mon = "allow r fsname=myfs"
        caps osd = "allow rw tag cephfs metadata=myfs, allow rw tag cephfs data=myfs"

Provisioner secret

# oc get secrets/rook-csi-cephfs-provisioner-user2 -o yaml
apiVersion: v1
data:
  userID: bmljazI=
  userKey: QVFDSkhVZG5IZURyR0JBQWQ5LzlRYzFvckN3S3dsUlpMZ3NEZVE9PQ==
kind: Secret
metadata:
  creationTimestamp: "2024-11-27T13:27:03Z"
  name: rook-csi-cephfs-provisioner-user2
  namespace: rook-ceph
  resourceVersion: "1722753"
  uid: 88222761-54a2-4eb0-9d2d-9c11326979a8
type: kubernetes.io/rook

Nodestage secret

# oc get secrets/rook-csi-cephfs-node-user2 -o yaml
apiVersion: v1
data:
  userID: bmljazI=
  userKey: QVFDSkhVZG5IZURyR0JBQWQ5LzlRYzFvckN3S3dsUlpMZ3NEZVE9PQ==
kind: Secret
metadata:
  creationTimestamp: "2024-11-27T13:27:03Z"
  name: rook-csi-cephfs-node-user2
  namespace: rook-ceph
  resourceVersion: "1722754"
  uid: 4e9525bd-4854-4cce-9007-58fd261c6c1a
type: kubernetes.io/rook

1. Dynamic PVCs

Resources

❯ oc get sc
NAME          PROVISIONER                     RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
rook-cephfs   rook-ceph.cephfs.csi.ceph.com   Delete          Immediate           true                   17m

❯ oc get pvc
NAME                STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   VOLUMEATTRIBUTESCLASS   AGE
cephfs-pvc          Bound    pvc-39a11e4c-2ddd-46c6-9b5a-6b004bd4eced   1Gi        RWO            rook-cephfs    <unset>                 18m

Logs

I1127 13:29:09.069933       1 utils.go:266] ID: 108 Req-ID: pvc-39a11e4c-2ddd-46c6-9b5a-6b004bd4eced GRPC call: /csi.v1.Controller/CreateVolume
I1127 13:29:09.077837       1 utils.go:267] ID: 108 Req-ID: pvc-39a11e4c-2ddd-46c6-9b5a-6b004bd4eced GRPC request: {"capacity_range":{"required_bytes":1073741824},"name":"pvc-39a11e4c-2ddd-46c6-9b5a-6b004bd4eced","parameters":{"clusterID":"rook-ceph","csi.storage.k8s.io/pv/name":"pvc-39a11e4c-2ddd-46c6-9b5a-6b004bd4eced","csi.storage.k8s.io/pvc/name":"cephfs-pvc","csi.storage.k8s.io/pvc/namespace":"rook-ceph","fsName":"myfs","pool":"myfs-replicated"},"secrets":"***stripped***","volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{"mode":7}}]}
I1127 13:29:09.170334       1 omap.go:89] ID: 108 Req-ID: pvc-39a11e4c-2ddd-46c6-9b5a-6b004bd4eced got omap values: (pool="myfs-metadata", namespace="csi", name="csi.volumes.default"): map[]
I1127 13:29:09.185399       1 omap.go:159] ID: 108 Req-ID: pvc-39a11e4c-2ddd-46c6-9b5a-6b004bd4eced set omap keys (pool="myfs-metadata", namespace="csi", name="csi.volumes.default"): map[csi.volume.pvc-39a11e4c-2ddd-46c6-9b5a-6b004bd4eced:595c630d-6e17-4c00-a66e-91785fb01c6d])
I1127 13:29:09.190423       1 omap.go:159] ID: 108 Req-ID: pvc-39a11e4c-2ddd-46c6-9b5a-6b004bd4eced set omap keys (pool="myfs-metadata", namespace="csi", name="csi.volume.595c630d-6e17-4c00-a66e-91785fb01c6d"): map[csi.imagename:csi-vol-595c630d-6e17-4c00-a66e-91785fb01c6d csi.volname:pvc-39a11e4c-2ddd-46c6-9b5a-6b004bd4eced csi.volume.owner:rook-ceph])
I1127 13:29:09.191264       1 fsjournal.go:318] ID: 108 Req-ID: pvc-39a11e4c-2ddd-46c6-9b5a-6b004bd4eced Generated Volume ID (0001-0009-rook-ceph-0000000000000001-595c630d-6e17-4c00-a66e-91785fb01c6d) and subvolume name (csi-vol-595c630d-6e17-4c00-a66e-91785fb01c6d) for request name (pvc-39a11e4c-2ddd-46c6-9b5a-6b004bd4eced)
I1127 13:29:09.470449       1 controllerserver.go:475] ID: 108 Req-ID: pvc-39a11e4c-2ddd-46c6-9b5a-6b004bd4eced cephfs: successfully created backing volume named csi-vol-595c630d-6e17-4c00-a66e-91785fb01c6d for request name pvc-39a11e4c-2ddd-46c6-9b5a-6b004bd4eced
I1127 13:29:09.472306       1 utils.go:273] ID: 108 Req-ID: pvc-39a11e4c-2ddd-46c6-9b5a-6b004bd4eced GRPC response: {"volume":{"capacity_bytes":1073741824,"volume_context":{"clusterID":"rook-ceph","fsName":"myfs","pool":"myfs-replicated","subvolumeName":"csi-vol-595c630d-6e17-4c00-a66e-91785fb01c6d","subvolumePath":"/volumes/csi/csi-vol-595c630d-6e17-4c00-a66e-91785fb01c6d/19ea74a6-2409-4220-b930-55deb650dc2a"},"volume_id":"0001-0009-rook-ceph-0000000000000001-595c630d-6e17-4c00-a66e-91785fb01c6d"}}

2. Static PVCs

Resources

❯ oc get pv
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                         STORAGECLASS   VOLUMEATTRIBUTESCLASS   REASON   AGE
cephfs-static-pv                           1Gi        RWX            Retain           Bound    rook-ceph/cephfs-static-pvc                  <unset>                          10m

❯ oc get pvc
NAME                STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   VOLUMEATTRIBUTESCLASS   AGE
cephfs-static-pvc   Bound    cephfs-static-pv                           1Gi        RWX                           <unset>                 10m

[Madhu-1]

It looks like there are more changes than expected in this file related to formatting, Do we need this change?

[black-dragon74]

I could revert the formatting changes. The md files inside charts are using the outdated syntax. Prettier auto formatted them and I decided to stick with it.

What would you suggest?

[Madhu-1]

i would suggest keeping the changes minimal and relevant to the PR as different developers might use different prettier configurations

[black-dragon74]

Done!