Add static & dynamic provisioning options for using efs-utils crossac…

[seanzatzdev-amazon]

…count mount option for cross-account AZ mapping between client instance and mount target

Background
efs-utils v1.36.0 added a new feature to allow for AZ mapping between the client instance and EFS mount target over cross-account mounts. This means that, with the proper setup, users can ensure their client instances are in the same physical AZ as their EFS mount target, which has been a pain-point for users in the past.

Is this a bug fix or adding new feature?
This PR adds a new feature.
What is this PR about? / Why do we need it?
This PR adds the option for EFS CSI Driver users to ensure their client instances & EFS mount targets are in the same physical AZ when doing cross-account static & dynamic provisioning. This was impossible to ensure before, and led to unnecessary cross-AZ costs as well as lack of assurance of availability for users' services.
What testing is done?
unit/e2e, manual

#DYNAMIC PROVISIONING 
[zatzsea@dev-dsk-zatzsea-1a-5f552df4 eks-dynamic-prov]$ kubectl create secret generic x-account --namespace=kube-system --from-literal=awsRoleArn='arn:aws:iam::673675429741:role/EFSCrossAccountAZMapRole' --from-literal=crossaccount='true'
secret/x-account created
[zatzsea@dev-dsk-zatzsea-1a-5f552df4 eks-dynamic-prov]$ kubectl apply -f storageclass.yaml 
storageclass.storage.k8s.io/efs-sc created
[zatzsea@dev-dsk-zatzsea-1a-5f552df4 eks-dynamic-prov]$ kubectl apply -f pod.yaml 
persistentvolumeclaim/efs-claim created
pod/efs-app created
[zatzsea@dev-dsk-zatzsea-1a-5f552df4 eks-dynamic-prov]$ kubectl get pod -A
NAMESPACE     NAME                                  READY   STATUS    RESTARTS        AGE
default       efs-app                               1/1     Running   0               22s
kube-system   aws-node-6wwzc                        2/2     Running   10 (7d1h ago)   41d
kube-system   aws-node-wvvkk                        2/2     Running   10 (7d1h ago)   41d
kube-system   coredns-6787556b84-d9pfd              1/1     Running   5 (7d1h ago)    41d
kube-system   coredns-6787556b84-fgz9t              1/1     Running   5 (7d1h ago)    41d
kube-system   efs-csi-controller-7b588877b5-grtx5   3/3     Running   0               2m3s
kube-system   efs-csi-controller-7b588877b5-ljh2m   3/3     Running   0               2m3s
kube-system   efs-csi-node-m8ssm                    3/3     Running   0               2m2s
kube-system   efs-csi-node-tw6qf                    3/3     Running   0               2m2s
kube-system   kube-proxy-rdgpk                      1/1     Running   5 (7d1h ago)    41d
kube-system   kube-proxy-wzjsk                      1/1     Running   5 (7d1h ago)    41d
[zatzsea@dev-dsk-zatzsea-1a-5f552df4 eks-dynamic-prov]$ kubectl exec --stdin --tty efs-app /bin/bash
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.
[root@efs-app /]# cat data/out 
Wed Mar 27 17:04:32 UTC 2024
Wed Mar 27 17:04:37 UTC 2024
Wed Mar 27 17:04:42 UTC 2024
Wed Mar 27 17:04:47 UTC 2024
Wed Mar 27 17:04:52 UTC 2024
Wed Mar 27 17:04:57 UTC 2024
Wed Mar 27 17:05:02 UTC 2024
Wed Mar 27 17:05:07 UTC 2024
Wed Mar 27 17:05:12 UTC 2024
Wed Mar 27 17:05:17 UTC 2024
Wed Mar 27 17:05:22 UTC 2024
Wed Mar 27 17:05:27 UTC 2024
Wed Mar 27 17:05:32 UTC 2024
Wed Mar 27 17:05:37 UTC 2024
[root@efs-app /]# echo $(date)
Wed Mar 27 17:05:53 UTC 2024
[root@efs-app /]# exit
exit
[zatzsea@dev-dsk-zatzsea-1a-5f552df4 eks-dynamic-prov]$ kubectl delete -f pod.yaml 
persistentvolumeclaim "efs-claim" deleted
pod "efs-app" deleted
[zatzsea@dev-dsk-zatzsea-1a-5f552df4 eks-dynamic-prov]$ kubectl delete -f storageclass.yaml 
storageclass.storage.k8s.io "efs-sc" deleted
[zatzsea@dev-dsk-zatzsea-1a-5f552df4 eks-dynamic-prov]$ kubectl delete secret x-account -n kube-system
secret "x-account" deleted
[zatzsea@dev-dsk-zatzsea-1a-5f552df4 eks-dynamic-prov]$ cd ../eks
eks-dynamic-prov/ eks-static-prov/  
[zatzsea@dev-dsk-zatzsea-1a-5f552df4 eks-dynamic-prov]$ cd ../eks-static-prov/


# STATIC PROVISIONING
[zatzsea@dev-dsk-zatzsea-1a-5f552df4 eks-static-prov]$ kubectl apply -f storageclass.yaml 
storageclass.storage.k8s.io/efs-sc created
[zatzsea@dev-dsk-zatzsea-1a-5f552df4 eks-static-prov]$ cat pv.yaml 
apiVersion: v1
kind: PersistentVolume
metadata:
  name: efs-pv
spec:
  capacity:
    storage: 5Gi
  volumeMode: Filesystem
  accessModes:
    - ReadWriteOnce
  storageClassName: efs-sc
  persistentVolumeReclaimPolicy: Retain
  csi:
    driver: efs.csi.aws.com
    volumeHandle: fs-0c5a0e9cd917e2c9a
    volumeAttributes:
      crossaccount: "true"
[zatzsea@dev-dsk-zatzsea-1a-5f552df4 eks-static-prov]$ kubectl apply -f pv.yaml 
persistentvolume/efs-pv created
[zatzsea@dev-dsk-zatzsea-1a-5f552df4 eks-static-prov]$ kubectl apply -f claim.yaml 
persistentvolumeclaim/efs-claim created
[zatzsea@dev-dsk-zatzsea-1a-5f552df4 eks-static-prov]$ kubectl get pod -A
NAMESPACE     NAME                                  READY   STATUS    RESTARTS        AGE
kube-system   aws-node-6wwzc                        2/2     Running   10 (7d1h ago)   41d
kube-system   aws-node-wvvkk                        2/2     Running   10 (7d1h ago)   41d
kube-system   coredns-6787556b84-d9pfd              1/1     Running   5 (7d1h ago)    41d
kube-system   coredns-6787556b84-fgz9t              1/1     Running   5 (7d1h ago)    41d
kube-system   efs-csi-controller-7b588877b5-grtx5   3/3     Running   0               5m51s
kube-system   efs-csi-controller-7b588877b5-ljh2m   3/3     Running   0               5m51s
kube-system   efs-csi-node-m8ssm                    3/3     Running   0               5m50s
kube-system   efs-csi-node-tw6qf                    3/3     Running   0               5m50s
kube-system   kube-proxy-rdgpk                      1/1     Running   5 (7d1h ago)    41d
kube-system   kube-proxy-wzjsk                      1/1     Running   5 (7d1h ago)    41d
[zatzsea@dev-dsk-zatzsea-1a-5f552df4 eks-static-prov]$ kubectl apply -f pod.yaml 
pod/efs-app created
[zatzsea@dev-dsk-zatzsea-1a-5f552df4 eks-static-prov]$ kubectl get pod -A
NAMESPACE     NAME                                  READY   STATUS    RESTARTS        AGE
default       efs-app                               1/1     Running   0               3s
kube-system   aws-node-6wwzc                        2/2     Running   10 (7d1h ago)   41d
kube-system   aws-node-wvvkk                        2/2     Running   10 (7d1h ago)   41d
kube-system   coredns-6787556b84-d9pfd              1/1     Running   5 (7d1h ago)    41d
kube-system   coredns-6787556b84-fgz9t              1/1     Running   5 (7d1h ago)    41d
kube-system   efs-csi-controller-7b588877b5-grtx5   3/3     Running   0               6m3s
kube-system   efs-csi-controller-7b588877b5-ljh2m   3/3     Running   0               6m3s
kube-system   efs-csi-node-m8ssm                    3/3     Running   0               6m2s
kube-system   efs-csi-node-tw6qf                    3/3     Running   0               6m2s
kube-system   kube-proxy-rdgpk                      1/1     Running   5 (7d1h ago)    41d
kube-system   kube-proxy-wzjsk                      1/1     Running   5 (7d1h ago)    41d
[zatzsea@dev-dsk-zatzsea-1a-5f552df4 eks-static-prov]$ kubectl exec --stdin --tty efs-app /bin/bash
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.
[root@efs-app /]# echo $(date)
Wed Mar 27 17:09:09 UTC 2024
[root@efs-app /]# cat data/out.txt 
Wed Mar 27 17:08:50 UTC 2024
Wed Mar 27 17:08:55 UTC 2024
Wed Mar 27 17:09:00 UTC 2024
Wed Mar 27 17:09:05 UTC 2024
Wed Mar 27 17:09:10 UTC 2024
[root@efs-app /]# exit
exit

[seanzatzdev-amazon]

/ok-to-test

[seanzatzdev-amazon]

/lgtm

[k8s-ci-robot]

@seanzatzdev-amazon: you cannot LGTM your own PR.

In response to this:

/lgtm

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[mskanth972]

/lgtm

[mskanth972]

/approve

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: mskanth972, seanzatzdev-amazon

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [mskanth972,seanzatzdev-amazon]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment