Implement dynamic provision support

[leakingtapan]

Option 1

User create PVC, its creation will trigger the driver to create a NEW EFS filesystem. The storageclass will be defined as follows:

StorageClass

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: efs-sc
provisioner: efs.csi.aws.com
parameters:
  encrypted: "true",
  kmsKeyId: {keyId},
  performanceMode: "generalPurpose",
  provisionedThroughputInMibps: "512",
  throughputMode: "provisioned"

Open question

• How to configure the driver to create EFS mount targets for each availability with user defines subnets and security groups?

Option 2 (Preferred)

User creates PVC, its creation will trigger the driver to provision a subdirectory within an existing EFS filesystem as the volume. The storageclass will be defined as follows:

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: efs-sc
provisioner: efs.csi.aws.com
parameters:
  filesystemId: {EFS filesystem ID}

Provisioing EFS filesystem

Since we model the storageclass as the configuration for creating new directories inside existing EFS filesystem, user will need a different way to provision EFS system. This can be achieved by create a new service operator for EFS using service operator framework. User will need to create a custom resource that defines a EFS filesystem, and the operator will create the EFS filesystem by watching the custom resource.

For various configurations supported by EFS, eg performanceMode, throughputMode, kmsKeyId, etc, they will be a field within the EFS CRD.

Ref:

• https://docs.aws.amazon.com/sdk-for-go/api/service/efs/#CreateFileSystemInput
• https://docs.aws.amazon.com/sdk-for-go/api/service/efs/#CreateMountTargetInput

[ReillyTevera]

Is this issue supposed to cover automatically creating new EFS file systems based off of PVC requests? In that case can we cover the ability to set various characteristics of that file system based off of the PVC (probably via annotations)? Such as the ability to configure performance mode, throughput mode, life cycle, encryption etc.

[leakingtapan]

Is this issue supposed to cover automatically creating new EFS file systems based off of PVC requests?

Yes

In that case can we cover the ability to set various characteristics of that file system based off of the PVC (probably via annotations)?

Storageclass is the mechanism to pass filesystem configurations. See EBS's storageclass as an example. Do you have to use PVC annotation for this?

[ReillyTevera]

No, storageClass should be sufficient for us.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[leakingtapan]

/remove-lifecycle stale

[wongma7]

We'll also need to call CreateMountTarget: https://docs.aws.amazon.com/efs/latest/ug/API_CreateMountTarget.html . This will be trickier because the driver needs to know the VPC kubernetes nodes are in. It also needs to pick a subnet to add the mount target in. Optionally, it takes security group & IP address args. The security group must allow inbound NFS traffic, maybe the driver should be responsible for creating it too?

The driver could preemptively create mount targets in all zones and create a PV that points to the mount target's DNS name so that it's mountable from any zone.

Alternatively, the driver could take advantage of https://kubernetes.io/blog/2018/10/11/topology-aware-volume-provisioning-in-kubernetes/ to only create one mount target only for the zone a pod is scheduled to.

[leakingtapan]

punt this feature to after initial beta release

[ryanfaircloth]

The number of endpoints available is limited by the number of IPv4 addresses available in the vpc subnet. We really don't need a per endpoint per request we need a pattern that utilizes a folder structure within a single EFS instance

[leakingtapan]

Related to #63

[leakingtapan]

@rfaircloth-splunk @ReillyProcentive @wreed4

I updated the issue description with two options to tackle this problem. PTAL

[ReillyTevera]

I think Option 1 is the best. I think the best approach would be if StorageClass objects create EFS shares and PVC would create volumes in the specified StorageClasses provisioned EFS share.

This would allow for users who wish to have PVC volumes all use the same EFS volume (similar to how the external storage EFS provisioner works today) to do so while also allowing for creating a single EFS volume per PVC.

[whereisaaron]

Definitely Option 1. I think the more common pattern is PVCs are used for dynamic provisioning, not StorageClasses.

1. EFS filesystems should created using a CRD and an operator. I think this could be out of scope for the EFS driver and better handled by the AWS Service Operator that AWS is already working on.

2. Once you have an EFS filesystem created by the AWS Service Operator, one or more StorageClasses can be created for that EFS filesystem. Each storage class determines how paths in the EFS filesystem are dynamically allocates, named, restrictions on uid/gid, recycling, bit-rate limitations, AWS role used for mounting, NFS mount options etc.

3. The PVC's can dynamically create a volume for a StorageClass for an EFS filesystem. It may be possible for the PVC to optionally specify uid/gid, or a fixed path, path prefix, or sub-path, but only if the StorageClass allows that.

If the cluster admin creates two StorageClasses that allows PVC's to overlap each other, or conflict, that is their look-out 😄

[leakingtapan]

@whereisaaron If understand correctly, are you suggesting modeling EFS dynamic provisioning as creating volume from existing EFS filesystem? That's what being proposed as option 2. And option 1 is saying modeling dynamic provisioning as creating a new EFS filesystem.

[whereisaaron]

I guess I misunderstood the option 2 then @leakingtapan. It appeared to be using the StorageClass as the mechanism of dynamic provisioning rather than the PV? Or did it mean to say a PV would be created for an existing StorageClass like the example?

I think @ReillyProcentive is saying the same thing as me, and also said option 1, but I think that is because the examples are maybe confusing both of us.

[wreed4]

The way I read the options, neither were complete. Option one suggested the PVC creating a new filesystem for each PVC, but the storage class having the file system creation parameters. It suggested that the PVC use the storage class to create a new filesystem every time. Option two suggested the PVC creating sub-directories within the same filesystem, and the storage class not having file system creation parameters.

For what it's worth, I would like to see basically what @whereisaaron is proposing. The storage class defines the file system, you'd have one file system per storage class. Then the PVC dynamically provisions volumes within that filesystem defined by the storage class. That way, the user has control over which volumes go to which file system and both use-cases are satisfied.

Icing on the cake, for me, would be if the creation of the storageclass ALSO triggered a dynamic provisioning event, and created an EFS to match. It'd be a really handy feature to be able to entirely control your storage through the CSI driver and not have to also learn the AWS Service Operator. That way would require creating the EFS resource, then the storage class to match it instead of just the storage class. In this case, the storage class would have to support both dynamic and static provisioning (if the filesystem already exists (or an ID was provided), do nothing new).

[tyrken]

FWIW the old non-CSI EFS provisioner (https://github.com/kubernetes-incubator/external-storage/tree/master/aws/efs) works like Option 2, where the EFS file system, mount points & network security groups are all sorted out by "something else" first. In their case the necessary fs-xxxx id is passed in via a ConfigMap/container environment variable, rather than StorageClass parameters, but the effect is similar.

The provisioner then dynamically makes a sub-folder (normally with a unique group id) for each PVC that comes asking to keep them separate.

This seems nice & easy & so preferred to me, but I haven't used it much yet or thought hard about backups, security, etc.

If you do create file-systems (option 1), you will need to allow for controlling subnets & security groups for the mount points, & tagging also. Maybe option 2 is the goal for now & option 1 a later extension only if we can't get what we need (soon enough) with AWS Service Operator?

[leakingtapan]

Maybe option 2 is the goal for now & option 1 a later extension only if we can't get what we need (soon enough) with AWS Service Operator?

Yep. That's my intention as well.

[wreed4]

Is there any sort of timeline for when this feature will be available? We're having to do all kinds of workarounds because the csi driver does not dynamically provision volumes.

[jieyu]

EFS recently added support for Access Point:
https://docs.aws.amazon.com/efs/latest/ug/efs-access-points.html#enforce-identity-access-points

Maybe another option is to dynamic provision PVs by creating access points in an EFS filesystem (instead of creating subdirectories)?

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[leakingtapan]

/remove-lifecycle stale

[abdennour]

Can we use aws-efs-csi-driver in conjection with efs-provisioner for dynamic provisioning EFS shares.

User Create PVC --> EFS-Provisioner Creates new PV --> All PVs pointing to the same EFS resource but with different shares.

[alfredkrohmer]

We implemented this with a custom CSI driver that dynamically creates PVs in response to PVCs. The driver is only responsible for creating and binding the PVs; the actual mount is still performed by the AWS EFS CSI driver.

Here you go: https://github.com/LogMeIn/aws-efs-csi-pv-provisioner

I'd be happy to assist in adding this functionality directly into this driver.

[JonathanLachapelle]

We implemented this with a custom CSI driver that dynamically creates PVs in response to PVCs. The driver is only responsible for creating and binding the PVs; the actual mount is still performed by the AWS EFS CSI driver.

Here you go: https://github.com/LogMeIn/aws-efs-csi-pv-provisioner

I'd be happy to assist in adding this functionality directly into this driver.

That would be great, we are trying to build a multi-tenant cluster with EFS as the main statefull storage solution and without dynamic provisioning, it will be difficult to implement.

[gaurav-ackotech]

@devkid I used your repo code, but Lil bit confused I followed the steps till 6 As per your Read me. but not able to understand I need to deploy a PVC that is available under the deployment folder or the one which you pasted over readme.
or need to update As per below comment as per your read me ?
A corresponding PV called pvc-<UID of PVC> will be created bound to the PVC. This PV is utilizing the AWS EFS CSI driver. A subdirectory called <namespace of PVC>-<name of PVC>-<name of PV> will be created on the configured EFS file system:

Also some file you use storage class as efs and some efs-sc.

[alfredkrohmer]

@gaurav-ackotech The PV and PVC in the deploy/ folder are required for the provisioner itself (so that it can mount the root directory of the EFS into itself for management operations); these use the efs-sc storage class provided by the aws-efs-csi-driver itself (and that's why it has a PVC and a PV - the CSI driver alone doesn't provision any PVs). You can consider the PV and PVC part of the controller and don't really need to care about them much (except for substituting the FS ID).

The PVC in the Readme is an example for what you may want to use for your application; it has the efs storage class, so it will be handled by the provisioner, which will create a PV automatically to satisfy the claim - for this PVC a subfolder will be created on the EFS filesystem automatically.

[TBBle]

For visibility/cross-linking, aws-controllers-k8s/community#328 is the issue for EFS provisioning via the AWS Controllers for Kubernetes project. Once that is available, it would close the loop in terms of "doing it all in k8s".

It seems that per Option 2, the December 2019 discussion, and https://github.com/LogMeIn/aws-efs-csi-pv-provisioner, that the expectation is still that the administrator creates the CRD resource for ACK, and then extracts the fs-id from the object's status field and creates a StorageClass that points to it, for dynamic PVs to be created as subdirectories underneath.

So work on this would not be blocked by lack of aws-controllers-k8s/community#328, as Option 2 works if the administrator provisions EFS via the AWS Console and still just gets the right fs-id.

It might be feasible to implement a controller that auto-creates Storage Classes from EFS provisioning CRDs (or auto-creates EFS provisioning CRDs from Storage Classes) but I don't think that needs to be part of the EFS CSI Driver itself. It might make sense for ACK to do that, for example.