[Fargate] [EFS]: Dynamic EFS mounting

[JamesTheArchitect]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Tell us about your request
We need to be able to specify EFS mount points / volumes dynamically when we start a Task on a Fargate ECS cluster. When Running a Task Definition, there are overrides and I propose that mount points / volumes (or at least rootDirectory) be allowed to be overridden so that we can isolate EFS folders at runtime.

Which service(s) is this request for?
This is for Fargate and EFS on ECS.

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?
We are setting up containers for various tools and are providing source data for those tools. We run the tools under a specific folder structure and we need to keep data isolated. We need the ability to mount EFS volumes at specific folders. This could be done by creating new Access Points and Task Definitions for each job, but with thousands of jobs monthly, that is not viable. The only workaround (below) exposes source data and tool workspaces to all tools if they walked the /mnt/ folders.

Are you currently working around this issue?
Our current solution is to mount to root folders and then use Symbolic Links on the mounted folders to a defined structure.
For example:
Mounts in Task Definition:
EFS Access Point for /automation/content -> /mnt/.base/content as READONLY
EFS Access Point for /automation/tasks -> /mnt/.base/tasks as READWRITE

Symbolic Links:
/mnt/.base/content/$JobID/ -> /mnt/efs/content
/mnt/.base/tasks/$TaskID/ -> /mnt/efs/workspace

This does NOT isolate the data and workspaces of other jobs, but it does obfuscate them.

Additional context
We can't use mount inside the containers as it is a privileged action. There is another issue open to allow Fargate to use this, but I'd like to avoid that if possible and just have greater control over specifying my mount points at runtime.
Issue #1000

Workaround Example
In the Task Definition:

"mountPoints":[
{ "containerPath": "/mnt/.base/content", "sourceVolume": "content", "readOnly": true },
{ "containerPath": "/mnt/.base/tasks", "sourceVolume": "tasks"}]

"volumes": [{
"efsVolumeConfiguration": {
"fileSystemId": "fs-xxxxx",
"authorizationConfig": { "accessPointId": "fsap-xxxxxx" },
"transitEncryption": "ENABLED",
"rootDirectory": "/"},
"name": "content",
},{
"efsVolumeConfiguration": {
"fileSystemId": "fs-xxxxx",
"authorizationConfig": {"accessPointId": "fsap-xxxxx"},
"transitEncryption": "ENABLED",
"rootDirectory": "/"},
"name": "tasks",
}]

In the Entrypoint for my container:
ln -s /mnt/.base/content/$JobID/ /mnt/efs/content
ln -s /mnt/.base/tasks/$TaskID/ /mnt/efs/workspace

[AndrewHannigan]

+1

[JoeDupuis]

+1 we are running customer code inside the task and need to scope efs access to a specific customer owned folder.
We won't be able to move from our custom docker instance without this feature.