[Fargate] [request]: Allow privileged mode

[biltongza]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Tell us about your request
I want to be able to mount filesystems within my containers, and for that I need to be able to use privileged mode or add capabilities to the docker container.

Which service(s) is this request for?
Fargate

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?
Specifically, I am trying to mount an overlay in a container. This requires the container be run in privileged mode.

Are you currently working around this issue?
No, there does not appear to be a workaround.

[ecliptik]

We are interested in this since we are using Gitlab Kubernetes Runners for our CI/CD pipelines and using Docker-in-Docker to build application containers and push them to ECR.

Currently we have larger ec2 nodes that are tainted to only run Gitlab runner pods to avoid resource contention with other applications/service on the EKS clusters. These nodes are only used when CI/CD jobs are in us, and are idle for periods of time. Autoscaling them is also a challenge when many jobs are running at once since cold-starts of EC2 instances can take 2-3 minutes to join the cluster.

By using Fargate Profiles, we would like to setup the gitlab namespace so that any job pods created there will run on their own Fargate node instead of a static ec2 node. This would allow us greater resource flexibility, cost optimization, and separation of jobs to avoid resource contention.

We can do some of our jobs in Fargate like this, but a large majority of our jobs are using docker and docker-compose to build images and run CI/CD tests and would require privileged mode and volume sharing of the Docker socket.

[tata9001]

All the steps are dockerized in my company, so there is no way to use fargate as the CI/CD agent.
We are really looking forward to have this ability to run docker in docker on the Fargate platform.

[Waples]

Bump so this not being forgotten. I need to get priv mode working on our current AWS EKS Fargate CI/CD GitLab PoC.

[sandan]

+1 For EKS Fargate. We'd like more capabilities to be supported (specifically SYS_PTRACE, DAC_READ_SEARCH) to a container's securityContext. So far it looks like these work:

        securityContext:
          capabilities:
            drop:
              - all
            add:
              - SETPCAP
              - SETUID
              - SETGID

There's an open request for SYS_PTRACE: #1102.

[jonrau1]

+1 - Open Distro requires Privileged containers to run on EKS.

[FANMixco]

s3fs requires privileged also to sync folders:

s3fs-fuse/s3fs-fuse#1246

[binarynate]

+1 for supporting privileged or dockerSecurityOptions . I want to run Chromium in Fargate, but privileged or dockerSecurityOptions is required to enable Chromium's app sandbox.

[amunhoz]

+1

[bradynotarize]

Required for any container using s3fs-fuse.

[andymac4182]

Looking at running OpenVSCode in Fargate but need this to be able to run containers inside so docker-in-docker. https://github.com/gitpod-io/openvscode-server

[mayurvin]

I need the privileged mode on fargate with ECS so that I can install and run a docker daemon. At this time, docker daemon cannot be started unless the task is being run with the --privileged mode.
Please add this capability as soon as possible.

[nielsvanoosterom-varias]

+1 for this.

[jarredkenny]

I would like to be able to use docker-forticlient as a base image for a Fargate task so that my task can reach private resources behind a VPN without a 24/7 IPSec tunnel on a VPC.

https://github.com/HybirdCorp/docker-forticlient

[askkhan84]

Need this for DinD to run Github self hosted runners on Fargate

[tekknolagi]

+1 to use nsjail inside Fargate

[ecliptik]

Any updates on this? I've tried using buildah, podman, rootless, and nerdctl on fargate but none of them work.

Alternative is to bring up a separate docker build host and expose it over TCP, but that has major operational and security concerns.

[mreferre]

Any updates on this? I've tried using buildah, podman, rootless, and nerdctl on fargate but none of them work.

Alternative is to bring up a separate docker build host and expose it over TCP, but that has major operational and security concerns.

Can this help? Would Kaniko be a viable alternative for you? https://aws.amazon.com/blogs/containers/building-container-images-on-amazon-ecs-on-aws-fargate/

[pditommaso]

I think the main use case for this request is not to build containers with Fargate, but to be able to mount a file system via FUSE in fargate task. From the first comment:

I want to be able to mount filesystems within my containers, and for that I need to be able to use privileged mode or add capabilities to the docker container.

[mreferre]

Fair enough. I am not sure I'd characterize that as the "main use case" but definitely it was the initial theme of this issue and we should respect that (even though people added additional reasons for needing privileged access, including builds, so it diluted a bit the initial intent).

[ecliptik]

Would Kaniko be a viable alternative for you?

Kaniko is a no-go for us as it adds additional complexity into our Gitlab CI/CD pipelines.

I think the main use case for this request is not to build containers with Fargate, but to be able to mount a file system via FUSE in fargate task. From the first comment:

Our main request is to build container images using Fargate. We've raised this with AWS Enterprise Support as well, specifically for running Gitlab Runners in Fargate and building and pushing container images to ECR.

[anjalichaudhary]

Looking at running OpenVSCode in Fargate but need this to be able to run containers inside so docker-in-docker. https://github.com/gitpod-io/openvscode-server

@andymac4182 Were you able to find a solution for this? If yes, could you please share your approach.

[dasanjaneyuludarla]

+1

[andymac4182]

@anjalichaudhary Ended up just using Gitpod online. Would of preferred Fargate but it doesn't work :(

[nicodp-leap]

+1 for supporting privileged or dockerSecurityOptions . I want to run Chromium in Fargate, but privileged or dockerSecurityOptions is required to enable Chromium's app sandbox.

Did you find any solution? I'm having the same problem.

[binarynate]

@nicodp-leap I haven't found a solution, so I'm running Chromium on EC2 instead of Fargate. I would still love to be able to switch it over to Fargate, though.

[acidjazz]

+1 for this, need to use s3fuse

[FearlessHyena]

It's been almost 3 years since this was opened. Is it at least being considered?

[ashish0fficial]

Hi +1 for this -- many images wants to be run in privileged mode true or either tell us better solution for this

[davies]

+1 for this to use juicefs in containers, which turns S3 into a shared disk but requires FUSE.

[luthfimasruri]

Any solution for this?

[S0LERA]

+1 We need to use Rancher, that requires --privileged

[hilary-b]

+1 for building docker images

[junekhan]

+1 also for building docker images. We applied Kaniko as a makeshift, but now we need to build multi-arch docker images which Kaniko can't do.

[ClaytonOlleyNutrien]

Doesn't privileged mode basically mean root access to the host machine? On a cloud resource you don't "own" or manage directly? Maybe I'm missing something but I can't see how we'll ever get root level access to a Fargate machine.

[omeid]

@ClaytonOlleyNutrien The host machine in question for Fargate containers is a virtual machine, not the underlying physical hardware that runs the VM.

[esamattis]

Those who are interested in running jails or sandbox processes (ping @tekknolagi, @jimmybergman) you might not actually need privileged containers but just an ability to create user namespaces which normally do not require any extra permissions. Unprivileged user namespaces are the canonical way to create rootless sandboxes in Linux. See this issue for details: #2102

Unprivileged user namespaces could also allow to run rootless Docker or Podman in Fargate.

[one-summers-day]

+1 For rust.. when rust invokes an error it uses usys/unix/time.rs and that requires privileged mode (sudo) to run correctly T_T

[DoobleD]

We're interested in this capability as well. In our case, we use iptables and ipset from our container to dynamically block offending IP addresses, and without the privileged mode, we get these errors:

Kernel error received: Operation not permitted

Unless there is a (simple) solution to this, this is a deal breaker for us. We can't move to Fargate.

We also use sysctl to tune some kernel parameters from the container (net.core.somaxconn, net.ipv4.ip_local_port_range, fs.inotify.max_user_instances). Even though there is now solutions to set some of these in configuration, the privileged mode would make it much simpler by allowing the use of sysctl from the container.

BTW I'm sure there are very good reasons why this is not (yet) possible, and we have no idea if it would even be possible to allow this mode. Thanks to the AWS team for all the awesome services they provide. :)

[hyavari]

+1 for accessing XFRM in kernel and creating IPSec SAs.

[fernandomullerjr]

+1 We need to use Rancher, that requires --privileged

[DoobleD]

Just curious whether we might get this ability someday, or does the AWS team knows already that it won’t happen?

[one-summers-day]

Most likely isn't happening. Apparently it's a security vulnerability of some sort

[claytonolley]

@ClaytonOlleyNutrien The host machine in question for Fargate containers is a virtual machine, not the underlying physical hardware that runs the VM.

You'll still get root access to the VM which could be running Fargate containers for who knows how many customers. I'm guessing they cannot or will not add VM tenancy per account to Fargate.

[dfuentes77]

This is would be very beneficial to the capability of Fargate. I'm guessing this has already been considered but I would want to emphasize that if/when this is allowed, the idea of a hop limit configuration for Fargate tasks is now relevant. I would want containers inside a Fargate container to be able to use the AWS permissions attached to the Fargate container. I hope this is considered in the implementation.