ERROR controller-runtime.controller - Privileged containers are not allowed spec.containers[1].securityContext.privileged

[mssanjay]

Describe the bug
Controller is unable to create runner pods. I see error as below

2021-09-06T22:15:37.929Z ERROR controller-runtime.controller Reconciler error {"controller": "runner-controller", "request": "gha-runner-controller/example-runner", "error": "pods "example-runner" is forbidden: PodSecurityPolicy: unable to admit pod: [spec.containers[1].securityContext.privileged: Invalid value: true: Privileged containers are not allowed spec.containers[1].securityContext.privileged: Invalid value: true: Privileged containers are not allowed]"}

Checks
Manager logs shows below error

Privileged containers are not allowed spec.containers[1].securityContext.privileged

To Reproduce
Disable Previleged Pods in the cluster

Expected behavior
Runner Pod created

Screenshots
If applicable, add screenshots to help explain your problem.

Environment (please complete the following information):

• Controller Version - 0.18.2
• Deployment Method [e.g. Helm and Kustomize ] - Helm
• Helm Chart Version [e.g. 0.11.0, if applicable] -

Additional context
Add any other context about the problem here.

[mumoshu]

@mssanjay Hey! Just curious but what do you want me to do? The information provided by you says that you got the error due to that you created some PSP to disallow privileged pods in your cluster. Then, there might be no way to make actions runners work.

Perhaps anyone could try to integrate rootless docker into actions-runner-controller, but I'm not going to do that myself anytime soon. If you could, I'll gladly review your work!

[mssanjay]

@mumoshu Our corporate policy does not allow Privileged pods in our cluster. Hence the policy is in place. Also, I am using the image "docker:dind-rootless" which I thought is a rootless docker. So I am not sure why this error. Thanks for the update!

[mumoshu]

@mssanjay Ah, good to know that there's now rootless version of the official dind image!

Well, actions-runner-controller mostly hardcodes privileged: true. But you can maybe try setting securityContext.seLinuxOptions to non-nil to make it privileged: false.

https://github.com/actions-runner-controller/actions-runner-controller/blob/14564c7b8e0ad7adadcb18a41a15f831b580f931/controllers/runner_controller.go#L805-L812

If it doesn't work, I'd suggest trying the canary image of actions-runner-controller that has a brand new feature called Runner Sets. Perhaps you can overwrite privileged setting for docker container with it. Something like:

apiVersion: actions.summerwind.dev/v1alpha1
kind: RunnerSet
metadata:
  name: example
spec:
  ephemeral: false
  replicas: 2
  repository: mumoshu/actions-runner-controller-ci
  # Other mandatory fields from StatefulSet
  selector:
    matchLabels:
      app: example
  serviceName: example
  template:
    metadata:
      labels:
        app: example
    spec:
      containers:
      - name: docker
        privileged: false

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[rajbos]

I had the same thing (preparing to add PSP that will disallow privileged containers). I suspect it will work for the dind container, but the deployment will always need a privileged container since it spins up new pods?

Anyway, I've tried using a mimimal seLinuxOptions since that should set the privileged to false. Unfortunately applying this setup still runs the dind containers as privileged:

# runner-deployment.yaml
apiVersion: actions.summerwind.dev/v1alpha1
kind: RunnerDeployment
metadata:
  name: xxx-gh-runner-deployment
spec:
  template:
    spec:
      organization: xxx-gh
      labels:
        - xxx-gh-org
        - k8s

      securityContext:
        seLinuxOptions:
          # level: "s0"
          # role: "system_r"
          # type: "super_t"
          # user: "system_u"
       privileged: false

I've no clue what proper values could be for the level / role / type / user, I have tried that the values above, but still the same result.

Note: I'm checking the privileged status with this command, which returns true:

kubectl get pod xxx-gh-runner-deployment-mzfx9-rj9rs -o json | jq -r '.spec.containers[].securityContext'

Note: I had to modify the official dind:rootless images by adding an extra folder. The image can be found here in my public package feed.

[mumoshu]

If anyone could come up with a working changeset that enables you to run runners w/ dind-rootless w/o privileged, I'd be happy to review it :)

[elebioda]

@mumoshu Our corporate policy does not allow Privileged pods in our cluster. Hence the policy is in place. Also, I am using the image "docker:dind-rootless" which I thought is a rootless docker. So I am not sure why this error. Thanks for the update!

rootless docker still requires privileged set to true

[rajbos]

The controller still spins up both containers (the runner and the dind container) in the pod with --priviledged, which I haven't found a way to change @elebioda. Hence the error. Using a different image (dind-rootless) doesn't matter then, they're still spin up as privileged :-).

[elebioda]

@rajbos gotcha. Sadly, I am just having to disable the dind sidecar and use the runner for any workflow that holds no container actions... which is slim to none...

[mumoshu]

@elebioda Thanks for sharing! That's absolutely sad news to hear.

I'm now wondering if we could use rootless podman via a docker alias instead. I heard podman has docker-compatible CLI and it works without root. I hope it doesn't require privileges :)

[florenciacomuzzi]

I am seeing a similar error because I am running on a GKE Autopilot cluster which does not allow privileged pods:

2021-11-20T20:54:50.284Z        ERROR   actions-runner-controller.runner        Failed to create pod resource   {"runner": "actions-runner-system/example-runnerdeploy-lbdgs-pxdtk", "error": "admission webhook \"policycontrollerv2.common-webhooks.networking.gke.io\" denied the request: GKE Policy Controller rejected the request because it violates one or more policies: {\"[denied by autogke-disallow-privilege]\":[\"container docker is privileged; not allowed in Autopilot\"]}"}

[mumoshu]

I've researched this a bit.

The bad news is that the state-of-the-art looks like we still need (1)a rootful container or alike to run a dockerd(with root or non-root user, with or without priv) in the runner pod to run (2)action and service containers(requested by your GitHub Actions workflow).

(1) can be a runc container, a sysbox container, a firecracker VM, etc and (2) is usually a docker container, podman container(e.g. if you do ailas docker=podman in our runner's entrypoint.sh.

I believe we have been discussing about alternatives in (2) so far. But my theory says that we'd need to find one in (1), which isn't trivial and also out of the scope of actions-runner-controller.

The good news is:

• Privileged pods are not required if you don't use docker inside your actions workflows. Perhaps we'd better add a runner spec field to force disabling privilege, so that you can opt-out privileged pod?
• Automatically setting up a small self-managed K8s node per each runner pod by utilizing cluster-autoscaler should be possible. That's a little better than having a lot of privileged pods on a single node.
• I see a container runtime like sysbox has possibility to provide non-privileged pods the ability to run docker inside. See Add support for Sysbox PODs nestybox/sysbox#64 (comment), [1.3 - DO NOT MERGE] User namespace support containerd/cri#1527, and Support User Namespaces in pods kubernetes/enhancements#127 for more context regarding this

I haven't considered using other container runtimes like firecracker and kata container yet. In theory, it should be able to run any privileged containers safely inside a VM as a container. But I'm not sure if they are tested against our use-case. If anyone could give it a try and share your experience, it would be super helpful!

[roshvin]

Hi , even we are looking for running DinD in non-privileged mode . since our CaaS policy wont allow running any container as root.
Do you have any working solution or recommendation.

[jinalshah]

I am seeing a similar error because I am running on a GKE Autopilot cluster which does not allow privileged pods:

2021-11-20T20:54:50.284Z        ERROR   actions-runner-controller.runner        Failed to create pod resource   {"runner": "actions-runner-system/example-runnerdeploy-lbdgs-pxdtk", "error": "admission webhook \"policycontrollerv2.common-webhooks.networking.gke.io\" denied the request: GKE Policy Controller rejected the request because it violates one or more policies: {\"[denied by autogke-disallow-privilege]\":[\"container docker is privileged; not allowed in Autopilot\"]}"}

@flopotok did you ever manage to get this working on GKE Autopilot? As I've been running into the same issues.

[tedchang77]

We don't use docker inside our actions workflow. Is it possible to add a runner configuration where we can specify privileged: false?

[mumoshu]

Hey. Yep see #1383