Executor runAsNonRoot

[alexec]

Summary

Currently the executor runs an root. It would be better to run as non-root.

securityContext:
  runAsNonRoot: true
  runAsUser: 8737

This does not work with artifacts for PNS (in my POC) as PNS was not able to grab file handles needed to get files off the container. This works just fine if you do not use artifacts.

Use Cases

Relates to #1824
Relates to #2671
Relates to #3415
Relates to #2239

---

Message from the maintainers:

Impacted by this bug? Give it a 👍. We prioritise the issues with the most 👍.

[alexec]

@idristarwala @uipo78 @tghartland @ibexmonj @athornton @maryoush I'm trying to determine how to run a workflow pod as non-root.

• K8SAPI - should just work
• Kubelet - should just work
• Docker - don't know - needs to be root to access the Docker socket
• PNS - only works if no artifacts are involved

Have you had any luck?

These are all subject to the process run by the main container not needing to be root. It is pretty easy to accidentally do that.

[sujaykulkarn]

Hi @alexec I am able to run a workflow pod with K8SAPI backend as non-root provided these parameters in Workflow Spec.

securityContext:
  runAsNonRoot: true
  runAsUser: 1000

[alexec]

Thank you @sujaykulkarn. Now to find out if we can do that with Docker or PNS!

[chgl]

I am running an unprivileged workflow using PNS. It does require configuring a PSP to bind to the workflow's SA with the SYS_PTRACE and SYS_CHROOT capabilities, though. It used to work with just SYS_PTRACE, but since updating to v2.11.2 it requires SYS_CHROOT. Here's the PSP I use:

apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
  name: argo-workflow
  annotations:
    seccomp.security.alpha.kubernetes.io/allowedProfileNames: 'docker/default,runtime/default'
    apparmor.security.beta.kubernetes.io/allowedProfileNames: 'runtime/default'
    seccomp.security.alpha.kubernetes.io/defaultProfileName:  'runtime/default'
    apparmor.security.beta.kubernetes.io/defaultProfileName:  'runtime/default'
spec:
  privileged: false
  # Required to prevent escalations to root.
  allowPrivilegeEscalation: false
  # This is redundant with non-root + disallow privilege escalation,
  # but we can provide it for defense in depth.
  requiredDropCapabilities:
    - ALL
  allowedCapabilities:
    - 'SYS_PTRACE'
    - 'SYS_CHROOT'
  # Allow core volume types.
  volumes:
    - 'configMap'
    - 'emptyDir'
    - 'projected'
    - 'secret'
    - 'downwardAPI'
    # Assume that persistentVolumes set up by the cluster admin are safe to use.
    - 'persistentVolumeClaim'
  hostNetwork: false
  hostIPC: false
  hostPID: false
  runAsUser:
    # Require the container to run without root privileges.
    rule: 'MustRunAsNonRoot'
  seLinux:
    # This policy assumes the nodes are using AppArmor rather than SELinux.
    rule: 'RunAsAny'
  supplementalGroups:
    rule: 'MustRunAs'
    ranges:
      # Forbid adding the root group.
      - min: 1
        max: 65535
  fsGroup:
    rule: 'MustRunAs'
    ranges:
      # Forbid adding the root group.
      - min: 1
        max: 65535
  readOnlyRootFilesystem: false

With a workflow like this:

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: job-
spec:
  serviceAccountName: argo-workflow
  entrypoint: job
  securityContext:
    runAsUser: 65532
# ...

[alexec]

@chgl I found I could not get PNS to work with runAsNotRoot if there were output artifacts. Do you have these?

[chgl]

@alexec ah, good point. No output artifacts yet. I'll give it a try later and report back. Although I doubt I'll have more success.

[juliusvonkohout]

It seems like #4253 eliminates the need for SYS_PTRACE, but SYS_CHROOT is still needed to extract artifacts from base image layer.

So we need to run a recent containerd or CRI-O cluster, because docker still had some issues with allowing capabilities for non root. I just tested minikube with Kubernetes 1.17 and 1.18.

You need the following stuff. I or you can create a merge request. I do not care as long as i am mentioned as contributor. I am using argo via Kubeflow pipelines.

kubectl edit configmap workflow-controller-configmap -n kubeflow

apiVersion: v1
data:
  config: |
    {
    namespace: kubeflow,
    containerRuntimeExecutor: pns,
    "executor": {
    "args": [
      "--loglevel",
      "debug",
      "--gloglevel",
      "6"
      ]
    },

kubectl rollout restart deployment workflow-controller -n kubeflow

kubectl edit deployment/workflow-controller -n kubeflow

spec:
      containers:
      - args:
        - --configmap
        - workflow-controller-configmap
        - --executor-image
        - docker.io/jvonkoho/argoexec:v2.7.5-license-compliance
        command:
        - workflow-controller

gcr.io/ml-pipeline/argoexec:v2.7.5-license-compliance is not compatible because of
kubernetes/kubernetes#56374.
So we have to build docker.io/jvonkoho/argoexec:v2.7.5-license-compliance from the following dockerfile:

from gcr.io/ml-pipeline/argoexec:v2.7.5-license-compliance
# install setcap and getcap
RUN apt update && apt install -y libcap2-bin bash && apt clean  && rm -rf /var/lib/apt/lists/*
# Because of https://github.com/kubernetes/kubernetes/issues/56374 we have to use setcap
RUN setcap cap_sys_chroot,cap_sys_ptrace=+eip /usr/local/bin/argoexec && getcap /usr/local/bin/argoexec
# /argo/inputs/artifacts must be writable by the init container, that puts input artifacts there
RUN mkdir -p /argo/inputs/artifacts && chmod --recursive 777 /argo && chown 1001:1001 /argo

# docker build .
# docker image tag 9c015da08030 jvonkoho/argoexec:v2.7.5-license-compliance
# docker push jvonkoho/argoexec:v2.7.5-license-compliance

The downside is that this breaks rootless and capabilityless k8sapi executor. Since i got kubeflow pipelines to run on k8sapi i care more about k8sapi. kubeflow/pipelines#4645

Pod Security Policy

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: kubeflow-pipelines-clusterrole
rules:
- apiGroups:
  - policy
  resourceNames:
  - kubeflow-pipelines-psp
  resources:
  - podsecuritypolicies
  verbs:
  - use

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: kubeflow-pipelines-clusterrole-rolebinding
  namespace: kubeflow
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: kubeflow-pipelines-clusterrole
subjects:
- apiGroup: rbac.authorization.k8s.io
  kind: Group
  name: system:serviceaccounts:kubeflow

apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
  name: kubeflow-pipelines-psp
spec:
  allowPrivilegeEscalation: false
  allowedCapabilities:
  - SYS_PTRACE
  - SYS_CHROOT
  #defaultAddCapabilities: not needed after #3785 
  #- SYS_PTRACE
  #- SYS_CHROOT
  # We use 1000 because most non-root images use that user
  fsGroup:
    ranges:
    - max: 65535
      min: 1000
    rule: MustRunAs
  runAsGroup:
    ranges:
    - max: 65535
      min: 1000
    rule: MustRunAs
  runAsUser:
    ranges:
    - max: 65535
      min: 1000
    rule: MustRunAs
  seLinux:
    rule: RunAsAny
  supplementalGroups:
    ranges:
    - max: 65535
      min: 1000
    rule: MustRunAs
  volumes:
  - configMap
  - emptyDir
  - projected
  - secret
  - downwardAPI
  - persistentVolumeClaim

Actually we could also use min: 1 for all user ranges.

[alexec]

That is broadly correct. #4253 fails to a K8S API call. None of the executors can have artifacts on a base layer when you run as non-root.

[alexec]

I believe this is fixed by using the Emissary Executor.