Conform to the Kubernetes restricted pod security standard

[dholbach]

We should make Flux conform to the Kubernetes restricted pod security standard and update the documentation to reflect this.

Patch the deployment spec for:

• source-controller
• kustomize-controller
• helm-controller
• notification-controller
• image-reflector-controller
• image-automation-controller

From Ada Logics

The deployments of the Flux controllers lack container security options that mitigate privilege escalation risks. These are hardening options that we recommend using, and Flux already
makes use of allowPrivilegeEscalation: false on all of its controllers. However, in addition to the allowPrivilegeEscalation option Flux could harden its containers by:

• Dropping all Linux capabilities and enabling those needed
• Filtering syscalls by way of Seccomp

Docker drops many Linux capabilities by default but keeps others for convenience. Flux can harden its containers by having Docker drop all privileges a root user’s process can perform on a system and enabling only those needed. See here for details.

Seccomp filtering is a way of limiting the available system calls and as of v1.19 Kubernetes has support for specifying seccomp policies through the use of seccompProfile in the securityContext of pods. Please see here for details.

Recommendation
Ensure that the pod deployed by Flux has appropriate hardening applied through the use of dropping Linux capabilities and syscall filtering.

[stefanprodan]

We instruct users how to enforce a restricted pod security policy here: https://fluxcd.io/docs/installation/#pod-security-policy

[hiddeco]

That does not align with the recommendation, which is to ship with defaults enabled.

[stefanprodan]

That does not align with the recommendation, which is to ship with defaults enabled.

Yes, shipping a seccomp profile by default is not something that I would consider for Flux. Kubernetes has alpha support for it https://kubernetes.io/blog/2021/08/25/seccomp-default/

[gbstan92]

@stefanprodan the solution above does not cover the registry credentials sync.

Setting the Pod Security Policy in a similar way to the other components causes the following error:

[pjbgf]

@kingdonb you can assign this one to me.

[pjbgf]

Yes, shipping a seccomp profile by default is not something that I would consider for Flux. Kubernetes has alpha support for it https://kubernetes.io/blog/2021/08/25/seccomp-default/

I would agree that we don't want to create and maintain a custom seccomp profile at this point in time. However, enabling the default seccomp profile helps decreasing the impact of potential supply chain attacks.

Seccomp is a bit of a niche subject, but Kubernetes has two separate features for it: one to "enable seccomp across the cluster" (currently in alpha) and the seccomp support itself (GA at version 1.19). We would rely on the latter to comply with some of the recommendations.

The "secure by default" recommendations can be beneficial to the project as a whole, as running in least privileges at all times may help us understand the impact of new features in the security model at earlier stages (i.e. failing tests), and allow us to communicate explicitly as new permission requirements emerge.

I will propose an initial PR for the source-controller and based on feedback I may do the same for the other controllers.

[pjbgf]

@dholbach as part of this we will need to update the documentation around pod security policy. Do you mind adding it here as a check list?

Once all changes are applied we will confirm whether we will be able to support pod-security.kubernetes.io/enforce: restricted, and if so we should also highlight that in the new documentation.

[stefanprodan]

@pjbgf I have added check lists to the issue. Let me know if it covers all tasks.

[pjbgf]

It looks good to me, thank you for updating it. 👍

[davidkarlsen]

after this change I see:

k describe rs source-controller-69bfb4649c|tail -2
  Warning  FailedCreate  8m32s                 replicaset-controller  Error creating: pods "source-controller-69bfb4649c-v5jzg" is forbidden: unable to validate against any security context constraint: [pod.metadata.annotations.container.seccomp.security.alpha.kubernetes.io/manager: Forbidden: seccomp may not be set provider restricted: .spec.securityContext.fsGroup: Invalid value: []int64{1337}: 1337 is not an allowed group pod.metadata.annotations.container.seccomp.security.alpha.kubernetes.io/manager: Forbidden: seccomp may not be set provider "nonroot": Forbidden: not usable by user or serviceaccount provider "hostmount-anyuid": Forbidden: not usable by user or serviceaccount provider "log-collector-scc": Forbidden: not usable by user or serviceaccount provider "machine-api-termination-handler": Forbidden: not usable by user or serviceaccount provider "hostnetwork": Forbidden: not usable by user or serviceaccount provider "hostaccess": Forbidden: not usable by user or serviceaccount provider "node-exporter": Forbidden: not usable by user or serviceaccount provider "privileged": Forbidden: not usable by user or serviceaccount]
  Warning  FailedCreate  3m5s (x9 over 8m30s)  replicaset-controller  (combined from similar events): Error creating: pods "source-controller-69bfb4649c-8kz6r" is forbidden: unable to validate against any security context constraint: [pod.metadata.annotations.container.seccomp.security.alpha.kubernetes.io/manager: Forbidden: seccomp may not be set provider restricted: .spec.securityContext.fsGroup: Invalid value: []int64{1337}: 1337 is not an allowed group pod.metadata.annotations.container.seccomp.security.alpha.kubernetes.io/manager: Forbidden: seccomp may not be set provider "nonroot": Forbidden: not usable by user or serviceaccount provider "hostmount-anyuid": Forbidden: not usable by user or serviceaccount provider "log-collector-scc": Forbidden: not usable by user or serviceaccount provider "machine-api-termination-handler": Forbidden: not usable by user or serviceaccount provider "hostnetwork": Forbidden: not usable by user or serviceaccount provider "hostaccess": Forbidden: not usable by user or serviceaccount provider "node-exporter": Forbidden: not usable by user or serviceaccount provider "privileged": Forbidden: not usable by user or serviceaccount]

If I drop the seccomp profile it will schedule just fine.

openshift 4.9.x

Client Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.3", GitCommit:"816c97ab8cff8a1c72eccca1026f7820e93e0d25", GitTreeState:"clean", BuildDate:"2022-01-25T21:17:57Z", GoVersion:"go1.17.6", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.3+e790d7f", GitCommit:"3a0f2c90b43e6cffd07f57b5b78dd9f083e47ee2", GitTreeState:"clean", BuildDate:"2021-12-14T02:10:38Z", GoVersion:"go1.16.6", Compiler:"gc", Platform:"linux/amd64"}

[stefanprodan]

@davidkarlsen see here the OpenShift docs: https://fluxcd.io/docs/use-cases/openshift/#security-context-constraints

[oliverbaehler]

Encountering:

10m Warning FailedCreate replicaset/notification-controller-5f484fbcbb Error creating: pods "notification-controller-5f484fbcbb-" is forbidden: PodSecurityPolicy: unable to admit pod: [pod.metadata.annotations[container.seccomp.security.alpha.kubernetes.io/manager]: Forbidden: seccomp may not be set]

Link for https://fluxcd.io/docs/installation/#pod-security-policy seems dead?

[stefanprodan]

@oliverbaehler here is the new link https://fluxcd.io/docs/security/#pod-security-standard

You can remove the seccomp profile with a patch in flux-system/kustomization.yaml:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- gotk-components.yaml
- gotk-sync.yaml
patches:
  - target:
      kind: Deployment
    patch: |
      apiVersion: apps/v1
      kind: Deployment
      metadata:
        name: all
      spec:
        template:
          spec:
            containers:
              - name: manager
                securityContext:
                  seccompProfile:
                    $patch: delete

[stefanprodan]

@oliverbaehler I think your PSP is old, please update it like so:

apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
  name: pod-security-policy-restricted-psp
  annotations:
    seccomp.security.alpha.kubernetes.io/allowedProfileNames: 'docker/default,runtime/default'

In Kubernetes 1.19 docker/default was renamed to runtime/default, so you need to update your PSP for Flux to work.