security: Drop capabilities, set userid and enable seccomp #521

pjbgf · 2021-12-11T04:53:15Z

Further restricts the SecurityContext that the controller runs under, by enabling the default seccomp profile, dropping all linux capabilities.

This was set at container-level to ensure backwards compatibility with use cases in which more privileged sidecars are injected into the source-controller pod without setting less restrictive settings.

Note that seccomp will only be enabled if the container runtime and operational system supports it, otherwise the container will run unconfined (aka fail-open), which is the same behaviour as not setting the seccompProfile in the first place.

Relates to fluxcd/flux2#2014

Impacts weaveworks/flux2-openshift#10

BREAKING CHANGE:

The use of new seccomp API requires Kubernetes 1.19.
The controller container is now executed under 65534:65534 (userid:groupid). This change may break deployments that hard-coded the user name 'controller' in their PodSecurityPolicy.

pjbgf · 2021-12-11T04:53:58Z

config/manager/deployment.yaml

+        # Required for AWS IAM Role bindings
+        # https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts-technical-overview.html


Moved the comments here as they relate to fsGroup and not PodSecurityContext.

stefanprodan · 2021-12-11T07:32:43Z

config/manager/deployment.yaml

@@ -31,6 +31,12 @@ spec:
        securityContext:
          allowPrivilegeEscalation: false
          readOnlyRootFilesystem: true
+          capabilities:
+            drop: [ "ALL" ]
+          runAsUser: 65534


Setting a used ID here was proposed before, but we’ve rejected it due to breaking Flux on OpenShift and other Kubernetes distributions that enforce a specific ID range. I think this change should be made to the docs here https://fluxcd.io/docs/installation/#pod-security-policy

Think that shipping a secure default configuration and instructing people to change this if it does not work for their distribution is better than hoping people will find your security notes (and then also apply the required configuration).

@stefanprodan when this was last attempted, do you remember what user/group id was used?

The user ID picked here is linked to the user nobody, which is meant to be the user with least privileges and is generally defined as part of all main distributions.

The upstream pause container relies on the same approach since Kubernetes 1.21. Other security focused projects targetting OpenShift also rely on it.

I completely agree with you that we should not prioritise security over reliability. On that point, would this be better received if accompanied by an OpenShift E2E automated test?

I think it throws an error like this: unable to validate against any security context constraint: [fsGroup: Invalid value: []int64{65534}: 65534 is not an allowed group spec.containers[0].securityContext.securityContext.runAsUser: Invalid value: 1001: must be in the ranges: [1000810000, 1000819999]]

We do have e2e tests for OpenShift, Flux is on OperatorHub. Adding fs group or user ID to each controller repo, just to remove them in flux CLI makes zero sense to me.

Why would we remove them again in the Flux CLI if the OpenShift / OperatorHub setup has custom patches / instructions to make it work anyway?

(Aiming at the instructions here https://fluxcd.io/docs/use-cases/openshift/#security-context-constraints, which could be rewritten to something that deals with the error shared earlier).

They get removed here, not in CLI my bad.

pjbgf · 2021-12-13T14:35:31Z

config/manager/deployment.yaml

+          seccompProfile:
+            type: RuntimeDefault


This syntax requires Kubernetes 1.19. I am assuming this is OK as it aligns with flux check --pre which checks for >=1.19.0.

We do support older versions, see: https://fluxcd.io/docs/installation/#prerequisites

pjbgf · 2021-12-21T12:55:20Z

Rebased with changes of the statically build PR for testing in ARM64. Will rebase after that is merged.

stefanprodan

LGTM

I see no issues with secomp being supported by newer Kubernetes version as we've announced that the next Flux release will drop support for Kubernetes EOL versions.

Thanks @pjbgf 🏅

pjbgf · 2022-01-18T18:24:06Z

This was tested on Kubernetes 1.23.1 with restrict "pod security" (below) and it worked as expected.

apiVersion: v1
kind: Namespace
metadata:
  name: flux-system
  labels:
    pod-security.kubernetes.io/enforce: restricted
    pod-security.kubernetes.io/enforce-version: latest
    pod-security.kubernetes.io/warn: restricted
    pod-security.kubernetes.io/warn-version: latest

stefanprodan · 2022-01-19T14:16:35Z

@pjbgf please rebase

Further restricts the SecurityContext that the controller runs under, by enabling the default seccomp profile and dropping all linux capabilities. This was set at container-level to ensure backwards compatibility with use cases in which sidecars are injected into the source-controller pod without setting less restrictive settings. BREAKING CHANGE: The use of new seccomp API requires Kubernetes 1.19. Co-authored-by: Sanskar Jaiswal <sanskar.jaiswal@weave.works> Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>

BREAKING CHANGE: the controller container is now executed under 65534:65534 (userid:groupid). This change may break deployments that hard-coded the user name 'controller' in their PodSecurityPolicy. Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>

pjbgf commented Dec 11, 2021

View reviewed changes

stefanprodan reviewed Dec 11, 2021

View reviewed changes

pjbgf commented Dec 13, 2021

View reviewed changes

pjbgf force-pushed the main branch from c530c96 to 4936689 Compare December 21, 2021 12:56

pjbgf changed the title ~~security: Drop capabilities and enable seccomp~~ security: Drop capabilities, set user id and enable seccomp Dec 21, 2021

pjbgf changed the title ~~security: Drop capabilities, set user id and enable seccomp~~ security: Drop capabilities and enable seccomp Jan 18, 2022

pjbgf force-pushed the main branch from 4936689 to 6637603 Compare January 18, 2022 16:20

stefanprodan approved these changes Jan 18, 2022

View reviewed changes

pjbgf force-pushed the main branch from 6637603 to 1876081 Compare January 18, 2022 18:17

pjbgf changed the title ~~security: Drop capabilities and enable seccomp~~ security: Drop capabilities, set userid and enable seccomp Jan 18, 2022

Paulo Gomes and others added 2 commits January 19, 2022 14:57

Enforce runAsNonRoot

9ba76a1

BREAKING CHANGE: the controller container is now executed under 65534:65534 (userid:groupid). This change may break deployments that hard-coded the user name 'controller' in their PodSecurityPolicy. Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>

pjbgf force-pushed the main branch from 1876081 to 9ba76a1 Compare January 19, 2022 14:57

stefanprodan merged commit e84031a into fluxcd:main Jan 20, 2022

pjbgf mentioned this pull request Feb 16, 2022

REQUEST: New membership for @pjbgf fluxcd/community#175

Closed

5 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

security: Drop capabilities, set userid and enable seccomp #521

security: Drop capabilities, set userid and enable seccomp #521

pjbgf commented Dec 11, 2021 •

edited

Loading

pjbgf Dec 11, 2021

stefanprodan Dec 11, 2021

hiddeco Dec 11, 2021 •

edited

Loading

pjbgf Dec 13, 2021 •

edited

Loading

hiddeco Dec 13, 2021

stefanprodan Dec 13, 2021

hiddeco Dec 13, 2021 •

edited

Loading

stefanprodan Dec 13, 2021

pjbgf Dec 13, 2021

stefanprodan Dec 13, 2021 •

edited

Loading

pjbgf commented Dec 21, 2021

stefanprodan left a comment

pjbgf commented Jan 18, 2022

stefanprodan commented Jan 19, 2022

		# Required for AWS IAM Role bindings
		# https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts-technical-overview.html

security: Drop capabilities, set userid and enable seccomp #521

security: Drop capabilities, set userid and enable seccomp #521

Conversation

pjbgf commented Dec 11, 2021 • edited Loading

pjbgf Dec 11, 2021

Choose a reason for hiding this comment

stefanprodan Dec 11, 2021

Choose a reason for hiding this comment

hiddeco Dec 11, 2021 • edited Loading

Choose a reason for hiding this comment

pjbgf Dec 13, 2021 • edited Loading

Choose a reason for hiding this comment

hiddeco Dec 13, 2021

Choose a reason for hiding this comment

stefanprodan Dec 13, 2021

Choose a reason for hiding this comment

hiddeco Dec 13, 2021 • edited Loading

Choose a reason for hiding this comment

stefanprodan Dec 13, 2021

Choose a reason for hiding this comment

pjbgf Dec 13, 2021

Choose a reason for hiding this comment

stefanprodan Dec 13, 2021 • edited Loading

Choose a reason for hiding this comment

pjbgf commented Dec 21, 2021

stefanprodan left a comment

Choose a reason for hiding this comment

pjbgf commented Jan 18, 2022

stefanprodan commented Jan 19, 2022

pjbgf commented Dec 11, 2021 •

edited

Loading

hiddeco Dec 11, 2021 •

edited

Loading

pjbgf Dec 13, 2021 •

edited

Loading

hiddeco Dec 13, 2021 •

edited

Loading

stefanprodan Dec 13, 2021 •

edited

Loading