Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

security: Drop capabilities, set userid and enable seccomp #521

Merged
merged 2 commits into from
Jan 20, 2022

Conversation

pjbgf
Copy link
Member

@pjbgf pjbgf commented Dec 11, 2021

Further restricts the SecurityContext that the controller runs under, by enabling the default seccomp profile, dropping all linux capabilities.

This was set at container-level to ensure backwards compatibility with use cases in which more privileged sidecars are injected into the source-controller pod without setting less restrictive settings.

Note that seccomp will only be enabled if the container runtime and operational system supports it, otherwise the container will run unconfined (aka fail-open), which is the same behaviour as not setting the seccompProfile in the first place.

Relates to fluxcd/flux2#2014

Impacts weaveworks/flux2-openshift#10

BREAKING CHANGE:

  • The use of new seccomp API requires Kubernetes 1.19.
  • The controller container is now executed under 65534:65534 (userid:groupid). This change may break deployments that hard-coded the user name 'controller' in their PodSecurityPolicy.

Comment on lines +24 to +25
# Required for AWS IAM Role bindings
# https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts-technical-overview.html
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Moved the comments here as they relate to fsGroup and not PodSecurityContext.

@@ -31,6 +31,12 @@ spec:
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
capabilities:
drop: [ "ALL" ]
runAsUser: 65534
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Setting a used ID here was proposed before, but we’ve rejected it due to breaking Flux on OpenShift and other Kubernetes distributions that enforce a specific ID range. I think this change should be made to the docs here https://fluxcd.io/docs/installation/#pod-security-policy

Copy link
Member

@hiddeco hiddeco Dec 11, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Think that shipping a secure default configuration and instructing people to change this if it does not work for their distribution is better than hoping people will find your security notes (and then also apply the required configuration).

Copy link
Member Author

@pjbgf pjbgf Dec 13, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@stefanprodan when this was last attempted, do you remember what user/group id was used?

The user ID picked here is linked to the user nobody, which is meant to be the user with least privileges and is generally defined as part of all main distributions.

The upstream pause container relies on the same approach since Kubernetes 1.21. Other security focused projects targetting OpenShift also rely on it.

I completely agree with you that we should not prioritise security over reliability. On that point, would this be better received if accompanied by an OpenShift E2E automated test?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it throws an error like this: unable to validate against any security context constraint: [fsGroup: Invalid value: []int64{65534}: 65534 is not an allowed group spec.containers[0].securityContext.securityContext.runAsUser: Invalid value: 1001: must be in the ranges: [1000810000, 1000819999]]

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We do have e2e tests for OpenShift, Flux is on OperatorHub. Adding fs group or user ID to each controller repo, just to remove them in flux CLI makes zero sense to me.

Copy link
Member

@hiddeco hiddeco Dec 13, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why would we remove them again in the Flux CLI if the OpenShift / OperatorHub setup has custom patches / instructions to make it work anyway?

(Aiming at the instructions here https://fluxcd.io/docs/use-cases/openshift/#security-context-constraints, which could be rewritten to something that deals with the error shared earlier).

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

They get removed here, not in CLI my bad.

seccompProfile:
type: RuntimeDefault
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This syntax requires Kubernetes 1.19. I am assuming this is OK as it aligns with flux check --pre which checks for >=1.19.0.

Copy link
Member

@stefanprodan stefanprodan Dec 13, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We do support older versions, see: https://fluxcd.io/docs/installation/#prerequisites

@pjbgf
Copy link
Member Author

pjbgf commented Dec 21, 2021

Rebased with changes of the statically build PR for testing in ARM64. Will rebase after that is merged.

@pjbgf pjbgf changed the title security: Drop capabilities and enable seccomp security: Drop capabilities, set user id and enable seccomp Dec 21, 2021
@pjbgf pjbgf changed the title security: Drop capabilities, set user id and enable seccomp security: Drop capabilities and enable seccomp Jan 18, 2022
Copy link
Member

@stefanprodan stefanprodan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

I see no issues with secomp being supported by newer Kubernetes version as we've announced that the next Flux release will drop support for Kubernetes EOL versions.

Thanks @pjbgf 🏅

@pjbgf
Copy link
Member Author

pjbgf commented Jan 18, 2022

This was tested on Kubernetes 1.23.1 with restrict "pod security" (below) and it worked as expected.

apiVersion: v1
kind: Namespace
metadata:
  name: flux-system
  labels:
    pod-security.kubernetes.io/enforce: restricted
    pod-security.kubernetes.io/enforce-version: latest
    pod-security.kubernetes.io/warn: restricted
    pod-security.kubernetes.io/warn-version: latest

@pjbgf pjbgf changed the title security: Drop capabilities and enable seccomp security: Drop capabilities, set userid and enable seccomp Jan 18, 2022
@stefanprodan
Copy link
Member

@pjbgf please rebase

Paulo Gomes and others added 2 commits January 19, 2022 14:57
Further restricts the SecurityContext that the controller runs under, by enabling the default seccomp profile and dropping all linux capabilities.
This was set at container-level to ensure backwards compatibility with
use cases in which sidecars are injected into the source-controller pod
without setting less restrictive settings.

BREAKING CHANGE: The use of new seccomp API requires Kubernetes 1.19.

Co-authored-by: Sanskar Jaiswal <sanskar.jaiswal@weave.works>
Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
BREAKING CHANGE: the controller container is now executed under 65534:65534 (userid:groupid). This change may break deployments that hard-coded the user name 'controller' in their PodSecurityPolicy.

Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants