Enhancing SCC to Gate Runtime Classes

[haircommander]

This enhancement describes the approach for gating runtime classes with SCC

Signed-off-by: Peter Hunt pehunt@redhat.com

[haircommander]

PTAL @deads2k @sttts

[deads2k]

you need to describe how the ordering of SCCs will be done.

[haircommander]

I'm sorry, I don't really understand what you mean by this. Do you have a pointer to a similar description?

[sttts]

https://github.com/openshift/apiserver-library-go/blob/master/pkg/securitycontextconstraints/util/sort/bypriority.go is the order @deads2k is talking about, defined through points https://github.com/openshift/apiserver-library-go/blob/6f1013f42f98d74a6b368d670215493f8425194f/pkg/securitycontextconstraints/util/sort/byrestrictions.go#L107. It determines which SCC is selected if multiple are matching SCCs are available. In general, "the most restricted SCC" should be chosen.

[sttts]

Probably you want some subset order. If one is not the subset of the other, maybe set size matters, and if that is equal you could take something arbitrary but deterministic like alphabetically comparing the alphabetically first element in each set.

[haircommander]

Okay I have added a section and some ideas. A subset order makes sense, possibly weighing each value similar to how RunAsUser values are weighted https://github.com/openshift/apiserver-library-go/blob/6f1013f42f98d74a6b368d670215493f8425194f/pkg/securitycontextconstraints/util/sort/byrestrictions.go#L129..L134

[deads2k]

Trying to think about what runtimeclasses do, it seem that they fall into a few categories

1. add a controlled capability that isn't reflected in current SCC and needs similar controls. I think the thing currently called high-perf but which actually does something with CPU controls falls into this case. Things like this probably want the same SCC rules we already have and a separate control about whether a user or serviceaccount/namespace can use them. I would recommend namespace based on previous SCC/PSP problems and questions.
2. make existing security policy settings safe for a wider variety of users. I think this is what kata does.
3. provide a custom set of restrictions that isn't represented well by security policy. I think this is what windows does and I think with some coercion, it could also be what builds do.

For cases of 2, the benefit to adding to SCC is clear and results in distinct SCCs created for each runtime class that can be granted to users.

For cases of 3, the inclusion in SCC seems unnecessary and potentially confusing since it's not necessarily honoring the pod security policy and it would be seemingly simpler to control access to the runtimeclass as a whole.

For cases of 1, we can end up with significant fanout as we introduce duplicate SCCs to try to restrict access while keeping the fundamental SCC mutations consistent for different runtime classes.

Cases 1 and 3 appear to be well suited towards a switch that allows or denies access based on user and/or service account like SCC today, but only making a decision based on runtimeclass. After that decision is made, SCC could be made to apply. This would give control but avoid a proliferation of SCCs.

I could also imagine cases of 3 wanting to skip SCC altogether and simply write their own custom admission to force values to match what the runtimeclass "means". I think builds would fall into this category since they are well controlled and have a limit set of shapes.

[haircommander]

Trying to think about what runtimeclasses do, it seem that they fall into a few categories

1. add a controlled capability that isn't reflected in current SCC and needs similar controls.  I think the thing currently called high-perf but which actually does something with CPU controls falls into this case.  Things like this probably want the same SCC rules we already have and a separate control about whether a user or serviceaccount/namespace can use them.  I would recommend namespace based on previous SCC/PSP problems and questions.

2. make existing security policy settings safe for a wider variety of users.  I think this is what kata does.

3. provide a custom set of restrictions that isn't represented well by security policy.  I think this is what windows does and I think with some coercion, it could also be what builds do.

For cases of 2, the benefit to adding to SCC is clear and results in distinct SCCs created for each runtime class that can be granted to users.

For cases of 3, the inclusion in SCC seems unnecessary and potentially confusing since it's not necessarily honoring the pod security policy and it would be seemingly simpler to control access to the runtimeclass as a whole.

All of the examples we've added are just that. I am compelled by your position that SCCs are not the right tool to force builder pods to use a certain runtime class, but it could be useful to prevent users from having access to the capabilities builder pods have. Though, I don't want discussion about that to take away from the other reasons we should add this to the SCC structure. If you feel it does, I will drop the case.

For cases of 1, we can end up with significant fanout as we introduce duplicate SCCs to try to restrict access while keeping the fundamental SCC mutations consistent for different runtime classes.

The way I see the access to low-latency capabilities is similar to how AnyUid or AdditionalCapabilities works: vanilla k8s + cri-o has a set of capabilities it gives all users (in this case, access to a runtime class that can configure CPU scheduling), and the Openshift multi-tenant model requires that capability is restricted from some set of users. I think your notion that existing SCCs will be duplicated comes my poor representation of the way we'd enable the low-latency runtime class. Is there anyway I can make it clearer?

Cases 1 and 3 appear to be well suited towards a switch that allows or denies access based on user and/or service account like SCC today, but only making a decision based on runtimeclass. After that decision is made, SCC could be made to apply. This would give control but avoid a proliferation of SCCs.

why would we add an additional admission controller for something that has the same knowledge as SCC (whether a user/service account is allowed to do something) but uses it in a slightly different way. This "switch" you've described is exactly what I thought SCC was. Are there other examples in openshift of additional admission controllers gating user access in a similar way SCC does?

I could also imagine cases of 3 wanting to skip SCC altogether and simply write their own custom admission to force values to match what the runtimeclass "means". I think builds would fall into this category since they are well controlled and have a limit set of shapes.

yeah I can see this, though ultimately we'll need this enhancement to prevent generic users from accessing this runtime class, regardless of whether we use SCC to force the builder pods to use the runtime class.

All in all, I am hearing we need this enhancement for at least the kata use case. If you'd like me to drop the other cases, that's fine with me. As mentioned before, the scope of this PR is not to decide how exactly this feature will be used, but rather adding the feature itself

[haircommander]

@deads2k @sttts PTAL

[haircommander]

comments addressed @sttts @deads2k @zanetworker

[zanetworker]

LGTM :)

[sttts]

I don't think we should or even can assign values to known runtime classes. Admission should not know about runtime classes, i.e. the meaning of the strings, at all. Think about version skew cases between apiserver and kubelet and runtime.

Also I don't think the actual point values of individual classes matter at all. Note that the user is specifying the runtime class explicitly in the pod. Hence, we choose SCCs that contain that in RequiredRuntimeClasses. Why should the other runtime classes in the set influence the choice?

I would go this way:

• points([..., "*", ...]) = 4
• otherwise: points(X) = max(len(X), 3)

It orders SCCs with up to 3 items by length. I don't think it is of value to consider bigger sets. That will never happen in practice.

Note that we also have to think about precedence of these points to the existing ones. If you look into https://github.com/openshift/apiserver-library-go/blob/master/pkg/securitycontextconstraints/util/sort/byrestrictions.go and compare with the other point values, where do you see runtime classes?

• if you have the choice between: a) privileged + ["foo"] b) unprivileged, ["*"] – for a pod that does not require privileged, but wants runtime class "foo"?
• if you have the choice between: a) hostPort + ["foo"] b) no host port, ["*"] – for a pod that does not request host ports, but wants runtime class "foo"?
• if you have the choice between: a) hostNetwork + ["foo"] b) pod network, ["*"] – for a pod that does not require host networking, but wants runtime class "foo"?
• if you have the choice between: a) hostPath mount + ["foo"] b) no hostPath mounts, ["*"] – for a pod that does not request a hostPath, but wants runtime class "foo"?
• if you have the choice between: a) runAsAnyUserPoints + ["foo"] b) runAsUserPoints, ["*"] – for a pod that does not specify the user, but wants runtime class "foo"?
• if you have the choice between: a) all capabilities + ["foo"] b) no extra capabilities, ["*"] – for a pod that does not ask for additional capabilities, but wants runtime class "foo"?

[haircommander]

I think I could use some help here. I have added some open questions about how to weight them relatively when we don't know each field. I am still not convinced we can work around needing some Enum type, and to handle apiserver drift, to be able to accurately describe the relative weight among each of the runtime classes.

I am also pretty sure running a container in a VM, or in a user namespace is more secure than dropping all capabilities, regardless of what that capability is (see https://man7.org/linux/man-pages/man7/user_namespaces.7.html). How do we handle the fact that capMinPoints is 0?

[sttts]

How do we handle the fact that capMinPoints is 0?

Why do we have to? We might have to shift all cap value up with some offset if you want to put runtime classes "before".

I am also pretty sure running a container in a VM, or in a user namespace is more secure than dropping all capabilities, regardless of what that capability is

So everything other than default "" is more secure, independent from capabilities? This suggests we have to give non-default runtime classes more than capMaxPoints = 9999.

What about the other 5 questions in my comment above?

[sttts]

About "VM+any capability is more secure than non-VM+no-capability", to be very clear (and as a corollay: "we need enums"):

I don't think this order matters, and hence the corollary doesn't matter either: every pod either has empty runtime class (= default) or it has an explicit runtime class string specified. If that is true, every admission request will either

1. consider all SCCs with default class (among the required ones),
2. or it will consider all SCCs with some other runtime class.
   In either case, the other listed runtime class in those considered SCCs should not influence the order (because they don't matter for the specified runtime class).

Or in other words: the runtime class is not chosen by admission, but it only filters the set of applicable SCCs.

[haircommander]

How do we handle the fact that capMinPoints is 0?

Why do we have to? We might have to shift all cap value up with some offset if you want to put runtime classes "before".

Ah I didn't know we were so flexible with moving the other options points arounds, sounds good.

I am also pretty sure running a container in a VM, or in a user namespace is more secure than dropping all capabilities, regardless of what that capability is

So everything other than default "" is more secure, independent from capabilities? This suggests we have to give non-default runtime classes more than capMaxPoints = 9999.

I think it's closer that: running a container in a vm or user namespace makes capMaxPoints for that pod really equal 0. Like, it takes away all of the security risk of capabilities. Same goes for runAsAnyUser. These runtime classes nullify the risk.

What about the other 5 questions in my comment above?

The other options (privileged, hostPort, hostPath, hostNetwork) + either VM or user ns is riskier than without those security options and "". Neither do anything to mitigate these security capabilities.

My general issue with this approach of ascribing value to runtime classes in general is that we are basically hard coding a value for a very flexible field.

For instance, runtime classes could be the way we introduce crun into openshift, allowing users to opt into using crun when it becomes a tech preview. Technically, the runtime class that would use crun is no safer than runc (they are basically a 1:1 mapping).

Do you have insight on how to future proof the field?

[sttts]

with #562 (comment) I question this.

[sttts]

or maybe make clearer what "restrictive" means. Correct: it matches less pods. Incorrect: when chosen it is more secure.

[sttts]

I think this is wrong, compare #562 (comment): we should not consider concrete runtime classes here and don't have to.

[openshift-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
To complete the pull request process, please assign stevekuznetsov after the PR has been reviewed.
You can assign the PR to them by writing /assign @stevekuznetsov in a comment when ready.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[openshift-ci]

@haircommander: The following test failed, say /retest to rerun all failed tests:

Test name	Commit	Details	Rerun command
ci/prow/markdownlint	4893af9	link	/test markdownlint

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

[openshift-bot]

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale

[openshift-bot]

Stale issues rot after 30d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle rotten
/remove-lifecycle stale

[jsafrane]

/unassign

[openshift-bot]

Rotten issues close after 30d of inactivity.

Reopen the issue by commenting /reopen.
Mark the issue as fresh by commenting /remove-lifecycle rotten.
Exclude this issue from closing again by commenting /lifecycle frozen.

/close

[openshift-ci]

@openshift-bot: Closed this PR.

In response to this:

Rotten issues close after 30d of inactivity.

Reopen the issue by commenting /reopen.
Mark the issue as fresh by commenting /remove-lifecycle rotten.
Exclude this issue from closing again by commenting /lifecycle frozen.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.