Skip to content

Commit

Permalink
Address KEP feedback
Browse files Browse the repository at this point in the history
  • Loading branch information
@tallclair
tallclair committed Jan 27, 2024
1 parent 52b3359 commit 63c253d
Showing 1 changed file with 50 additions and 43 deletions.
93 changes: 50 additions & 43 deletions keps/sig-node/24-apparmor/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -98,35 +98,30 @@ migration. AppArmor is also feature gated, via the `AppArmor` gate.

### Goals

- Promote AppArmor to GA
- Fully document and formally spec the feature support
- Add equivalent API fields to replace AppArmor annotations and provide a pod
level field, which applies to all containers.
- Deprecate the AppArmor annotations
- Allow running Pods with AppArmor confinement

### Non-Goals

This KEP proposes the absolute minimum to get AppArmor to GA, therefore all
functional enhancements are out of scope, including:
This KEP proposes the absolute minimum to provide generally available AppArmor
confinement for Pods and their containers. Further functional enhancements are out of scope,
including:

- Defining any standard "Kubernetes branded" AppArmor profiles
- Formally specifying the AppArmor profile format in Kubernetes
- Providing mechanisms for loading profiles from outside of the node
- Changing the semantics around AppArmor support
- Providing mechanisms for defining custom profiles using the Kubernetes API, or for
loading profiles from outside of the node.
- Windows support

## Proposal

AppArmor is not available on every Linux distribution. Beside this, container runtimes have AppArmor
as compile-time feature which may be disabled as well. With the GA API the error handling will be
unchanged, and behave exactly the same as the current error propagation paths (see
[Failure and Fallback Strategy](#failure-and-fallback-strategy)).
Add a new field to the Pod API that allows defining the AppArmor profile. The new field should be
part of the security context.

### API

The AppArmor API will be functionally equivalent to the current beta API, with
the enhancement of adding pod level profiles to match the behavior with seccomp.
This includes the Pod API, which specifies what profile the containers run with.
Pods and PodTemplate will include an `appArmorProfile` field that you can set either for a Pod's
security context or for an individual container. If AppArmor options are defined at both the pod and
container level, the container-level options override the pod options.

#### Pod API

Expand Down Expand Up @@ -190,7 +185,7 @@ profile cannot be set.
##### Localhost Profile

This KEP proposes LocalhostProfile as the only source of user-defined
profiles at this point. User-defined profiles are essential for users to realize
profiles. User-defined profiles are essential for users to realize
the full benefits out of AppArmor, allowing them to decrease their attack
surface based on their own workloads.

Expand Down Expand Up @@ -267,16 +262,20 @@ for backwards compatibility.

#### Node Status

The Kubelet currently appends the AppArmor status to the node ready condition message:

https://github.com/kubernetes/kubernetes/blob/5c664599a9a80f01c30d924702f7ad799f7dd956/pkg/kubelet/nodestatus/setters.go#L530-L536
The Kubelet SHOULD NOT append the AppArmor status to the node ready condition message.

The ready condition is certainly not the right place for this message, but more generally the
Kubelet does not broadcast the status of every optional feature. For GA, this message will simply be
removed.
kubelet does not broadcast the status of every optional feature. (A beta implementation of
this feature, added before the Kubernetes enhancement process was formalized, did customize
the node ready condition message).

## Design Details

When an AppArmor profile is set on a container (or pod), the kubelet will pass the option on to the
container runtime, which is responsible for running the container with the desired profile. Profiles
must be loaded into the kernel before the container is started (profile loading is out of scope for
this KEP). For more details, see https://kubernetes.io/docs/tutorials/security/apparmor/.

### Test Plan

[X] I/we understand the owners of the involved components may require updates to
Expand Down Expand Up @@ -315,12 +314,13 @@ below are the ones we mapped and their outcome once this KEP is implemented:

| Scenario | API Server Result | Kubelet Result |
| -------------------------------------------------------------------------------------------------- | -------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------- |
| 1) Using localhost or `runtime/default` profile when container runtime does not support AppArmor. | Pod created | The outcome is container runtime dependent. In this scenario containers may 1) fail to start or 2) run normally without having its policies enforced. |
| 1) Using localhost or explicit `runtime/default` profile when container runtime does not support AppArmor. | Pod created | The outcome is container runtime dependent. In this scenario containers may 1) fail to start or 2) run normally without having its policies enforced. |
| 2) Using custom or `runtime/default` profile that restricts actions a container is trying to make. | Pod created | The outcome is workload and AppArmor dependent. In this scenario containers may 1) fail to start, 2) misbehave or 3) log violations. |
| 3) Using a localhost profile that does not exist on the node. | Pod created | Container runtime dependent: containers fail to start. Retry respecting RestartPolicy and back-off delay. Error message in event. |
| 4) Using an unsupported runtime profile (i.e. `runtime/default-audit`). | Fails validation: pod **not** created. | N/A |
| 5) Using localhost profile with invalid (empty) name | Fails validation: pod **not** created. | N/A |
| 6) AppArmor is disabled by the host or the build | Pod created. | Kubelet puts Pod in blocked state. |
| 6) Using localhost or explicit `runtime/default` profile when AppArmor is disabled by the host or build | Pod created. | Kubelet puts Pod in blocked ssstate. |
| 7) Using implicit (default) `runtime/default` profile when AppArmor is disabled by the host or build. | Pod created | Container created without AppArmor enforcement. |

Scenario 2 is the expected behavior of using AppArmor and it is included here
for completeness.
Expand Down Expand Up @@ -360,12 +360,12 @@ This will be enforced in API validation.

Container-level AppArmor profiles override anything set at the pod-level.

#### PodSecurity Admission
#### Pod Security Admission

The PodSecurity admission plugin will be updated to evaluate AppArmorProfile fields in addition to
The Pod Security admission plugin will be updated to evaluate AppArmorProfile fields in addition to
annotations.

The [PodSecurity baseline policy](https://kubernetes.io/docs/concepts/security/pod-security-standards/#baseline)
The [policy for the **baseline** Pod security standard](https://kubernetes.io/docs/concepts/security/pod-security-standards/#baseline)
forbids setting an `Unconfined` profile, but allows unset, `RuntimeDefault` and `Localhost`
profiles. In the case of localhost profiles, this can include OS profiles intended for other system
daemons, so additional profile restrictions are encouraged (e.g. via
Expand All @@ -383,14 +383,14 @@ Ephemeral container's will never sync with an AppArmor annotation.

#### PodTemplates

PodTemplates (e.g. ReplaceSets, Deployments, StatefulSets, etc.) will be
PodTemplates (and their embeddings within e.g. ReplicaSets, Deployments, StatefulSets, etc.) will be
ignored. The field/annotation resolution will happen on template instantiation.

#### Warnings

To raise awareness of existing controllers using the AppArmor annotations that need to be migrated,
a warning will be emitted when only annotations are set (no fields) on pod creation, or pod template (including
embedded pod templates) create & update.
To raise awareness of workloads using the beta AppArmor annotations that need to be migrated, a
warning will be emitted when only AppArmor annotations are set (no fields) on pod creation, or pod
template (including workload resources with an embedded pod template) create & update.

#### Kubelet fallback

Expand Down Expand Up @@ -464,7 +464,8 @@ _This section is excluded, as it is the subject of the entire proposal._

###### How can this feature be enabled / disabled in a live cluster?

AppArmor is controlled by the `AppArmor` feature gate, but no new feature gate is added for the changes outlined in this proposal.
AppArmor is controlled by the `AppArmor` feature gate (already beta by the time this KEP was
formally opened).

- [X] Feature gate
- Feature gate name: `AppArmor`
Expand All @@ -478,10 +479,9 @@ No - AppArmor has been enabled by default since Kubernetes v1.4.

###### Can the feature be disabled once it has been enabled (i.e. can we roll back the enablement)?

Yes, it works in the same way as before moving the feature to GA. However, the
GA related changes are backwards compatible, and the API supports rollback of
the Kubernetes API Server as described in the [Version Skew
Strategy](#version-skew-strategy).
Yes. Containers already running with AppArmor enforcement will continue to do so, but on restart
will fallback to the container runtime default. Pods created with AppArmor disabled will have their
fields & annotations stripped.

###### What happens if we reenable the feature if it was previously rolled back?

Expand Down Expand Up @@ -607,17 +607,24 @@ N/A

## Implementation History

- 2016-07-25: AppArmor design proposal
- 2016-07-25: [AppArmor design proposal](https://github.com/kubernetes/design-proposals-archive/blob/main/auth/apparmor.md)
- 2016-09-26: AppArmor beta release with v1.4
- 2020-01-10: Initial KEP
- 2020-01-10: Initial (retrospective) KEP

## Drawbacks

Promoting AppArmor as-is to GA may be seen as "blessing" the current
functionality, and make it harder to make some of the enhancements listed under
[Non-Goals](#non-goals). Since the current behavior is unguarded, I think we
already need to treat the behavior as GA.
- Custom AppArmor profiles are not fully managed by Kubernetes
- AppArmor support adds a dimension to the feature compatibility matrix, as support is not
guaranteed in linux

## Alternatives

N/A
### Syncing fields & annotations on workload resources

AppArmor fields & annotations on Pods are immutable, which means that syncing fields & annotations
is a one-time operation. This is not true for workload resources (ReplicaSets, Deployments, etc).

In order to support syncing fields on workload resources, we need to account for clients that only
pay attention to one of the field/annotation settings. When combined with the validation requirement
that fields & annotations match, getting this right in both the patch & update cases adds
significant complexity.

0 comments on commit 63c253d

Please sign in to comment.