From 685e057392e9f18af979568daea8b7611da4c5df Mon Sep 17 00:00:00 2001 From: Sascha Grunert Date: Fri, 10 Jan 2020 13:54:51 +0100 Subject: [PATCH 01/28] Add AppArmor GA KEP Signed-off-by: Sascha Grunert Signed-off-by: Sascha Grunert --- keps/sig-node/24-apparmor-ga/README.md | 679 +++++++++++++++++++++++++ keps/sig-node/24-apparmor-ga/kep.yaml | 48 ++ 2 files changed, 727 insertions(+) create mode 100644 keps/sig-node/24-apparmor-ga/README.md create mode 100644 keps/sig-node/24-apparmor-ga/kep.yaml diff --git a/keps/sig-node/24-apparmor-ga/README.md b/keps/sig-node/24-apparmor-ga/README.md new file mode 100644 index 00000000000..1bceb24af81 --- /dev/null +++ b/keps/sig-node/24-apparmor-ga/README.md @@ -0,0 +1,679 @@ +# AppArmor graduation to General Availability (GA) + +## Table of Contents + + +- [Release Signoff Checklist](#release-signoff-checklist) +- [Summary](#summary) +- [Motivation](#motivation) + - [Goals](#goals) + - [Non-Goals](#non-goals) +- [Proposal](#proposal) + - [API](#api) + - [Pod API](#pod-api) + - [Localhost Profile](#localhost-profile) + - [RuntimeDefault Profile](#runtimedefault-profile) +- [Design Details](#design-details) + - [Failure and Fallback Strategy](#failure-and-fallback-strategy) + - [Version Skew Strategy](#version-skew-strategy) + - [Pod Creation](#pod-creation) + - [PodSecurityPolicy Enforcement](#podsecuritypolicy-enforcement) + - [Pod Update](#pod-update) + - [PodTemplates](#podtemplates) + - [Runtime Profiles](#runtime-profiles) + - [Kubelet Backwards compatibility](#kubelet-backwards-compatibility) + - [Upgrade / Downgrade](#upgrade--downgrade) + - [Test Plan](#test-plan) + - [Graduation Criteria](#graduation-criteria) +- [Production Readiness Review Questionnaire](#production-readiness-review-questionnaire) + - [Feature enablement and rollback](#feature-enablement-and-rollback) + - [Rollout, Upgrade and Rollback Planning](#rollout-upgrade-and-rollback-planning) + - [Monitoring requirements](#monitoring-requirements) + - [Dependencies](#dependencies) + - [Scalability](#scalability) + - [Troubleshooting](#troubleshooting) +- [Implementation History](#implementation-history) +- [Drawbacks](#drawbacks) +- [Alternatives](#alternatives) + - [Updating PodSecurityPolicy API](#updating-podsecuritypolicy-api) + + +## Release Signoff Checklist + +Items marked with (R) are required _prior to targeting to a milestone / release_. + +- [ ] (R) Enhancement issue in release milestone, which links to KEP dir in + [kubernetes/enhancements] (not the initial KEP PR) +- [ ] (R) KEP approvers have approved the KEP status as `implementable` +- [ ] (R) Design details are appropriately documented +- [ ] (R) Test plan is in place, giving consideration to SIG Architecture and + SIG Testing input +- [ ] (R) Graduation criteria is in place +- [ ] (R) Production readiness review completed +- [ ] Production readiness review approved +- [ ] "Implementation History" section is up-to-date for milestone +- [ ] User-facing documentation has been created in [kubernetes/website], for + publication to [kubernetes.io] +- [ ] Supporting documentation e.g., additional design documents, links to + mailing list discussions/SIG meetings, relevant PRs/issues, release notes + +**Note:** Any PRs to move a KEP to `implementable` or significant changes once +it is marked `implementable` should be approved by each of the KEP approvers. If +any of those approvers is no longer appropriate than changes to that list should +be approved by the remaining approvers and/or the owning SIG (or SIG-arch for +cross cutting KEPs). + +**Note:** This checklist is iterative and should be reviewed and updated every +time this enhancement is being considered for a milestone. + +[kubernetes.io]: https://kubernetes.io/ +[kubernetes/enhancements]: https://github.com/kubernetes/enhancements/issues +[kubernetes/kubernetes]: https://github.com/kubernetes/kubernetes +[kubernetes/website]: https://github.com/kubernetes/website + +## Summary + +This is a proposal to upgrade the AppArmor annotation on pods to a dedicated +field, and graduate the feature to GA. This proposal aims to do the _bare +minimum_ to clean up the feature, without blocking future enhancements. + +## Motivation + +AppArmor support has been added with Kubernetes v1.4 and is already in beta. +Profiles have to be available on each node whereas the container runtime ensures +that the profile is loaded when specified at pod or PSP level. Profiles can be +specified per-container via the pod’s metadata annotation: + +``` +container.apparmor.security.beta.kubernetes.io/: {unconfined,runtime/default,localhost/} +``` + +The feature has been more or less unchanged ever since. The main motivation +behind this KEP is to promote the AppArmor feature to GA. + +_NOTE: Seccomp was in a very similar state, but with some subtle differences. +Promoting Seccomp to GA will be covered by a [separate +KEP](/keps/sig-node/20190717-seccomp-ga.md)._ + +### Goals + +- Declare AppArmor as GA +- Fully document and formally spec the feature support +- Add equivalent API fields to replace AppArmor annotations and provide a pod + level field, which applies to all containers. +- Deprecate the AppArmor annotations + +### Non-Goals + +This KEP proposes the absolute minimum to get AppArmor to GA, therefore all +functional enhancements are out of scope, including: + +- Defining any standard "Kubernetes branded" AppArmor profiles +- Formally specifying the AppArmor profile format in Kubernetes +- Providing mechanisms for loading profiles from outside of the node +- Changing the semantics around AppArmor support +- Windows support + +## Proposal + +AppArmor is not available on every Linux distribution. Beside this, container +runtimes have AppArmor as compile-time feature which may be disabled as well. +With the GA API we do not change the error handling and behave exactly the same +as the current error propagation paths. + +### API + +The AppArmor API will be functionally equivalent to the current beta API, with +the enhancement of adding pod level profiles to match the behavior with seccomp. +This includes the Pod API, which specifies what profile the containers run with. +There are no changes planned to the current `PodSecurityPolicy` API as part of +this KEP. + +#### Pod API + +The Pod AppArmor API is generally immutable, except in `PodTemplates`. + +```go +type PodSecurityContext struct { + ... + // The AppArmor options to use by the containers in this pod. + // +optional + AppArmor *AppArmorProfile + ... +} + +type SecurityContext struct { + ... + // The AppArmor options to use by this container. If AppArmor options are + // provided at both the pod & container level, the container options + // override the pod options. + // +optional + AppArmor *AppArmorProfile + ... +} + +// Only one profile source may be set. +// +union +type AppArmorProfile struct { + // +unionDescriminator + Type AppArmorProfileType + + // LocalhostProfile indicates a loaded profile on the node that should be used. + // The profile must be preconfigured on the node to work. + // Must match the loaded name of the profile. + // Must only be set if type is "Localhost". + // +optional + LocalhostProfile *string +} + +type AppArmorProfileType string + +const ( + AppArmorProfileTypeUnconfined AppArmorProfileType = "Unconfined" + AppArmorProfileTypeRuntimeDefault AppArmorProfileType = "RuntimeDefault" + AppArmorProfileTypeLocalhost AppArmorProfileType = "Localhost" +) +``` + +This API makes the options more explicit and leaves room for new profile sources +to be added in the future (e.g. Kubernetes predefined profiles or ConfigMap +profiles) and for future extensions, such as defining the behavior when a +profile cannot be set. + +##### Localhost Profile + +This KEP proposes we GA LocalhostProfile as the only source of user-defined +profiles at this point. User-defined profiles are essential for users to realize +the full benefits out of AppArmor, allowing them to decrease their attack +surface based on their own workloads. + +###### Updating AppArmor profiles + +AppArmor profiles are applied at container creation time. The underlying +container runtime only references already loaded profiles by its name. +Therefore, updating the profiles content requires a manual reload via +`apparmor_parser`. + +Note that changing profiles is not recommended and may cause containers to fail +on next restart, in the case of the new profile being more restrictive, invalid +or the file no longer available on the host. + +Currently, users have no way to tell whether their physical profiles have been +deleted or modified. This KEP proposes no changes to the existing functionality. + +The recommended approach for rolling out changes to AppArmor profiles is to +always create _new profiles_ instead of updating existing ones. Create and +deploy a new version of the existing Pod Template, changing the profile name to +the newly created profile. Redeploy, once working delete the former Pod +Template. This will avoid disruption on in-flight workloads. + +The current behavior lacks features to facilitate the maintenance of AppArmor +profiles across the cluster. Two examples being: 1) the lack of profile +synchronization across nodes and 2) how difficult it can be to identify that +profiles have been changed on disk/memory, after pods started using it. However, +given its current "pseudo-GA" state, we don't want to change it with this KEP. +Out of tree enhancements like the +[security-profiles-operator](https://github.com/kubernetes-sigs/security-profiles-operator) can +provide such enhanced functionality on top. + +###### Profiles managed by the cluster admins + +The current support relies on profiles being loaded on all cluster nodes +where the pods using them may be scheduled. It is also the cluster admin's +responsibility to ensure the profiles are correctly saved and synchronised +across the all nodes. Existing mechanisms like node `labels` and `nodeSelectors` +can be used to ensure that pods are scheduled on nodes supporting their desired +profiles. + +##### RuntimeDefault Profile + +We propose maintaining the support to a single runtime profile, which will be +defined by using the `AppArmorProfileTypeRuntimeDefault`. The reasons being: + +- No changes to the current behavior. Users are currently not allowed to specify + other runtime profiles. The existing API server rejects runtime profile names + that are different than `runtime/default`. +- Most runtimes only support the default profile, although the CRI is flexible + enough to allow the kubelet to send other profile names. +- Dockershim does not currently provide a way to pass other runtime profile + names. +- Multiple runtime profiles has never been requested as a feature. + +If built-in support for multiple runtime profiles is needed in the future, a new +KEP will be created to cover its details. + +## Design Details + +### Failure and Fallback Strategy + +There are different scenarios in which applying an AppArmor profile may fail, +below are the ones we mapped and their outcome once this KEP is implemented: + +| Scenario | API Server Result | Kubelet Result | +| -------------------------------------------------------------------------------------------------- | -------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------- | +| 1) Using custom or `runtime/default` profile when container runtime does not support AppArmor. | Pod created | The outcome is container runtime dependent. In this scenario containers may 1) fail to start or 2) run normally without having its policies enforced. | +| 2) Using custom or `runtime/default` profile that restricts actions a container is trying to make. | Pod created | The outcome is workload and AppArmor dependent. In this scenario containers may 1) fail to start, 2) misbehave or 3) log violations. | +| 3) Using custom profile that does not exist on the node. | Pod created | Containers fail to start. Retry respecting RestartPolicy and back-off delay. | +| 4) Using an unsupported runtime profile (i.e. `runtime/default-audit`). | Pod **not** created. | N/A | +| 5) Using localhost profile with invalid name | Pod **not** created. | N/A | +| 6) AppArmor is disabled by the host or the build | Pod **not** created. | Kubelet puts Pod in blocked state. | + +Scenario 2 is the expected behavior of using AppArmor and it is included here +for completeness. + +Scenario 5 represents the case of failing the existing validation, which is +defined at [Pod API](#pod-api). + +### Version Skew Strategy + +All API skew is resolved in the API server. New Kubelets will only use the +AppArmor values specified in the fields, and ignore the annotations. + +#### Pod Creation + +If no AppArmor annotations or fields are specified, no action is necessary. + +If the `AppArmor` feature is disabled per feature gate, then the annotations and +fields are cleared (current behavior). + +If _only_ AppArmor fields are specified, add the corresponding annotations. This +ensures that the fields are enforced even if the node version trails the API +version (see [Version Skew Strategy](##version-skew-strategy)). + +If _only_ AppArmor annotations are specified, copy the values into the +corresponding fields. This ensures that existing applications continue to +enforce AppArmor, and prevents the kubelet from needing to resolve annotations & +fields. If the annotation is empty, then the `runtime/default` profile will be +used by the CRI container runtime. If a localhost profile is specified, then +container runtimes will strip the `localhost/` prefix, too. This will be covered +by e2e tests during the GA promotion. + +If both AppArmor annotations _and_ fields are specified, the values MUST match. +This will be enforced in API validation. + +If a Pod with a container specifies an AppArmor profile by field/annotation, +then the container will only apply the Pod level field/annotation if no none +are set on the container level. + +To raise awareness of annotation usage (in case of old automation), a warning +mechanism will be used to highlight that support will be dropped in v1.24. +The mechanisms being considered are audit annotations, annotations on the +object, events, or a warning as described in [KEP +#1693](/keps/sig-api-machinery/1693-warnings). + +#### PodSecurityPolicy Enforcement + +The PodSecurityPolicy admission controller must continue to check the PSP object +for annotations, as well as for fields. + +When setting default profiles, PSP only needs to set the field, but should only +do so if neither the field nor the annotation are set. The API machinery will +handle setting the annotation as necessary. + +When enforcing allowed profiles, the PSP should check BOTH the annotations & +fields. This is implicitly given because annotations as well as the new field +will be immutable. + +#### Pod Update + +The AppArmor fields on a pod are immutable, which also applies to the +[annotation](https://github.com/kubernetes/kubernetes/blob/b46612a74224b0871a97dae819f5fb3a1763d0b9/pkg/apis/core/validation/validation.go#L177-L182). + +When an [Ephemeral Container](20190212-ephemeral-containers.md) is added, it +will follow the same rules for using or overriding the pod's AppArmor profile. +Ephemeral container's will never sync with an AppArmor annotation. + +#### PodTemplates + +PodTemplates (e.g. ReplaceSets, Deployments, StatefulSets, etc.) will be +ignored. The field/annotation resolution will happen on template instantiation. + +To raise awareness of existing controllers using the AppArmor annotations that +need to be migrated, a warning mechanism will be used to highlight that support +will be dropped in v1.24. + +The mechanisms being considered are audit annotations, annotations on the +object, events, or a warning as described in [KEP +#1693](/keps/sig-api-machinery/1693-warnings). + +#### Runtime Profiles + +The API Server will continue to reject annotations with runtime profiles +different than `runtime/default`, to maintain the existing behavior. + +Violations would lead to the error message: + +``` +Invalid value: "runtime/profile-name": must be a valid AppArmor profile +``` + +#### Kubelet Backwards compatibility + +The changes brought to the Kubelet by this KEP will ensure backwards +compatibility in a similar way the changes above define it at API Server level. +Therefore, the AppArmor profiles will be applied following the priority order: + +1. Container-specific field. +2. Container-specific annotation. +3. Pod-wide field. + +In case annotations and fields at either container or pod level exist, the +kubelet will ignore the annotations and will only apply the profile defined on +the relevant field. + +#### Upgrade / Downgrade + +Nodes do not currently support in-place upgrades, so pods will be recreated on +node upgrade and downgrade. No special handling or consideration is needed to +support this. + +On the API server side, we've already taken version skew in HA clusters into +account. The same precautions make upgrade & downgrade handling a non-issue. + +Since +[we support](https://kubernetes.io/docs/setup/release/version-skew-policy/) up +to 2 minor releases of version skew between the master and node, annotations +must continue to be supported and backfilled for at least 2 versions passed the +initial implementation. However, we can decide to extend support farther to +reduce breakage. If this feature is implemented in v1.20, I propose v1.24 as a +target for removal of the old behavior. + +### Test Plan + +AppArmor already has [e2e tests][https://github.com/kubernetes/kubernetes/blob/6596a14/test/e2e_node/apparmor_test.go], +but the tests are guarded by the `[Feature:AppArmor]` tag and not run in the +standard test suites. + +Tests will be tagged as `[Feature:AppArmor]` like it is implemented right now, +but they will be migrated to use the new fields API. + +New tests will be added covering the annotation/field conflict cases described +under [Version Skew Strategy](#version-skew-strategy). + +Test coverage for localhost profiles will be added, unless we decide to [keep +localhost support in alpha](#alternatives). + +### Graduation Criteria + +_This section is excluded, as it is the subject of the entire proposal._ + +## Production Readiness Review Questionnaire + + + +### Feature enablement and rollback + +_This section must be completed when targeting alpha to a release._ + +- **How can this feature be enabled / disabled in a live cluster?** + + The feature can be enabled/disabled by the `AppArmor` feature gate. This + feature gate can be used to disable the feature until it reaches GA. + +- **Can the feature be disabled once it has been enabled (i.e. can we rollback + the enablement)?** + + Yes, it works in the same way as before moving the feature to GA. However, the + GA related changes are backwards compatible, and the API supports rollback of + the Kubernetes API Server as described in the [Version Skew + Strategy](#version-skew-strategy). + +- **What happens if we reenable the feature if it was previously rolled back?** + + N/A - the feature is already enabled by default since Kubernetes v1.4. + +- **Are there any tests for feature enablement/disablement?** + The e2e framework does not currently support enabling and disabling feature + gates. However, unit tests in each component dealing with managing data created + with and without the feature are necessary. At the very least, think about + conversion tests if API types are being modified. + + N/A - the feature is already enabled by default since Kubernetes v1.4. + +### Rollout, Upgrade and Rollback Planning + +_This section must be completed when targeting beta graduation to a release._ + +- **How can a rollout fail? Can it impact already running workloads?** + Try to be as paranoid as possible - e.g. what if some components will restart + in the middle of rollout? + + The [Version Skew Strategy](#version-skew-strategy) section covers this point. + Running workloads should have no impact as the Kubelet will support either the + existing annotations or the new fields introduced by this KEP. + +- **What specific metrics should inform a rollback?** + + N/A - the feature is already enabled by default since Kubernetes v1.4. + +- **Were upgrade and rollback tested? Was upgrade->downgrade->upgrade path tested?** + Describe manual testing that was done and the outcomes. + Longer term, we may want to require automated upgrade/rollback tests, but we + are missing a bunch of machinery and tooling and do that now. + + Automated tests will cover the scenarios with and without the changes proposed + on this KEP. As defined under [Version Skew Strategy](#version-skew-strategy), + we are assuming the cluster may have kubelets with older versions (without + this KEP' changes), therefore this will be covered as part of the new tests. + +### Monitoring requirements + +_This section must be completed when targeting beta graduation to a release._ + +- **How can an operator determine if the feature is in use by workloads?** + Ideally, this should be a metrics. Operations against Kubernetes API (e.g. + checking if there are objects with field X set) may be last resort. Avoid + logs or events for this purpose. + + The feature is built into the kubelet and api server components. No metric is + planned at this moment. The way to determine usage is by checking whether the + pods/containers have a AppArmorProfile set. + +- **What are the SLIs (Service Level Indicators) an operator can use to + determine the health of the service?** + + - [ ] Metrics + - Metric name: + - [Optional] Aggregation method: + - Components exposing the metric: + - [ ] Other (treat as last resort) + - Details: + + N/A + +- **What are the reasonable SLOs (Service Level Objectives) for the above SLIs?** + At the high-level this usually will be in the form of "high percentile of SLI + per day <= X". It's impossible to provide a comprehensive guidance, but at the + very high level (they needs more precise definitions) those may be things + like: + + - per-day percentage of API calls finishing with 5XX errors <= 1% + - 99% percentile over day of absolute value from (job creation time minus expected + job creation time) for cron job <= 10% + - 99,9% of /health requests per day finish with 200 code + + N/A + +- **Are there any missing metrics that would be useful to have to improve + observability in this feature?** + Describe the metrics themselves and the reason they weren't added (e.g. cost, + implementation difficulties, etc.). + + N/A + +### Dependencies + +_This section must be completed when targeting beta graduation to a release._ + +- **Does this feature depend on any specific services running in the cluster?** + Think about both cluster-level services (e.g. metrics-server) as well + as node-level agents (e.g. specific version of CRI). Focus on external or + optional services that are needed. For example, if this feature depends on + a cloud provider API, or upon an external software-defined storage or network + control plane. + + For each of the fill in the following, thinking both about running user + workloads and creating new ones, as well as about cluster-level services (e.g. + DNS): + + This KEP adds no new dependencies. + +### Scalability + +_For alpha, this section is encouraged: reviewers should consider these questions +and attempt to answer them._ + +_For beta, this section is required: reviewers must answer these questions._ + +_For GA, this section is required: approvers should be able to confirms the +previous answers based on experience in the field._ + +- **Will enabling / using this feature result in any new API calls?** + + NO + +- **Will enabling / using this feature result in introducing new API types?** + + NO + +- **Will enabling / using this feature result in any new calls to cloud + provider?** + + NO + +- **Will enabling / using this feature result in increasing size or count + of the existing API objects?** + + NO + +- **Will enabling / using this feature result in increasing time taken by any + operations covered by [existing SLIs/SLOs][]?** + + NO + +- **Will enabling / using this feature result in non-negligible increase of + resource usage (CPU, RAM, disk, IO, ...) in any components?** + Things to keep in mind include: additional in-memory state, additional + non-trivial computations, excessive access to disks (including increased log + volume), significant amount of data send and/or received over network, etc. + This through this both in small and large cases, again with respect to the + [supported limits][]. + + NO + +### Troubleshooting + +Troubleshooting section serves the `Playbook` role as of now. We may consider +splitting it into a dedicated `Playbook` document (potentially with some +monitoring details). For now we leave it here though. + +_This section must be completed when targeting beta graduation to a release._ + +- **How does this feature react if the API server and/or etcd is unavailable?** + + This is integral part of both API server and the Kubelet. All their + dependencies will impact + +- **What are other known failure modes?** + + No impact is being foreseen to running workloads based on the nature of + changes brought by this KEP. + + Although some general errors and failures can be seen on [Failure and Fallback + Strategy](#failure-and-fallback-strategy). + +* **What steps should be taken if SLOs are not being met to determine the problem?** + + N/A + +[supported limits]: https://git.k8s.io/community//sig-scalability/configs-and-limits/thresholds.md +[existing slis/slos]: https://git.k8s.io/community/sig-scalability/slos/slos.md#kubernetes-slisslos + +## Implementation History + +- 2020-01-10: Initial KEP +- 2020-08-24: Major rework and sync with seccomp + +## Drawbacks + +Promoting AppArmor as-is to GA may be seen as "blessing" the current +functionality, and make it harder to make some of the enhancements listed under +[Non-Goals](#non-goals). Since the current behavior is unguarded, I think we +already need to treat the behavior as GA. + +## Alternatives + +### Updating PodSecurityPolicy API + +PodSecurityPolicy AppArmor fields are mutable. On an update, the same rules are +applied as for creation, ignoring the old values. + +If only AppArmor annotations or fields are specified in the updated PSP, no +action is necessary, and the specified values are used. + +If both AppArmor annotations _and_ fields are specified in the updated PSP, the +values MUST match. + +```go +type PodSecurityPolicySpec struct { + ... + // AppArmor is the strategy that will dictate allowable and default AppArmor + // profiles for the container. + // +optional + AppArmor *ApparmorStrategyOptions + ... +} + +type AppArmorStrategyOptions struct { + // The default profile to set on the pod, if none is specified. + // The default MUST be allowed by the allowedProfiles. + // +optional + DefaultProfile *v1.AppArmorProfile + + // The set of profiles that may be set on the pod or containers. + // If unspecified, AppArmor profiles are unrestricted by this policy. + // +optional + AllowedProfiles *AppArmorProfileSet +} + +// A set of AppArmor profiles. This struct should be a plural of +// `v1.AppArmorProfile`. +// All values are optional, and an unspecified field excludes all profiles of +// that type from the set. +type AppArmorProfileSet struct { + // The allowed AppArmor profile types. + // +optional + Types []AppArmorProfileType + + // The allowed runtimeProfiles. A value of '*' allows all runtimeProfiles. + // +optional + RuntimeProfiles []string + + // The allowed localhostProfiles. Values may end in '*' to include all + // localhostProfiles with a prefix. + // +optional + LocalhostProfiles []string +} +``` diff --git a/keps/sig-node/24-apparmor-ga/kep.yaml b/keps/sig-node/24-apparmor-ga/kep.yaml new file mode 100644 index 00000000000..83fbcda95ba --- /dev/null +++ b/keps/sig-node/24-apparmor-ga/kep.yaml @@ -0,0 +1,48 @@ +title: AppArmor graduation to General Availability (GA) +kep-number: 24 +authors: + - "@saschagrunert" + - "@tallclair" +owning-sig: sig-node +participating-sigs: + - sig-api-machinery + - sig-auth +status: provisional +creation-date: 2020-01-10 +reviewers: + - "@liggitt" + - "@pjbgf" + - "@tallclair" +approvers: + - TBD +prr-approvers: + - "@deads2k" +see-also: + - "/keps/sig-node/20190717-seccomp-ga.md" + +# The target maturity stage in the current dev cycle for this KEP. +stage: stable + +# The most recent milestone for which work toward delivery of this KEP has been +# done. This can be the current (upcoming) milestone, if it is being actively +# worked on. +latest-milestone: "v1.23" + +# The milestone at which this feature was, or is targeted to be, at each stage. +milestone: + alpha: "v1.4" + beta: "v1.4" + stable: "v1.23" + +# The following PRR answers are required at alpha release +# List the feature gate name and the components for which it must be enabled +# feature-gates: +# - name: MyFeature +# components: +# - kube-apiserver +# - kube-controller-manager +# disable-supported: true + +# The following PRR answers are required at beta release +metrics: + - N/A From f3489bae52e317902beee41f75d3cb6a1eaed037 Mon Sep 17 00:00:00 2001 From: jan0ski Date: Mon, 25 Apr 2022 17:47:36 -0400 Subject: [PATCH 02/28] Remove PSP references --- keps/sig-node/24-apparmor-ga/README.md | 74 +------------------------- 1 file changed, 1 insertion(+), 73 deletions(-) diff --git a/keps/sig-node/24-apparmor-ga/README.md b/keps/sig-node/24-apparmor-ga/README.md index 1bceb24af81..0ed932ba116 100644 --- a/keps/sig-node/24-apparmor-ga/README.md +++ b/keps/sig-node/24-apparmor-ga/README.md @@ -17,7 +17,6 @@ - [Failure and Fallback Strategy](#failure-and-fallback-strategy) - [Version Skew Strategy](#version-skew-strategy) - [Pod Creation](#pod-creation) - - [PodSecurityPolicy Enforcement](#podsecuritypolicy-enforcement) - [Pod Update](#pod-update) - [PodTemplates](#podtemplates) - [Runtime Profiles](#runtime-profiles) @@ -34,8 +33,6 @@ - [Troubleshooting](#troubleshooting) - [Implementation History](#implementation-history) - [Drawbacks](#drawbacks) -- [Alternatives](#alternatives) - - [Updating PodSecurityPolicy API](#updating-podsecuritypolicy-api) ## Release Signoff Checklist @@ -126,8 +123,6 @@ as the current error propagation paths. The AppArmor API will be functionally equivalent to the current beta API, with the enhancement of adding pod level profiles to match the behavior with seccomp. This includes the Pod API, which specifies what profile the containers run with. -There are no changes planned to the current `PodSecurityPolicy` API as part of -this KEP. #### Pod API @@ -301,19 +296,6 @@ The mechanisms being considered are audit annotations, annotations on the object, events, or a warning as described in [KEP #1693](/keps/sig-api-machinery/1693-warnings). -#### PodSecurityPolicy Enforcement - -The PodSecurityPolicy admission controller must continue to check the PSP object -for annotations, as well as for fields. - -When setting default profiles, PSP only needs to set the field, but should only -do so if neither the field nor the annotation are set. The API machinery will -handle setting the annotation as necessary. - -When enforcing allowed profiles, the PSP should check BOTH the annotations & -fields. This is implicitly given because annotations as well as the new field -will be immutable. - #### Pod Update The AppArmor fields on a pod are immutable, which also applies to the @@ -615,6 +597,7 @@ _This section must be completed when targeting beta graduation to a release._ - 2020-01-10: Initial KEP - 2020-08-24: Major rework and sync with seccomp +- 2021-04-25: PSP mentions ## Drawbacks @@ -622,58 +605,3 @@ Promoting AppArmor as-is to GA may be seen as "blessing" the current functionality, and make it harder to make some of the enhancements listed under [Non-Goals](#non-goals). Since the current behavior is unguarded, I think we already need to treat the behavior as GA. - -## Alternatives - -### Updating PodSecurityPolicy API - -PodSecurityPolicy AppArmor fields are mutable. On an update, the same rules are -applied as for creation, ignoring the old values. - -If only AppArmor annotations or fields are specified in the updated PSP, no -action is necessary, and the specified values are used. - -If both AppArmor annotations _and_ fields are specified in the updated PSP, the -values MUST match. - -```go -type PodSecurityPolicySpec struct { - ... - // AppArmor is the strategy that will dictate allowable and default AppArmor - // profiles for the container. - // +optional - AppArmor *ApparmorStrategyOptions - ... -} - -type AppArmorStrategyOptions struct { - // The default profile to set on the pod, if none is specified. - // The default MUST be allowed by the allowedProfiles. - // +optional - DefaultProfile *v1.AppArmorProfile - - // The set of profiles that may be set on the pod or containers. - // If unspecified, AppArmor profiles are unrestricted by this policy. - // +optional - AllowedProfiles *AppArmorProfileSet -} - -// A set of AppArmor profiles. This struct should be a plural of -// `v1.AppArmorProfile`. -// All values are optional, and an unspecified field excludes all profiles of -// that type from the set. -type AppArmorProfileSet struct { - // The allowed AppArmor profile types. - // +optional - Types []AppArmorProfileType - - // The allowed runtimeProfiles. A value of '*' allows all runtimeProfiles. - // +optional - RuntimeProfiles []string - - // The allowed localhostProfiles. Values may end in '*' to include all - // localhostProfiles with a prefix. - // +optional - LocalhostProfiles []string -} -``` From 3ae85d2eca86908866797395dd99ea77a1b59d26 Mon Sep 17 00:00:00 2001 From: jan0ski Date: Mon, 25 Apr 2022 17:48:02 -0400 Subject: [PATCH 03/28] Update kep metadata --- keps/sig-node/24-apparmor-ga/README.md | 7 ++----- keps/sig-node/24-apparmor-ga/kep.yaml | 5 +++-- 2 files changed, 5 insertions(+), 7 deletions(-) diff --git a/keps/sig-node/24-apparmor-ga/README.md b/keps/sig-node/24-apparmor-ga/README.md index 0ed932ba116..a5e2121687a 100644 --- a/keps/sig-node/24-apparmor-ga/README.md +++ b/keps/sig-node/24-apparmor-ga/README.md @@ -89,8 +89,7 @@ The feature has been more or less unchanged ever since. The main motivation behind this KEP is to promote the AppArmor feature to GA. _NOTE: Seccomp was in a very similar state, but with some subtle differences. -Promoting Seccomp to GA will be covered by a [separate -KEP](/keps/sig-node/20190717-seccomp-ga.md)._ +Promoting Seccomp to GA was covered in [KEP-135](/keps/sig-node/135-seccomp/README.md)._ ### Goals @@ -180,7 +179,7 @@ profile cannot be set. This KEP proposes we GA LocalhostProfile as the only source of user-defined profiles at this point. User-defined profiles are essential for users to realize the full benefits out of AppArmor, allowing them to decrease their attack -surface based on their own workloads. +surface based on their own workloads. ###### Updating AppArmor profiles @@ -230,8 +229,6 @@ defined by using the `AppArmorProfileTypeRuntimeDefault`. The reasons being: that are different than `runtime/default`. - Most runtimes only support the default profile, although the CRI is flexible enough to allow the kubelet to send other profile names. -- Dockershim does not currently provide a way to pass other runtime profile - names. - Multiple runtime profiles has never been requested as a feature. If built-in support for multiple runtime profiles is needed in the future, a new diff --git a/keps/sig-node/24-apparmor-ga/kep.yaml b/keps/sig-node/24-apparmor-ga/kep.yaml index 83fbcda95ba..1de689a9366 100644 --- a/keps/sig-node/24-apparmor-ga/kep.yaml +++ b/keps/sig-node/24-apparmor-ga/kep.yaml @@ -7,6 +7,7 @@ owning-sig: sig-node participating-sigs: - sig-api-machinery - sig-auth + - sig-security status: provisional creation-date: 2020-01-10 reviewers: @@ -26,13 +27,13 @@ stage: stable # The most recent milestone for which work toward delivery of this KEP has been # done. This can be the current (upcoming) milestone, if it is being actively # worked on. -latest-milestone: "v1.23" +latest-milestone: "v1.25" # The milestone at which this feature was, or is targeted to be, at each stage. milestone: alpha: "v1.4" beta: "v1.4" - stable: "v1.23" + stable: "v1.26" # The following PRR answers are required at alpha release # List the feature gate name and the components for which it must be enabled From 8aef2f2163b6a174b03d5fd3298bdabcea715b85 Mon Sep 17 00:00:00 2001 From: jan0ski Date: Mon, 16 May 2022 22:13:09 -0400 Subject: [PATCH 04/28] Update KEP template and metadata --- keps/sig-node/24-apparmor-ga/24.yaml | 6 + keps/sig-node/24-apparmor-ga/README.md | 306 ++++++++++++++++--------- keps/sig-node/24-apparmor-ga/kep.yaml | 15 +- 3 files changed, 212 insertions(+), 115 deletions(-) create mode 100644 keps/sig-node/24-apparmor-ga/24.yaml diff --git a/keps/sig-node/24-apparmor-ga/24.yaml b/keps/sig-node/24-apparmor-ga/24.yaml new file mode 100644 index 00000000000..363dbc9e707 --- /dev/null +++ b/keps/sig-node/24-apparmor-ga/24.yaml @@ -0,0 +1,6 @@ +# The KEP must have an approver from the +# "prod-readiness-approvers" group +# of http://git.k8s.io/enhancements/OWNERS_ALIASES +kep-number: 24 +alpha: + approver: "@deads2k" diff --git a/keps/sig-node/24-apparmor-ga/README.md b/keps/sig-node/24-apparmor-ga/README.md index a5e2121687a..e519a196323 100644 --- a/keps/sig-node/24-apparmor-ga/README.md +++ b/keps/sig-node/24-apparmor-ga/README.md @@ -54,19 +54,14 @@ Items marked with (R) are required _prior to targeting to a milestone / release_ - [ ] Supporting documentation e.g., additional design documents, links to mailing list discussions/SIG meetings, relevant PRs/issues, release notes -**Note:** Any PRs to move a KEP to `implementable` or significant changes once -it is marked `implementable` should be approved by each of the KEP approvers. If -any of those approvers is no longer appropriate than changes to that list should -be approved by the remaining approvers and/or the owning SIG (or SIG-arch for -cross cutting KEPs). - -**Note:** This checklist is iterative and should be reviewed and updated every -time this enhancement is being considered for a milestone. + [kubernetes.io]: https://kubernetes.io/ -[kubernetes/enhancements]: https://github.com/kubernetes/enhancements/issues -[kubernetes/kubernetes]: https://github.com/kubernetes/kubernetes -[kubernetes/website]: https://github.com/kubernetes/website +[kubernetes/enhancements]: https://git.k8s.io/enhancements +[kubernetes/kubernetes]: https://git.k8s.io/kubernetes +[kubernetes/website]: https://git.k8s.io/website ## Summary @@ -78,7 +73,7 @@ minimum_ to clean up the feature, without blocking future enhancements. AppArmor support has been added with Kubernetes v1.4 and is already in beta. Profiles have to be available on each node whereas the container runtime ensures -that the profile is loaded when specified at pod or PSP level. Profiles can be +that the profile is loaded when specified at pod level. Profiles can be specified per-container via the pod’s metadata annotation: ``` @@ -288,7 +283,7 @@ then the container will only apply the Pod level field/annotation if no none are set on the container level. To raise awareness of annotation usage (in case of old automation), a warning -mechanism will be used to highlight that support will be dropped in v1.24. +mechanism will be used to highlight that support will be dropped in v1.29. The mechanisms being considered are audit annotations, annotations on the object, events, or a warning as described in [KEP #1693](/keps/sig-api-machinery/1693-warnings). @@ -309,7 +304,7 @@ ignored. The field/annotation resolution will happen on template instantiation. To raise awareness of existing controllers using the AppArmor annotations that need to be migrated, a warning mechanism will be used to highlight that support -will be dropped in v1.24. +will be dropped in v1.30. The mechanisms being considered are audit annotations, annotations on the object, events, or a warning as described in [KEP @@ -354,7 +349,7 @@ Since to 2 minor releases of version skew between the master and node, annotations must continue to be supported and backfilled for at least 2 versions passed the initial implementation. However, we can decide to extend support farther to -reduce breakage. If this feature is implemented in v1.20, I propose v1.24 as a +reduce breakage. If this feature is implemented in v1.26, I propose v1.30 as a target for removal of the old behavior. ### Test Plan @@ -381,17 +376,16 @@ _This section is excluded, as it is the subject of the entire proposal._ ### Feature enablement and rollback -_This section must be completed when targeting alpha to a release._ + -- **How can this feature be enabled / disabled in a live cluster?** +###### How can this feature be enabled / disabled in a live cluster? - The feature can be enabled/disabled by the `AppArmor` feature gate. This - feature gate can be used to disable the feature until it reaches GA. + + +- [X] Feature gate + - Feature gate name: `AppArmor` + - Components depending on the feature gate: + - kube-apiserver + - kubelet + +###### Does enabling the feature change any default behavior? + + N/A - the feature is already enabled by default since Kubernetes v1.4. + +###### Can the feature be disabled once it has been enabled (i.e. can we roll back the enablement)? Yes, it works in the same way as before moving the feature to GA. However, the GA related changes are backwards compatible, and the API supports rollback of the Kubernetes API Server as described in the [Version Skew Strategy](#version-skew-strategy). -- **What happens if we reenable the feature if it was previously rolled back?** +###### What happens if we reenable the feature if it was previously rolled back? N/A - the feature is already enabled by default since Kubernetes v1.4. -- **Are there any tests for feature enablement/disablement?** - The e2e framework does not currently support enabling and disabling feature - gates. However, unit tests in each component dealing with managing data created - with and without the feature are necessary. At the very least, think about - conversion tests if API types are being modified. +###### Are there any tests for feature enablement/disablement? + N/A - the feature is already enabled by default since Kubernetes v1.4. ### Rollout, Upgrade and Rollback Planning -_This section must be completed when targeting beta graduation to a release._ + + +###### How can a rollout or rollback fail? Can it impact already running workloads? + The [Version Skew Strategy](#version-skew-strategy) section covers this point. Running workloads should have no impact as the Kubelet will support either the existing annotations or the new fields introduced by this KEP. -- **What specific metrics should inform a rollback?** +###### What specific metrics should inform a rollback? N/A - the feature is already enabled by default since Kubernetes v1.4. -- **Were upgrade and rollback tested? Was upgrade->downgrade->upgrade path tested?** - Describe manual testing that was done and the outcomes. - Longer term, we may want to require automated upgrade/rollback tests, but we - are missing a bunch of machinery and tooling and do that now. +###### Were upgrade and rollback tested? Was the upgrade->downgrade->upgrade path tested? + Automated tests will cover the scenarios with and without the changes proposed on this KEP. As defined under [Version Skew Strategy](#version-skew-strategy), we are assuming the cluster may have kubelets with older versions (without this KEP' changes), therefore this will be covered as part of the new tests. +###### Is the rollout accompanied by any deprecations and/or removals of features, APIs, fields of API types, flags, etc.? + + The promotion of AppArmor to GA would deprecate the beta annotations as described in the + [Version Skew Strategy](#version-skew-strategy). + ### Monitoring requirements -_This section must be completed when targeting beta graduation to a release._ + -- **How can an operator determine if the feature is in use by workloads?** - Ideally, this should be a metrics. Operations against Kubernetes API (e.g. - checking if there are objects with field X set) may be last resort. Avoid - logs or events for this purpose. +###### How can an operator determine if the feature is in use by workloads? + + The feature is built into the kubelet and api server components. No metric is planned at this moment. The way to determine usage is by checking whether the pods/containers have a AppArmorProfile set. -- **What are the SLIs (Service Level Indicators) an operator can use to - determine the health of the service?** +###### How can someone using this feature know that it is working for their instance? - - [ ] Metrics - - Metric name: - - [Optional] Aggregation method: - - Components exposing the metric: - - [ ] Other (treat as last resort) - - Details: + + +- [ ] Events - Event Reason: +- [ ] API .status + - Condition name: + - Other field: +- [ ] Other (treat as last resort) + - Details: - N/A -- **What are the reasonable SLOs (Service Level Objectives) for the above SLIs?** - At the high-level this usually will be in the form of "high percentile of SLI - per day <= X". It's impossible to provide a comprehensive guidance, but at the - very high level (they needs more precise definitions) those may be things - like: +###### What are the reasonable SLOs (Service Level Objectives) for the enhancement? + N/A -- **Are there any missing metrics that would be useful to have to improve - observability in this feature?** - Describe the metrics themselves and the reason they weren't added (e.g. cost, - implementation difficulties, etc.). +###### What are the SLIs (Service Level Indicators) an operator can use to determine the health of the service? + + + + N/A + + +###### Are there any missing metrics that would be useful to have to improve observability of this feature? + N/A ### Dependencies -_This section must be completed when targeting beta graduation to a release._ + -- **Does this feature depend on any specific services running in the cluster?** - Think about both cluster-level services (e.g. metrics-server) as well - as node-level agents (e.g. specific version of CRI). Focus on external or - optional services that are needed. For example, if this feature depends on - a cloud provider API, or upon an external software-defined storage or network - control plane. +###### Does this feature depend on any specific services running in the cluster? - For each of the fill in the following, thinking both about running user - workloads and creating new ones, as well as about cluster-level services (e.g. - DNS): + This KEP adds no new dependencies. -### Scalability + -_For GA, this section is required: approvers should be able to confirms the -previous answers based on experience in the field._ +###### Will enabling / using this feature result in any new API calls? -- **Will enabling / using this feature result in any new API calls?** + NO -- **Will enabling / using this feature result in introducing new API types?** +###### Will enabling / using this feature result in introducing new API types? NO -- **Will enabling / using this feature result in any new calls to cloud - provider?** +###### Will enabling / using this feature result in any new calls to the cloud provider? NO -- **Will enabling / using this feature result in increasing size or count - of the existing API objects?** +###### Will enabling / using this feature result in increasing size or count of the existing API objects? NO -- **Will enabling / using this feature result in increasing time taken by any - operations covered by [existing SLIs/SLOs][]?** +###### Will enabling / using this feature result in increasing time taken by any operations covered by existing SLIs/SLOs? NO -- **Will enabling / using this feature result in non-negligible increase of - resource usage (CPU, RAM, disk, IO, ...) in any components?** - Things to keep in mind include: additional in-memory state, additional - non-trivial computations, excessive access to disks (including increased log - volume), significant amount of data send and/or received over network, etc. - This through this both in small and large cases, again with respect to the - [supported limits][]. +###### Will enabling / using this feature result in non-negligible increase of resource usage (CPU, RAM, disk, IO, ...) in any components? + NO ### Troubleshooting -Troubleshooting section serves the `Playbook` role as of now. We may consider -splitting it into a dedicated `Playbook` document (potentially with some -monitoring details). For now we leave it here though. + -- **How does this feature react if the API server and/or etcd is unavailable?** +###### How does this feature react if the API server and/or etcd is unavailable? - This is integral part of both API server and the Kubelet. All their - dependencies will impact + No impact to running workloads. -- **What are other known failure modes?** +###### What are other known failure modes? No impact is being foreseen to running workloads based on the nature of changes brought by this KEP. @@ -583,7 +674,7 @@ _This section must be completed when targeting beta graduation to a release._ Although some general errors and failures can be seen on [Failure and Fallback Strategy](#failure-and-fallback-strategy). -* **What steps should be taken if SLOs are not being met to determine the problem?** +###### What steps should be taken if SLOs are not being met to determine the problem? N/A @@ -595,6 +686,7 @@ _This section must be completed when targeting beta graduation to a release._ - 2020-01-10: Initial KEP - 2020-08-24: Major rework and sync with seccomp - 2021-04-25: PSP mentions +- 2022-05-07: Rework, removal of PSP mentions ## Drawbacks diff --git a/keps/sig-node/24-apparmor-ga/kep.yaml b/keps/sig-node/24-apparmor-ga/kep.yaml index 1de689a9366..35af1dc6b49 100644 --- a/keps/sig-node/24-apparmor-ga/kep.yaml +++ b/keps/sig-node/24-apparmor-ga/kep.yaml @@ -16,8 +16,7 @@ reviewers: - "@tallclair" approvers: - TBD -prr-approvers: - - "@deads2k" + see-also: - "/keps/sig-node/20190717-seccomp-ga.md" @@ -37,12 +36,12 @@ milestone: # The following PRR answers are required at alpha release # List the feature gate name and the components for which it must be enabled -# feature-gates: -# - name: MyFeature -# components: -# - kube-apiserver -# - kube-controller-manager -# disable-supported: true +feature-gates: + - name: AppArmor + components: + - kube-apiserver + - kubelet +disable-supported: false # The following PRR answers are required at beta release metrics: From 24ca43823dcff7d7c9b7d5a8466c900ecad814c3 Mon Sep 17 00:00:00 2001 From: jan0ski Date: Mon, 16 May 2022 22:29:52 -0400 Subject: [PATCH 05/28] Add initial design and beta release to implementation history --- keps/sig-node/24-apparmor-ga/README.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/keps/sig-node/24-apparmor-ga/README.md b/keps/sig-node/24-apparmor-ga/README.md index e519a196323..001bc6282b7 100644 --- a/keps/sig-node/24-apparmor-ga/README.md +++ b/keps/sig-node/24-apparmor-ga/README.md @@ -683,6 +683,8 @@ details). For now, we leave it here. ## Implementation History +- 2016-07-25: AppArmor design proposal +- 2016-09-26: AppArmor beta release with v1.4 - 2020-01-10: Initial KEP - 2020-08-24: Major rework and sync with seccomp - 2021-04-25: PSP mentions From de1d3bd5546c8c09988ac0afac63c23636b0a006 Mon Sep 17 00:00:00 2001 From: jan0ski Date: Mon, 16 May 2022 22:40:12 -0400 Subject: [PATCH 06/28] Update stable target to 1.25 --- keps/sig-node/24-apparmor-ga/README.md | 4 ++-- keps/sig-node/24-apparmor-ga/kep.yaml | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/keps/sig-node/24-apparmor-ga/README.md b/keps/sig-node/24-apparmor-ga/README.md index 001bc6282b7..745423ccd18 100644 --- a/keps/sig-node/24-apparmor-ga/README.md +++ b/keps/sig-node/24-apparmor-ga/README.md @@ -304,7 +304,7 @@ ignored. The field/annotation resolution will happen on template instantiation. To raise awareness of existing controllers using the AppArmor annotations that need to be migrated, a warning mechanism will be used to highlight that support -will be dropped in v1.30. +will be dropped in v1.29. The mechanisms being considered are audit annotations, annotations on the object, events, or a warning as described in [KEP @@ -349,7 +349,7 @@ Since to 2 minor releases of version skew between the master and node, annotations must continue to be supported and backfilled for at least 2 versions passed the initial implementation. However, we can decide to extend support farther to -reduce breakage. If this feature is implemented in v1.26, I propose v1.30 as a +reduce breakage. If this feature is implemented in v1.25, I propose v1.29 as a target for removal of the old behavior. ### Test Plan diff --git a/keps/sig-node/24-apparmor-ga/kep.yaml b/keps/sig-node/24-apparmor-ga/kep.yaml index 35af1dc6b49..94a29123f0d 100644 --- a/keps/sig-node/24-apparmor-ga/kep.yaml +++ b/keps/sig-node/24-apparmor-ga/kep.yaml @@ -26,13 +26,13 @@ stage: stable # The most recent milestone for which work toward delivery of this KEP has been # done. This can be the current (upcoming) milestone, if it is being actively # worked on. -latest-milestone: "v1.25" +latest-milestone: "v1.24" # The milestone at which this feature was, or is targeted to be, at each stage. milestone: alpha: "v1.4" beta: "v1.4" - stable: "v1.26" + stable: "v1.25" # The following PRR answers are required at alpha release # List the feature gate name and the components for which it must be enabled From fca578e9a190315a39a8a28678c060f874d4b76e Mon Sep 17 00:00:00 2001 From: Tommy McCormick <16668493+jan0ski@users.noreply.github.com> Date: Wed, 22 Jun 2022 21:13:58 -0400 Subject: [PATCH 07/28] Change target to 1.26 --- keps/sig-node/24-apparmor-ga/README.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/keps/sig-node/24-apparmor-ga/README.md b/keps/sig-node/24-apparmor-ga/README.md index 745423ccd18..3670cc7afd4 100644 --- a/keps/sig-node/24-apparmor-ga/README.md +++ b/keps/sig-node/24-apparmor-ga/README.md @@ -283,7 +283,7 @@ then the container will only apply the Pod level field/annotation if no none are set on the container level. To raise awareness of annotation usage (in case of old automation), a warning -mechanism will be used to highlight that support will be dropped in v1.29. +mechanism will be used to highlight that support will be dropped in v1.30. The mechanisms being considered are audit annotations, annotations on the object, events, or a warning as described in [KEP #1693](/keps/sig-api-machinery/1693-warnings). @@ -304,7 +304,7 @@ ignored. The field/annotation resolution will happen on template instantiation. To raise awareness of existing controllers using the AppArmor annotations that need to be migrated, a warning mechanism will be used to highlight that support -will be dropped in v1.29. +will be dropped in v1.30. The mechanisms being considered are audit annotations, annotations on the object, events, or a warning as described in [KEP @@ -349,7 +349,7 @@ Since to 2 minor releases of version skew between the master and node, annotations must continue to be supported and backfilled for at least 2 versions passed the initial implementation. However, we can decide to extend support farther to -reduce breakage. If this feature is implemented in v1.25, I propose v1.29 as a +reduce breakage. If this feature is implemented in v1.26, I propose v1.30 as a target for removal of the old behavior. ### Test Plan From db15333f0b4aa0f1770586f14abbf03c66381994 Mon Sep 17 00:00:00 2001 From: Tommy McCormick <16668493+jan0ski@users.noreply.github.com> Date: Wed, 22 Jun 2022 21:14:56 -0400 Subject: [PATCH 08/28] Change target to 1.26 --- keps/sig-node/24-apparmor-ga/kep.yaml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/keps/sig-node/24-apparmor-ga/kep.yaml b/keps/sig-node/24-apparmor-ga/kep.yaml index 94a29123f0d..35af1dc6b49 100644 --- a/keps/sig-node/24-apparmor-ga/kep.yaml +++ b/keps/sig-node/24-apparmor-ga/kep.yaml @@ -26,13 +26,13 @@ stage: stable # The most recent milestone for which work toward delivery of this KEP has been # done. This can be the current (upcoming) milestone, if it is being actively # worked on. -latest-milestone: "v1.24" +latest-milestone: "v1.25" # The milestone at which this feature was, or is targeted to be, at each stage. milestone: alpha: "v1.4" beta: "v1.4" - stable: "v1.25" + stable: "v1.26" # The following PRR answers are required at alpha release # List the feature gate name and the components for which it must be enabled From 87f054054a13b8a96042288ed3652e4421169171 Mon Sep 17 00:00:00 2001 From: Tommy McCormick <16668493+jan0ski@users.noreply.github.com> Date: Wed, 28 Sep 2022 08:02:05 -0400 Subject: [PATCH 09/28] Delete 24.yaml --- keps/sig-node/24-apparmor-ga/24.yaml | 6 ------ 1 file changed, 6 deletions(-) delete mode 100644 keps/sig-node/24-apparmor-ga/24.yaml diff --git a/keps/sig-node/24-apparmor-ga/24.yaml b/keps/sig-node/24-apparmor-ga/24.yaml deleted file mode 100644 index 363dbc9e707..00000000000 --- a/keps/sig-node/24-apparmor-ga/24.yaml +++ /dev/null @@ -1,6 +0,0 @@ -# The KEP must have an approver from the -# "prod-readiness-approvers" group -# of http://git.k8s.io/enhancements/OWNERS_ALIASES -kep-number: 24 -alpha: - approver: "@deads2k" From 9b5bcff6446d7ac402e31d5a986104912c71a999 Mon Sep 17 00:00:00 2001 From: Tommy McCormick <16668493+jan0ski@users.noreply.github.com> Date: Wed, 28 Sep 2022 08:19:24 -0400 Subject: [PATCH 10/28] change wording from "GA" to general AA support --- .../{24-apparmor-ga => 24-apparmor}/README.md | 31 ++++++------------- .../{24-apparmor-ga => 24-apparmor}/kep.yaml | 2 +- 2 files changed, 10 insertions(+), 23 deletions(-) rename keps/sig-node/{24-apparmor-ga => 24-apparmor}/README.md (95%) rename keps/sig-node/{24-apparmor-ga => 24-apparmor}/kep.yaml (95%) diff --git a/keps/sig-node/24-apparmor-ga/README.md b/keps/sig-node/24-apparmor/README.md similarity index 95% rename from keps/sig-node/24-apparmor-ga/README.md rename to keps/sig-node/24-apparmor/README.md index 3670cc7afd4..729b6e70792 100644 --- a/keps/sig-node/24-apparmor-ga/README.md +++ b/keps/sig-node/24-apparmor/README.md @@ -1,4 +1,4 @@ -# AppArmor graduation to General Availability (GA) +# Add AppArmor Support ## Table of Contents @@ -65,30 +65,17 @@ Items marked with (R) are required _prior to targeting to a milestone / release_ ## Summary -This is a proposal to upgrade the AppArmor annotation on pods to a dedicated -field, and graduate the feature to GA. This proposal aims to do the _bare -minimum_ to clean up the feature, without blocking future enhancements. +This is a proposal to add AppArmor support to the Kubernetes API. This proposal aims to do the _bare +minimum_ to clean up the feature from its beta release, without blocking future enhancements. ## Motivation -AppArmor support has been added with Kubernetes v1.4 and is already in beta. -Profiles have to be available on each node whereas the container runtime ensures -that the profile is loaded when specified at pod level. Profiles can be -specified per-container via the pod’s metadata annotation: - -``` -container.apparmor.security.beta.kubernetes.io/: {unconfined,runtime/default,localhost/} -``` - -The feature has been more or less unchanged ever since. The main motivation -behind this KEP is to promote the AppArmor feature to GA. - -_NOTE: Seccomp was in a very similar state, but with some subtle differences. -Promoting Seccomp to GA was covered in [KEP-135](/keps/sig-node/135-seccomp/README.md)._ +AppArmor can enable users to run a more secure deployment, and/or provide better auditing and +monitoring of their systems. AppArmor for should be supported to provide users a simpler alternative +to SELinux, or to provide an interface for users that are already maintaining a set of AppArmor profiles. ### Goals -- Declare AppArmor as GA - Fully document and formally spec the feature support - Add equivalent API fields to replace AppArmor annotations and provide a pod level field, which applies to all containers. @@ -171,7 +158,7 @@ profile cannot be set. ##### Localhost Profile -This KEP proposes we GA LocalhostProfile as the only source of user-defined +This KEP proposes LocalhostProfile as the only source of user-defined profiles at this point. User-defined profiles are essential for users to realize the full benefits out of AppArmor, allowing them to decrease their attack surface based on their own workloads. @@ -243,7 +230,7 @@ below are the ones we mapped and their outcome once this KEP is implemented: | 3) Using custom profile that does not exist on the node. | Pod created | Containers fail to start. Retry respecting RestartPolicy and back-off delay. | | 4) Using an unsupported runtime profile (i.e. `runtime/default-audit`). | Pod **not** created. | N/A | | 5) Using localhost profile with invalid name | Pod **not** created. | N/A | -| 6) AppArmor is disabled by the host or the build | Pod **not** created. | Kubelet puts Pod in blocked state. | +| 6) AppArmor is disabled by the host or the build | Pod created. | Kubelet puts Pod in blocked state. | Scenario 2 is the expected behavior of using AppArmor and it is included here for completeness. @@ -279,7 +266,7 @@ If both AppArmor annotations _and_ fields are specified, the values MUST match. This will be enforced in API validation. If a Pod with a container specifies an AppArmor profile by field/annotation, -then the container will only apply the Pod level field/annotation if no none +then the container will only apply the Pod level field/annotation if none are set on the container level. To raise awareness of annotation usage (in case of old automation), a warning diff --git a/keps/sig-node/24-apparmor-ga/kep.yaml b/keps/sig-node/24-apparmor/kep.yaml similarity index 95% rename from keps/sig-node/24-apparmor-ga/kep.yaml rename to keps/sig-node/24-apparmor/kep.yaml index 35af1dc6b49..481f14167d9 100644 --- a/keps/sig-node/24-apparmor-ga/kep.yaml +++ b/keps/sig-node/24-apparmor/kep.yaml @@ -1,4 +1,4 @@ -title: AppArmor graduation to General Availability (GA) +title: Add AppArmor Support kep-number: 24 authors: - "@saschagrunert" From ac8121e66fed78886d6b8c68ab320d49eccd2a23 Mon Sep 17 00:00:00 2001 From: Tommy McCormick <16668493+jan0ski@users.noreply.github.com> Date: Wed, 28 Sep 2022 08:28:22 -0400 Subject: [PATCH 11/28] Add mention of legacy default and allowed profile annotations --- keps/sig-node/24-apparmor/README.md | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/keps/sig-node/24-apparmor/README.md b/keps/sig-node/24-apparmor/README.md index 729b6e70792..faa7d9e91e7 100644 --- a/keps/sig-node/24-apparmor/README.md +++ b/keps/sig-node/24-apparmor/README.md @@ -17,6 +17,7 @@ - [Failure and Fallback Strategy](#failure-and-fallback-strategy) - [Version Skew Strategy](#version-skew-strategy) - [Pod Creation](#pod-creation) + - [PodSecurityPolicy Support](#pod-security-policy-fields) - [Pod Update](#pod-update) - [PodTemplates](#podtemplates) - [Runtime Profiles](#runtime-profiles) @@ -275,6 +276,17 @@ The mechanisms being considered are audit annotations, annotations on the object, events, or a warning as described in [KEP #1693](/keps/sig-api-machinery/1693-warnings). +#### PodSecurityPolicy Support + +PodSecurityPolicy support has been removed in Kubernetes 1.25. Due to this, +the equivalent functionality for annotations such as the below will not be supported: + + * `apparmor.security.beta.kubernetes.io/defaultProfileName` + * `apparmor.security.beta.kubernetes.io/allowedProfileNames` + +To provide this functionality, users will need to implement equivalent rules in admission control, +by injecting or validating a `LocalHostProfile` field for Pods or individual containers. + #### Pod Update The AppArmor fields on a pod are immutable, which also applies to the From 79fb505b813e819799e937a68c59c7814fa71f17 Mon Sep 17 00:00:00 2001 From: Tommy McCormick <16668493+jan0ski@users.noreply.github.com> Date: Wed, 28 Sep 2022 08:48:37 -0400 Subject: [PATCH 12/28] Add scalability section --- keps/sig-node/24-apparmor/README.md | 88 +++++++++++++++++++++++++++++ 1 file changed, 88 insertions(+) diff --git a/keps/sig-node/24-apparmor/README.md b/keps/sig-node/24-apparmor/README.md index faa7d9e91e7..7d3b2c52f53 100644 --- a/keps/sig-node/24-apparmor/README.md +++ b/keps/sig-node/24-apparmor/README.md @@ -648,6 +648,94 @@ This through this both in small and large cases, again with respect to the NO +### Scalability + + + +###### Will enabling / using this feature result in any new API calls? + + + +No + +###### Will enabling / using this feature result in introducing new API types? + + + +No, it will only add an `AppArmorProfile` field to existing types. + +###### Will enabling / using this feature result in any new calls to the cloud provider? + + + +No + +###### Will enabling / using this feature result in increasing size or count of the existing API objects? + + + +New container-level and pod-level fields. + +###### Will enabling / using this feature result in increasing time taken by any operations covered by existing SLIs/SLOs? + + + +No + +###### Will enabling / using this feature result in non-negligible increase of resource usage (CPU, RAM, disk, IO, ...) in any components? + + + +No + ### Troubleshooting ## Release Signoff Checklist @@ -783,3 +784,7 @@ Promoting AppArmor as-is to GA may be seen as "blessing" the current functionality, and make it harder to make some of the enhancements listed under [Non-Goals](#non-goals). Since the current behavior is unguarded, I think we already need to treat the behavior as GA. + +## Alternatives + +N/A \ No newline at end of file From 79d7f3b88a75a5833e0c54b3de3104f90605f949 Mon Sep 17 00:00:00 2001 From: Tommy McCormick <16668493+jan0ski@users.noreply.github.com> Date: Wed, 5 Oct 2022 10:52:48 -0400 Subject: [PATCH 14/28] reference annotation handling with version skew --- keps/sig-node/24-apparmor/README.md | 11 +++++------ 1 file changed, 5 insertions(+), 6 deletions(-) diff --git a/keps/sig-node/24-apparmor/README.md b/keps/sig-node/24-apparmor/README.md index d5fc0aefe9b..f3dcc24745f 100644 --- a/keps/sig-node/24-apparmor/README.md +++ b/keps/sig-node/24-apparmor/README.md @@ -348,9 +348,11 @@ Since [we support](https://kubernetes.io/docs/setup/release/version-skew-policy/) up to 2 minor releases of version skew between the master and node, annotations must continue to be supported and backfilled for at least 2 versions passed the -initial implementation. However, we can decide to extend support farther to -reduce breakage. If this feature is implemented in v1.26, I propose v1.30 as a -target for removal of the old behavior. +initial implementation. If this feature is implemented in v1.26, I propose v1.30 as a +target for removal of the old behavior. Specifically, annotation support will be removed +in the kubelet after this period, and fields will no longer be copied to annotations for older kubelet +versions. However, annotations submitted to the API server will continue to be copied to fields at the +kubelet indefinitely, as was done with Seccomp. ### Test Plan @@ -364,9 +366,6 @@ but they will be migrated to use the new fields API. New tests will be added covering the annotation/field conflict cases described under [Version Skew Strategy](#version-skew-strategy). -Test coverage for localhost profiles will be added, unless we decide to [keep -localhost support in alpha](#alternatives). - ### Graduation Criteria _This section is excluded, as it is the subject of the entire proposal._ From e29bc65072c8aa1455e222eabe12eea4441577d6 Mon Sep 17 00:00:00 2001 From: Tommy McCormick <16668493+jan0ski@users.noreply.github.com> Date: Thu, 6 Oct 2022 12:48:50 -0400 Subject: [PATCH 15/28] Update kep.yaml --- keps/sig-node/24-apparmor/kep.yaml | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/keps/sig-node/24-apparmor/kep.yaml b/keps/sig-node/24-apparmor/kep.yaml index 481f14167d9..5a9a028672a 100644 --- a/keps/sig-node/24-apparmor/kep.yaml +++ b/keps/sig-node/24-apparmor/kep.yaml @@ -8,12 +8,11 @@ participating-sigs: - sig-api-machinery - sig-auth - sig-security -status: provisional +status: implementable creation-date: 2020-01-10 reviewers: - "@liggitt" - "@pjbgf" - - "@tallclair" approvers: - TBD @@ -26,7 +25,7 @@ stage: stable # The most recent milestone for which work toward delivery of this KEP has been # done. This can be the current (upcoming) milestone, if it is being actively # worked on. -latest-milestone: "v1.25" +latest-milestone: "v1.26" # The milestone at which this feature was, or is targeted to be, at each stage. milestone: From 8ee5951e47c694c85ed1c305fd30da0a3e20a659 Mon Sep 17 00:00:00 2001 From: Tommy McCormick <16668493+jan0ski@users.noreply.github.com> Date: Thu, 6 Oct 2022 12:55:22 -0400 Subject: [PATCH 16/28] Update kep.yaml --- keps/sig-node/24-apparmor/kep.yaml | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/keps/sig-node/24-apparmor/kep.yaml b/keps/sig-node/24-apparmor/kep.yaml index 5a9a028672a..d37acfc8edd 100644 --- a/keps/sig-node/24-apparmor/kep.yaml +++ b/keps/sig-node/24-apparmor/kep.yaml @@ -2,7 +2,7 @@ title: Add AppArmor Support kep-number: 24 authors: - "@saschagrunert" - - "@tallclair" + - "@jan0ski" owning-sig: sig-node participating-sigs: - sig-api-machinery @@ -13,6 +13,7 @@ creation-date: 2020-01-10 reviewers: - "@liggitt" - "@pjbgf" + - "@tallclair" approvers: - TBD From 6de8b0031e0dfda6aeecab26a0c4e7fc91278de5 Mon Sep 17 00:00:00 2001 From: Tommy McCormick <16668493+jan0ski@users.noreply.github.com> Date: Thu, 6 Oct 2022 16:53:28 -0400 Subject: [PATCH 17/28] Address comments --- keps/sig-node/24-apparmor/README.md | 32 +++++++++++++++-------------- keps/sig-node/24-apparmor/kep.yaml | 6 ++++-- 2 files changed, 21 insertions(+), 17 deletions(-) diff --git a/keps/sig-node/24-apparmor/README.md b/keps/sig-node/24-apparmor/README.md index f3dcc24745f..2bb1d8e7bf2 100644 --- a/keps/sig-node/24-apparmor/README.md +++ b/keps/sig-node/24-apparmor/README.md @@ -41,7 +41,7 @@ Items marked with (R) are required _prior to targeting to a milestone / release_. -- [ ] (R) Enhancement issue in release milestone, which links to KEP dir in +- [X] (R) Enhancement issue in release milestone, which links to KEP dir in [kubernetes/enhancements] (not the initial KEP PR) - [ ] (R) KEP approvers have approved the KEP status as `implementable` - [ ] (R) Design details are appropriately documented @@ -50,7 +50,7 @@ Items marked with (R) are required _prior to targeting to a milestone / release_ - [ ] (R) Graduation criteria is in place - [ ] (R) Production readiness review completed - [ ] Production readiness review approved -- [ ] "Implementation History" section is up-to-date for milestone +- [X] "Implementation History" section is up-to-date for milestone - [ ] User-facing documentation has been created in [kubernetes/website], for publication to [kubernetes.io] - [ ] Supporting documentation e.g., additional design documents, links to @@ -163,7 +163,9 @@ profile cannot be set. This KEP proposes LocalhostProfile as the only source of user-defined profiles at this point. User-defined profiles are essential for users to realize the full benefits out of AppArmor, allowing them to decrease their attack -surface based on their own workloads. +surface based on their own workloads. Only profiles with a specified prefix will be +available to Localhost profiles. This prevents profiles meant for other system +daemons to be utilized by Kubernetes and will be configurable by the kubelet. ###### Updating AppArmor profiles @@ -227,7 +229,7 @@ below are the ones we mapped and their outcome once this KEP is implemented: | Scenario | API Server Result | Kubelet Result | | -------------------------------------------------------------------------------------------------- | -------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------- | -| 1) Using custom or `runtime/default` profile when container runtime does not support AppArmor. | Pod created | The outcome is container runtime dependent. In this scenario containers may 1) fail to start or 2) run normally without having its policies enforced. | +| 1) Using localhost or `runtime/default` profile when container runtime does not support AppArmor. | Pod created | The outcome is container runtime dependent. In this scenario containers may 1) fail to start or 2) run normally without having its policies enforced. | | 2) Using custom or `runtime/default` profile that restricts actions a container is trying to make. | Pod created | The outcome is workload and AppArmor dependent. In this scenario containers may 1) fail to start, 2) misbehave or 3) log violations. | | 3) Using custom profile that does not exist on the node. | Pod created | Containers fail to start. Retry respecting RestartPolicy and back-off delay. | | 4) Using an unsupported runtime profile (i.e. `runtime/default-audit`). | Pod **not** created. | N/A | @@ -252,9 +254,10 @@ If no AppArmor annotations or fields are specified, no action is necessary. If the `AppArmor` feature is disabled per feature gate, then the annotations and fields are cleared (current behavior). -If _only_ AppArmor fields are specified, add the corresponding annotations. This -ensures that the fields are enforced even if the node version trails the API -version (see [Version Skew Strategy](##version-skew-strategy)). +If _only_ AppArmor fields are specified, add the corresponding annotations. If these +are specified at the Pod level, copy the annotations to each container that does +not have annotations already specified. This ensures that the fields are enforced +even if the node version trails the API version (see [Version Skew Strategy](##version-skew-strategy)). If _only_ AppArmor annotations are specified, copy the values into the corresponding fields. This ensures that existing applications continue to @@ -331,7 +334,7 @@ Therefore, the AppArmor profiles will be applied following the priority order: 2. Container-specific annotation. 3. Pod-wide field. -In case annotations and fields at either container or pod level exist, the +In case both annotations and fields at the container level exist, the kubelet will ignore the annotations and will only apply the profile defined on the relevant field. @@ -472,7 +475,10 @@ will rollout across nodes. ###### What specific metrics should inform a rollback? - N/A - the feature is already enabled by default since Kubernetes v1.4. + Clusters upgrading while using beta AppArmor annotations will want to ensure + that profiles on upgraded nodes are loaded at the Kubelet's specified path prefix. + Containers of Pods loading AppArmor profiles will fail to start if they attempt to + load non-Kubernetes profiles. ###### Were upgrade and rollback tested? Was the upgrade->downgrade->upgrade path tested? -- [ ] Events - Event Reason: -- [ ] API .status - - Condition name: - - Other field: -- [ ] Other (treat as last resort) - - Details: +Pod events will provide details of profiles being successfully applied to specific containers. +- [X] Events - Event Reason: `AppArmor` ###### What are the reasonable SLOs (Service Level Objectives) for the enhancement? -- [X] Feature gate +- [X] Feature gate - Feature gate name: `AppArmor` - Components depending on the feature gate: - kube-apiserver - kubelet ###### Does enabling the feature change any default behavior? - + N/A - the feature is already enabled by default since Kubernetes v1.4. ###### Can the feature be disabled once it has been enabled (i.e. can we roll back the enablement)? @@ -474,9 +496,9 @@ will rollout across nodes. ###### What specific metrics should inform a rollback? - Clusters upgrading while using beta AppArmor annotations will want to ensure + Clusters upgrading while using beta AppArmor annotations will want to ensure that profiles on upgraded nodes are loaded at the Kubelet's specified path prefix. - Containers of Pods loading AppArmor profiles will fail to start if they attempt to + Containers of Pods loading AppArmor profiles will fail to start if they attempt to load non-Kubernetes profiles. Monitoring the below metrics can help identify these issues: @@ -496,8 +518,8 @@ are missing a bunch of machinery and tooling and can't do that now. this KEP' changes), therefore this will be covered as part of the new tests. ###### Is the rollout accompanied by any deprecations and/or removals of features, APIs, fields of API types, flags, etc.? - - The promotion of AppArmor to GA would deprecate the beta annotations as described in the + + The promotion of AppArmor to GA would deprecate the beta annotations as described in the [Version Skew Strategy](#version-skew-strategy). ### Monitoring requirements @@ -516,7 +538,7 @@ Ideally, this should be a metric. Operations against the Kubernetes API (e.g., checking if there are objects with field X set) may be a last resort. Avoid logs or events for this purpose. --> - + The feature is built into the kubelet and api server components. No metric is planned at this moment. The way to determine usage is by checking whether the pods/containers have a AppArmorProfile set. @@ -534,7 +556,7 @@ Recall that end users cannot usually observe component logs or access metrics. Pod events will provide details of profiles being successfully applied to specific containers. -- [X] Events - Event Reason: `AppArmor` +- [X] Events - Event Reason: `AppArmor` ###### What are the reasonable SLOs (Service Level Objectives) for the enhancement?