From da76cd07fa2958895ed28376a0c920b266b6529b Mon Sep 17 00:00:00 2001 From: Tim Allclair Date: Wed, 11 May 2022 11:18:45 -0700 Subject: [PATCH 1/5] Pod Security GA plan --- keps/sig-auth/2579-psp-replacement/README.md | 35 ++++++++++++-------- keps/sig-auth/2579-psp-replacement/kep.yaml | 8 ++--- 2 files changed, 26 insertions(+), 17 deletions(-) diff --git a/keps/sig-auth/2579-psp-replacement/README.md b/keps/sig-auth/2579-psp-replacement/README.md index 4e177d1ddf1..0ffb85841ba 100644 --- a/keps/sig-auth/2579-psp-replacement/README.md +++ b/keps/sig-auth/2579-psp-replacement/README.md @@ -735,15 +735,25 @@ We are targeting Beta in v1.23. #### GA -<<[UNRESOLVED]>> +Targeting GA in v1.25. -We are targeting GA in v1.24 to allow for migration off PodSecurityPolicy before it is removed in -v1.25. +**Conformance:** +- Enabling the admission controller with the "default-default" enforcing mode of privileged is + essentially a no-op without adding namespace labels, so it doesn't have any impact on + conformance. +- E2E framework has been updated to explicitly label test namespaces with the appropriate + enforcement level, using the `NamespacePodSecurityEnforceLevel` framework value. For GA, + conformance tests should be updated to use the most restrictive level possible. +- Pod Security Admission is *not* required for conformance. -- Examples of real world usage and positive user feedback. -- [Conformance test plan](#conformance) +**User Experience Improvements:** +- [Warn when labeling exempt namespaces](https://github.com/kubernetes/kubernetes/issues/109549) +- [Dedupe overlapping forbidden messages](https://github.com/kubernetes/kubernetes/issues/106129) +- [Aggregate identical warnings for multiple pods in a namespace](https://github.com/kubernetes/kubernetes/issues/103213) +- [Add context to failure messages](https://github.com/kubernetes/kubernetes/pull/105314) -<<[/UNRESOLVED]>> +**API Changes:** +- Add `pod-security.admission.config.k8s.io/v1` with no changes from the `v1beta1` API. ### Upgrade / Downgrade Strategy @@ -910,6 +920,8 @@ previous answers based on experience in the field._ There will be a hard cap on the number of pods analyzed, and a timeout for the review of those pods that ensures evaluation does not exceed a percentage of the time allocated to the request. See [Namespace policy update warnings](#namespace-policy-update-warnings). + - Timeout: minimum of 1 second or (remaining request deadline / 2) + - Max pods to check: 3000 ([benchmarks](https://github.com/kubernetes/kubernetes/pull/104588) indicate that 3000 pods should evaluate in under 10ms) * **Will enabling / using this feature result in introducing new API types?** - No. @@ -1046,13 +1058,10 @@ templated pod resources. This could be useful in CI/CD pipelines and tests. ### Conformance -As this feature progresses towards GA, we should think more about how it interacts with conformance. - -- Enabling the admission controller with the "default-default" enforcing mode of privileged is - essentially a no-op without adding namespace labels, so it shouldn't have any impact on - conformance. -- If we want a more restricted version to still be considered conformant, we might need to - explicitly label namespaces in the conformance tests with the privilege level the tests require. +Clusters requiring baseline or restricted Pod Security levels should still be able to pass +conformance. This might require +[Conformance Profiles](https://github.com/kubernetes/enhancements/tree/master/keps/sig-architecture/1618-conformance-profiles) +to be feasible. ## Implementation History diff --git a/keps/sig-auth/2579-psp-replacement/kep.yaml b/keps/sig-auth/2579-psp-replacement/kep.yaml index 479c7af0eba..4a3062d64ba 100644 --- a/keps/sig-auth/2579-psp-replacement/kep.yaml +++ b/keps/sig-auth/2579-psp-replacement/kep.yaml @@ -29,18 +29,18 @@ see-also: replaces: [] # The target maturity stage in the current dev cycle for this KEP. -stage: beta +stage: stable # The most recent milestone for which work toward delivery of this KEP has been # done. This can be the current (upcoming) milestone, if it is being actively # worked on. -latest-milestone: "v1.23" +latest-milestone: "v1.25" # The milestone at which this feature was, or is targeted to be, at each stage. milestone: alpha: "v1.22" beta: "v1.23" - stable: TBD + stable: "v1.25" # The following PRR answers are required at alpha release # List the feature gate name and the components for which it must be enabled @@ -48,7 +48,7 @@ feature-gates: - name: PodSecurity components: - kube-apiserver (PodSecurity admission plugin) -disable-supported: true +disable-supported: false # The following PRR answers are required at beta release metrics: From 32dc2c419087b492ab8b7a8de07b8c17500550ee Mon Sep 17 00:00:00 2001 From: Tim Allclair Date: Thu, 12 May 2022 09:48:18 -0700 Subject: [PATCH 2/5] Add test coverage sections --- keps/sig-auth/2579-psp-replacement/README.md | 76 +++++++++++--------- 1 file changed, 41 insertions(+), 35 deletions(-) diff --git a/keps/sig-auth/2579-psp-replacement/README.md b/keps/sig-auth/2579-psp-replacement/README.md index 0ffb85841ba..3cf41ab332f 100644 --- a/keps/sig-auth/2579-psp-replacement/README.md +++ b/keps/sig-auth/2579-psp-replacement/README.md @@ -24,6 +24,10 @@ - [Windows Support](#windows-support) - [Flexible Extension Support](#flexible-extension-support) - [Test Plan](#test-plan) + - [Prerequisite testing updates](#prerequisite-testing-updates) + - [Unit tests](#unit-tests) + - [Integration tests](#integration-tests) + - [e2e tests](#e2e-tests) - [Monitoring](#monitoring) - [Audit Annotations](#audit-annotations) - [PodSecurityPolicy Migration](#podsecuritypolicy-migration) @@ -553,41 +557,43 @@ publish the following tools: ### Test Plan -The admission controller can safely be enabled as a no-op with the default-defaults, i.e. everything -is privileged. This will let us run the admission controller in our standard E2E test jobs, by -relabeling specific test namespaces. - -**E2E Tests:** The following tests should be added: - -1. Enforce mode tests: - - Test all profile levels - - Test profile version support -2. Warning mode tests: - - Profile levels & version support -3. Namespace policy relabeling - - Ensure labeling completes even when there are warnings - - Test warning on violating pods - - Test dry-run mode - -Additionally, we should add tests to the upgrade test suite to ensure that version skew is properly -handled: - -- A minimally specified pod (just a container image) should always be allowed by the baseline - policy. -- A privileged pod should never be allowed by baseline or restricted -- A Fully specified pod within the bounds of baseline should be allowed by baseline, and rejected by - restricted. -- A minimally specified restricted pod should be allowed at a pinned version. - -**Integration Tests:** Audit mode tests should be added to integration testing, where we have -existing audit logging tests. - -**Manual Testing Resources:** Pod resources will be provided covering all dimensions of the baseline -& restricted profiles, for validation of 3rd party policy implementations. These have been drafted -by @JimBugwadia: https://github.com/JimBugwadia/pod-security-tests - -**Unit Tests:** Both the library and admission controller implementations will have thorough -coverage of unit tests. +##### Prerequisite testing updates + +None. + +##### Unit tests + +- `k8s.io/pod-security-admission/admission`: `2020-05-12` - `80.7% of statements` +- `k8s.io/pod-security-admission/admission/api`: `2020-05-12` - `1.4% of statements` (mostly boilerplate & generated code) +- `k8s.io/pod-security-admission/admission/api/load`: `2020-05-12` - `88.5% of statements` +- `k8s.io/pod-security-admission/admission/api/scheme`: `2020-05-12` - `100.0% of statements` +- `k8s.io/pod-security-admission/admission/api/v1alpha1`: `2020-05-12` - `1.7% of statements` (generated API) +- `k8s.io/pod-security-admission/admission/api/v1beta1`: `2020-05-12` - `1.7% of statements` (generated API) +- `k8s.io/pod-security-admission/admission/api/validation`: `2020-05-12` - `100.0% of statements` +- `k8s.io/pod-security-admission/api`: `2020-05-12` - `9.3% of statements` **room for improvement** +- `k8s.io/pod-security-admission/cmd/webhook`: `2020-05-12` - `no unit tests` (mostly server setup, covered by integration) +- `k8s.io/pod-security-admission/cmd/webhook/server`: `2020-05-12` - `no unit tests` (mostly server setup, covered by integration) +- `k8s.io/pod-security-admission/cmd/webhook/server/options`: `2020-05-12` - `no unit tests` (mostly server setup, covered by integration) +- `k8s.io/pod-security-admission/metrics`: `2020-05-12` - `93.8% of statements` +- `k8s.io/pod-security-admission/policy`: `2020-05-12` - `88.3% of statements` +- `k8s.io/pod-security-admission/test`: `2020-05-12` - `73.7% of statements` + +##### Integration tests + +`k8s.io/kubernetes/test/integration/auth/podsecurity_test.go` +https://storage.googleapis.com/k8s-triage/index.html?test=TestPodSecurity + +Pod Security admission has very thorough integration test coverage, including: +- Generated test fixtures for failing & passing pods across every type of check, version and level. +- Tests with only GA feature gates enabled, and the default set. +- Tests running as a built-in admission controller & webhook. +- Tests pods run directly & via a controller + +##### e2e tests + +There are no Pod Security specific E2E tests (we rely on integration test coverage instead), but the +Pod Security admission controller is enabled in E2E clusters, and all E2E test namespaces are +labeled with the enforcement label for Pod Security. ### Monitoring From eb1975608d30fe74c13f253014758ded3439b044 Mon Sep 17 00:00:00 2001 From: Tim Allclair Date: Thu, 12 May 2022 09:59:11 -0700 Subject: [PATCH 3/5] Clarify API changes --- keps/sig-auth/2579-psp-replacement/README.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/keps/sig-auth/2579-psp-replacement/README.md b/keps/sig-auth/2579-psp-replacement/README.md index 3cf41ab332f..3a01a2c02ac 100644 --- a/keps/sig-auth/2579-psp-replacement/README.md +++ b/keps/sig-auth/2579-psp-replacement/README.md @@ -759,7 +759,9 @@ Targeting GA in v1.25. - [Add context to failure messages](https://github.com/kubernetes/kubernetes/pull/105314) **API Changes:** -- Add `pod-security.admission.config.k8s.io/v1` with no changes from the `v1beta1` API. +- No changes to namespace label schema +- Add `pod-security.admission.config.k8s.io/v1` (admission configuration, not a REST API) with no + changes from the `v1beta1` API. ### Upgrade / Downgrade Strategy From e2224f8cf440857ad346d647cd5e9ebf5ad3c665 Mon Sep 17 00:00:00 2001 From: Tim Allclair Date: Thu, 12 May 2022 09:59:24 -0700 Subject: [PATCH 4/5] Add prod-readiness approvers file --- keps/prod-readiness/sig-auth/2579.yaml | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/keps/prod-readiness/sig-auth/2579.yaml b/keps/prod-readiness/sig-auth/2579.yaml index a0d79bc79d4..6532b1261d6 100644 --- a/keps/prod-readiness/sig-auth/2579.yaml +++ b/keps/prod-readiness/sig-auth/2579.yaml @@ -1,5 +1,10 @@ +# The KEP must have an approver from the +# "prod-readiness-approvers" group +# of http://git.k8s.io/enhancements/OWNERS_ALIASES kep-number: 2579 alpha: approver: "@deads2k" beta: approver: "@deads2k" +stable: + approver: "@deads2k" From 026a9f6bd5db860632840de9e94076688e657365 Mon Sep 17 00:00:00 2001 From: Tim Allclair Date: Thu, 12 May 2022 15:41:45 -0700 Subject: [PATCH 5/5] Fix dates --- keps/sig-auth/2579-psp-replacement/README.md | 28 ++++++++++---------- 1 file changed, 14 insertions(+), 14 deletions(-) diff --git a/keps/sig-auth/2579-psp-replacement/README.md b/keps/sig-auth/2579-psp-replacement/README.md index 3a01a2c02ac..3f28736da0c 100644 --- a/keps/sig-auth/2579-psp-replacement/README.md +++ b/keps/sig-auth/2579-psp-replacement/README.md @@ -563,20 +563,20 @@ None. ##### Unit tests -- `k8s.io/pod-security-admission/admission`: `2020-05-12` - `80.7% of statements` -- `k8s.io/pod-security-admission/admission/api`: `2020-05-12` - `1.4% of statements` (mostly boilerplate & generated code) -- `k8s.io/pod-security-admission/admission/api/load`: `2020-05-12` - `88.5% of statements` -- `k8s.io/pod-security-admission/admission/api/scheme`: `2020-05-12` - `100.0% of statements` -- `k8s.io/pod-security-admission/admission/api/v1alpha1`: `2020-05-12` - `1.7% of statements` (generated API) -- `k8s.io/pod-security-admission/admission/api/v1beta1`: `2020-05-12` - `1.7% of statements` (generated API) -- `k8s.io/pod-security-admission/admission/api/validation`: `2020-05-12` - `100.0% of statements` -- `k8s.io/pod-security-admission/api`: `2020-05-12` - `9.3% of statements` **room for improvement** -- `k8s.io/pod-security-admission/cmd/webhook`: `2020-05-12` - `no unit tests` (mostly server setup, covered by integration) -- `k8s.io/pod-security-admission/cmd/webhook/server`: `2020-05-12` - `no unit tests` (mostly server setup, covered by integration) -- `k8s.io/pod-security-admission/cmd/webhook/server/options`: `2020-05-12` - `no unit tests` (mostly server setup, covered by integration) -- `k8s.io/pod-security-admission/metrics`: `2020-05-12` - `93.8% of statements` -- `k8s.io/pod-security-admission/policy`: `2020-05-12` - `88.3% of statements` -- `k8s.io/pod-security-admission/test`: `2020-05-12` - `73.7% of statements` +- `k8s.io/pod-security-admission/admission`: `2022-05-12` - `80.7% of statements` +- `k8s.io/pod-security-admission/admission/api`: `2022-05-12` - `1.4% of statements` (mostly boilerplate & generated code) +- `k8s.io/pod-security-admission/admission/api/load`: `2022-05-12` - `88.5% of statements` +- `k8s.io/pod-security-admission/admission/api/scheme`: `2022-05-12` - `100.0% of statements` +- `k8s.io/pod-security-admission/admission/api/v1alpha1`: `2022-05-12` - `1.7% of statements` (generated API) +- `k8s.io/pod-security-admission/admission/api/v1beta1`: `2022-05-12` - `1.7% of statements` (generated API) +- `k8s.io/pod-security-admission/admission/api/validation`: `2022-05-12` - `100.0% of statements` +- `k8s.io/pod-security-admission/api`: `2022-05-12` - `9.3% of statements` **room for improvement** +- `k8s.io/pod-security-admission/cmd/webhook`: `2022-05-12` - `no unit tests` (mostly server setup, covered by integration) +- `k8s.io/pod-security-admission/cmd/webhook/server`: `2022-05-12` - `no unit tests` (mostly server setup, covered by integration) +- `k8s.io/pod-security-admission/cmd/webhook/server/options`: `2022-05-12` - `no unit tests` (mostly server setup, covered by integration) +- `k8s.io/pod-security-admission/metrics`: `2022-05-12` - `93.8% of statements` +- `k8s.io/pod-security-admission/policy`: `2022-05-12` - `88.3% of statements` +- `k8s.io/pod-security-admission/test`: `2022-05-12` - `73.7% of statements` ##### Integration tests