deprecate Token Controller

keps/prod-readiness/sig-auth/2799.yaml

@@ -0,0 +1,3 @@
+kep-number: 2799
+alpha:
+  approver: "@deads2k"

keps/sig-auth/2799-token-controller-deprecation/README.md

@@ -0,0 +1,395 @@
+# KEP-2799: Reduction of Secret-based Service Account Tokens
+
+<!-- toc -->
+- [Release Signoff Checklist](#release-signoff-checklist)
+- [Summary](#summary)
+- [Motivation](#motivation)
+  - [Goals](#goals)
+  - [Non-Goals](#non-goals)
+- [Proposal](#proposal)
+  - [Notes/Constraints/Caveats](#notesconstraintscaveats)
+  - [Risks and Mitigations](#risks-and-mitigations)
+- [Design Details](#design-details)
+  - [Token Controller](#token-controller)
+  - [Test Plan](#test-plan)
+  - [Graduation Criteria](#graduation-criteria)
+    - [LegacyServiceAccountTokenNoAutoGeneration](#legacyserviceaccounttokennoautogeneration)
+    - [Beta -&gt; GA Graduation](#beta---ga-graduation)
+    - [Alpha -&gt; Beta Graduation](#alpha---beta-graduation)
+    - [LegacyServiceAccountTokenCleanUp](#legacyserviceaccounttokencleanup)
+    - [Beta -&gt; GA Graduation](#beta---ga-graduation-1)
+    - [Alpha -&gt; Beta Graduation](#alpha---beta-graduation-1)
+  - [Upgrade / Downgrade Strategy](#upgrade--downgrade-strategy)
+  - [Version Skew Strategy](#version-skew-strategy)
+- [Production Readiness Review Questionnaire](#production-readiness-review-questionnaire)
+  - [Feature Enablement and Rollback](#feature-enablement-and-rollback)
+  - [Rollout, Upgrade and Rollback Planning](#rollout-upgrade-and-rollback-planning)
+  - [Monitoring Requirements](#monitoring-requirements)
+  - [Dependencies](#dependencies)
+  - [Scalability](#scalability)
+  - [Troubleshooting](#troubleshooting)
+- [Implementation History](#implementation-history)
+- [Drawbacks](#drawbacks)
+- [Alternatives](#alternatives)
+- [Infrastructure Needed (Optional)](#infrastructure-needed-optional)
+<!-- /toc -->
+
+## Release Signoff Checklist
+
+Items marked with (R) are required _prior to targeting to a milestone /
+release_.
+
+- [x] (R) Enhancement issue in release milestone, which links to KEP dir in [kubernetes/enhancements] (not the initial KEP PR)
+- [x] (R) KEP approvers have approved the KEP status as `implementable`
+- [x] (R) Design details are appropriately documented
+- [x] (R) Test plan is in place, giving consideration to SIG Architecture and SIG Testing input (including test refactors)
+- [x] (R) Graduation criteria is in place
+- [x] (R) Production readiness review completed
+- [x] (R) Production readiness review approved
+- [ ] "Implementation History" section is up-to-date for milestone
+- [ ] User-facing documentation has been created in [kubernetes/website], for publication to [kubernetes.io]
+- [ ] Supporting documentation—e.g., additional design documents, links to mailing list discussions/SIG meetings, relevant PRs/issues, release notes
+
+[kubernetes.io]: https://kubernetes.io/
+[kubernetes/enhancements]: https://git.k8s.io/enhancements
+[kubernetes/kubernetes]: https://git.k8s.io/kubernetes
+[kubernetes/website]: https://git.k8s.io/website
+
+## Summary
+
+This KEP proposes a namespace-scoped and service-account-scoped binary label to
+allow users to opt in/out the provision of secret-based service account tokens
+in Token Controller. In addition, it sketches out the phases to deprecate Token
+Controller.
+
+## Motivation
+
+As BoundServiceAccountTokenVolume is GA in 1.22, pods’ service account tokens
+would be obtained via TokenRequest API and stored as projected volume. This
+change obviates the need for auto-generation of secret-based service account
+tokens in Token Controller. The secret-based tokens are [not secure by design](https://github.com/kubernetes/enhancements/tree/master/keps/sig-auth/1205-bound-service-account-tokens#background)
+and the token controller is [fragile by design in some cases](https://github.com/kubernetes/kubernetes/issues/98474)
+where it is unable to handle the churns between secrets and service account
+controller loops.
+
+### Goals
+
+- No auto-generation of secret-based service account token.
+- Removal of secret-based service account token that are referenced in service
+  accounts.
+
+### Non-Goals
+
+## Proposal
+
+- In release 1.23, change the service account control loop in Token Controller
+  to not create secret for the service account. At the same time, warn usage of
+  auto-created secret-based service account tokens and enrouage users to use
+  TokenRequest API or manually-created secret-based service account tokens.
+- In release 1.25, purge auto-generated secret-based service account tokens.
+
+### Notes/Constraints/Caveats
+
+- For clusters in upgrade path, users should not upgrade to release 1.25+ unless
+  they are certain of no active usage of secret-based tokens. To consult that
+  information, metric `serviceaccount_stale_tokens_total` or audit annotation
+  `authentication.k8s.io/stale-token` could be used.
+- A warning mechanism should be implemented to push users to migrate and it
+  will exist for at least one year before release N+Y.
+
+### Risks and Mitigations
+
+- In release 1.23, there is only risk in implementation which would be mitigated by
+  tests.
+- In release 1.25, usage of secret-based token that are referenced in service
+  accounts might stop working. To mitigate, users can use TokenRequest API or
+  manually created tokens.
+
+## Design Details
+
+### Token Controller
+
+1.  Token Controller stops to create secret for service accounts.
+2.  Token Controller starts to remove secrets that are not referenced in service
+    accounts.
+
+### Test Plan
+
+- Unit tests
+- E2E tests
+- Upgrade tests
+
+### Graduation Criteria
+
+#### LegacyServiceAccountTokenNoAutoGeneration
+
+| Alpha | Beta | GA   |
+| ----- | ---- | ---- |
+| 1.23  | 1.24 | 1.25 |
+
+This feature gate controls the behavior of stopping auto-generation of
+secret-based service account tokens.
+
+#### Beta -> GA Graduation
+
+- [ ] Approved by PRR and scalability
+- [ ] Any known bugs fixed
+- [ ] Tests passing
+
+#### Alpha -> Beta Graduation
+
+- [ ] Approved by PRR and scalability
+- [ ] Any known bugs fixed
+- [ ] Tests passing
+
+#### LegacyServiceAccountTokenCleanUp
+
+| Alpha | Beta | GA   |
+| ----- | ---- | ---- |
+| 1.24  | 1.25 | 1.26 |
+
+This feature gate includes the removal of secret-based service account tokens
+that are referenced in service accounts.
+
+#### Beta -> GA Graduation
+
+- [ ] In use by multiple distributions
+- [ ] Approved by PRR and scalability
+- [ ] Any known bugs fixed
+- [ ] Tests passing
+
+#### Alpha -> Beta Graduation
+
+- [ ] In use by multiple distributions
+- [ ] Approved by PRR and scalability
+- [ ] Any known bugs fixed
+- [ ] Tests passing
+
+### Upgrade / Downgrade Strategy
+
+TODO
+
+### Version Skew Strategy
+
+The only touches control plane, so version skew strategy is not applicable.
+
+## Production Readiness Review Questionnaire
+
+### Feature Enablement and Rollback
+
+###### How can this feature be enabled / disabled in a live cluster?
+
+- [x] Feature gate (also fill in values in `kep.yaml`)
+  - Feature gate name: LegacyServiceAccountTokenNoAutoGeneration
+  - Components depending on the feature gate: kube-controller-manager
+  - Feature gate name: LegacyServiceAccountTokenCleanUp:
+  - Components depending on the feature gate: kube-controller-manager
+
+###### Does enabling the feature change any default behavior?
+
+- LegacyServiceAccountTokenNoAutoGeneration: no.
+- LegacyServiceAccountTokenCleanUp: auto-generated legacy tokens will be removed.
+
+###### Can the feature be disabled once it has been enabled (i.e. can we roll back the enablement)?
+
+yes
+
+###### What happens if we reenable the feature if it was previously rolled back?
+
+the same as enable the feature.
+
+###### Are there any tests for feature enablement/disablement?
+
+no as there is no API changes which could be covered by unit tests.
+
+### Rollout, Upgrade and Rollback Planning
+
+<!--
+This section must be completed when targeting beta to a release.
+-->
+
+###### How can a rollout fail? Can it impact already running workloads?
+
+TODO in beta
+
+###### What specific metrics should inform a rollback?
+
+`serviceaccount_stale_tokens_total`: cumulative stale projected service
+account tokens used.
+
+###### Were upgrade and rollback tested? Was the upgrade->downgrade->upgrade path tested?
+
+TODO in beta
+
+###### Is the rollout accompanied by any deprecations and/or removals of features, APIs, fields of API types, flags, etc.?
+
+no
+
+### Monitoring Requirements
+
+<!--
+This section must be completed when targeting beta to a release.
+-->
+
+###### How can an operator determine if the feature is in use by workloads?
+
+<!--
+Ideally, this should be a metric. Operations against the Kubernetes API (e.g.,
+checking if there are objects with field X set) may be a last resort. Avoid
+logs or events for this purpose.
+-->
+
+###### What are the SLIs (Service Level Indicators) an operator can use to determine the health of the service?
+
+<!--
+Pick one more of these and delete the rest.
+-->
+
+- [ ] Metrics
+  - Metric name:
+  - [Optional] Aggregation method:
+  - Components exposing the metric:
+- [ ] Other (treat as last resort)
+  - Details:
+
+###### What are the reasonable SLOs (Service Level Objectives) for the above SLIs?
+
+<!--
+At a high level, this usually will be in the form of "high percentile of SLI
+per day <= X". It's impossible to provide comprehensive guidance, but at the very
+high level (needs more precise definitions) those may be things like:
+  - per-day percentage of API calls finishing with 5XX errors <= 1%
+  - 99% percentile over day of absolute value from (job creation time minus expected
+    job creation time) for cron job <= 10%
+  - 99,9% of /health requests per day finish with 200 code
+-->
+
+###### Are there any missing metrics that would be useful to have to improve observability of this feature?
+
+<!--
+Describe the metrics themselves and the reasons why they weren't added (e.g., cost,
+implementation difficulties, etc.).
+-->
+
+### Dependencies
+
+<!--
+This section must be completed when targeting beta to a release.
+-->
+
+###### Does this feature depend on any specific services running in the cluster?
+
+<!--
+Think about both cluster-level services (e.g. metrics-server) as well
+as node-level agents (e.g. specific version of CRI). Focus on external or
+optional services that are needed. For example, if this feature depends on
+a cloud provider API, or upon an external software-defined storage or network
+control plane.
+
+For each of these, fill in the following—thinking about running existing user workloads
+and creating new ones, as well as about cluster-level services (e.g. DNS):
+  - [Dependency name]
+    - Usage description:
+      - Impact of its outage on the feature:
+      - Impact of its degraded performance or high-error rates on the feature:
+-->
+
+### Scalability
+
+<!--
+For alpha, this section is encouraged: reviewers should consider these questions
+and attempt to answer them.
+
+For beta, this section is required: reviewers must answer these questions.
+
+For GA, this section is required: approvers should be able to confirm the
+previous answers based on experience in the field.
+-->
+
+###### Will enabling / using this feature result in any new API calls?
+
+no.
+
+###### Will enabling / using this feature result in introducing new API types?
+
+no.
+
+###### Will enabling / using this feature result in any new calls to the cloud provider?
+
+no.
+
+###### Will enabling / using this feature result in increasing size or count of the existing API objects?
+
+no.
+
+###### Will enabling / using this feature result in increasing time taken by any operations covered by existing SLIs/SLOs?
+
+no.
+
+###### Will enabling / using this feature result in non-negligible increase of resource usage (CPU, RAM, disk, IO, ...) in any components?
+
+no.
+
+### Troubleshooting
+
+<!--
+This section must be completed when targeting beta to a release.
+
+The Troubleshooting section currently serves the `Playbook` role. We may consider
+splitting it into a dedicated `Playbook` document (potentially with some monitoring
+details). For now, we leave it here.
+-->
+
+###### How does this feature react if the API server and/or etcd is unavailable?
+
+###### What are other known failure modes?
+
+<!--
+For each of them, fill in the following information by copying the below template:
+  - [Failure mode brief description]
+    - Detection: How can it be detected via metrics? Stated another way:
+      how can an operator troubleshoot without logging into a master or worker node?
+    - Mitigations: What can be done to stop the bleeding, especially for already
+      running user workloads?
+    - Diagnostics: What are the useful log messages and their required logging
+      levels that could help debug the issue?
+      Not required until feature graduated to beta.
+    - Testing: Are there any tests for failure mode? If not, describe why.
+-->
+
+###### What steps should be taken if SLOs are not being met to determine the problem?
+
+## Implementation History
+
+<!--
+Major milestones in the lifecycle of a KEP should be tracked in this section.
+Major milestones might include:
+- the `Summary` and `Motivation` sections being merged, signaling SIG acceptance
+- the `Proposal` section being merged, signaling agreement on a proposed design
+- the date implementation started
+- the first Kubernetes release where an initial version of the KEP was available
+- the version of Kubernetes where the KEP graduated to general availability
+- when the KEP was retired or superseded
+-->
+
+## Drawbacks
+
+<!--
+Why should this KEP _not_ be implemented?
+-->
+
+## Alternatives
+
+<!--
+What other approaches did you consider, and why did you rule them out? These do
+not need to be as detailed as the proposal, but should include enough
+information to express the idea and why it was not acceptable.
+-->
+
+## Infrastructure Needed (Optional)
+
+<!--
+Use this section if you need things from the project/SIG. Examples include a
+new subproject, repos requested, or GitHub details. Listing these here allows a
+SIG to get the process for these resources started right away.
+-->