Skip to content

Commit

Permalink
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Add LegacyServiceAccountTokenCleanUp feature to beta
Browse files Browse the repository at this point in the history
yt2985 committed Nov 15, 2023
1 parent 7899eb0 commit e652da3
Showing 3 changed files with 130 additions and 3 deletions.
107 changes: 107 additions & 0 deletions content/en/docs/reference/access-authn-authz/service-accounts-admin.md
Original file line number Diff line number Diff line change
@@ -140,6 +140,62 @@ using [TokenRequest](/docs/reference/kubernetes-api/authentication-resources/tok
to obtain short-lived API access tokens is recommended instead.
{{< /note >}}

## Auto-generated legacy ServiceAccount token clean up {#auto-generated-legacy-serviceaccount-token-clean-up}

Before version 1.24, Kubernetes automatically generated Secret-based tokens for
ServiceAccounts. To distinguish between automatically generated tokens and
manually created ones, Kubernetes checks for a reference from the
ServiceAccount's secrets field. If the Secret is referenced in the `secrets`
field, it is considered an auto-generated legacy token. Otherwise, it is
considered a manually created legacy token. For example:

```yaml
apiVersion: v1
kind: ServiceAccount
metadata:
name: build-robot
namespace: default
secrets:
- name: build-robot-secret # usually NOT present for a manually generated token
```

Beginning from version 1.29, legacy ServiceAccount tokens that were generated
automatically will be marked as invalid if they remain unused for a certain
period of time (set to default at one year). Tokens that continue to be unused
for this defined period (again, by default, one year) will subsequently be
purged by the control plane.

If users use an invalidated auto-generated token, the token validator will

1. add an audit annotation for the key-value pair
`authentication.k8s.io/legacy-token-invalidated: <secret name>/<namepace>`,
1. increment the `invalid_legacy_auto_token_uses_total` metric count,
1. update the Secret label `kubernetes.io/legacy-token-last-used` with the new
date,
1. return an error indicating that the token has been invalidated.

When receiving this validation error, users can update the Secret to remove the
`kubernetes.io/legacy-token-invalid-since` label to temporarily allow use of
this token.

Here's an example of an auto-generated legacy token that has been marked with the
`kubernetes.io/legacy-token-last-used` and `kubernetes.io/legacy-token-invalid-since`
labels:

```yaml
apiVersion: v1
kind: Secret
metadata:
name: build-robot-secret
namespace: default
labels:
kubernetes.io/legacy-token-last-used: 2022-10-24
kubernetes.io/legacy-token-invalid-since: 2023-10-25
annotations:
kubernetes.io/service-account.name: build-robot
type: kubernetes.io/service-account-token
```

## Control plane details

### ServiceAccount controller
@@ -193,6 +249,51 @@ it does the following when a Pod is created:
1. If the spec of the incoming Pod doesn't already contain any `imagePullSecrets`, then the
admission controller adds `imagePullSecrets`, copying them from the `ServiceAccount`.

### Legacy ServiceAccount token tracking controller

{{< feature-state for_k8s_version="v1.28" state="stable" >}}

This controller generates a ConfigMap called
`kube-system/kube-apiserver-legacy-service-account-token-tracking` in the
`kube-system` namespace. The ConfigMap records the timestamp when legacy service
account tokens began to be monitored by the system.

### Legacy ServiceAccount token cleaner

{{< feature-state for_k8s_version="v1.29" state="beta" >}}

The legacy ServiceAccount token cleaner runs as part of the
`kube-controller-manager` and checks every 24 hours to see if any auto-generated
legacy ServiceAccount token has not been used in a *specified amount of time*.
If so, the cleaner marks those tokens as invalid.

The cleaner works by first checking the ConfigMap created by the control plane
(provided that `LegacyServiceAccountTokenTracking` is enabled). If the current
time is a *specified amount of time* after the date in the ConfigMap, the
cleaner then loops through the list of Secrets in the cluster and evaluates each
Secret that has the type `kubernetes.io/service-account-token`.

If a Secret meets all of the following conditions, the cleaner marks it as
invalid:

- The Secret is auto-generated, meaning that it is bi-directionally referenced
by a ServiceAccount.
- The Secret is not currently mounted by any pods.
- The Secret has not been used in a *specified amount of time* since it was
created or since it was last used.

The cleaner marks a Secret invalid by adding a label called
`kubernetes.io/legacy-token-invalid-since` to the Secret, with the current date
as the value. If an invalid Secret is not used in a *specified amount of time*,
the cleaner will delete it.

{{< note >}}
All the *specified amount of time* above defaults to one year. The cluster
administrator can configure this value through the
`--legacy-service-account-token-clean-up-period` command line argument for the
`kube-controller-manager` component.
{{< /note >}}

### TokenRequest API

{{< feature-state for_k8s_version="v1.22" state="stable" >}}
@@ -300,6 +401,12 @@ token: ...
If you launch a new Pod into the `examplens` namespace, it can use the `myserviceaccount`
service-account-token Secret that you just created.

{{< warning >}}
Do not reference manually created Secrets in the `secrets` field of a
ServiceAccount. Or the manually created Secrets will be cleaned if it is not used for a long
time. Please refer to [auto-generated legacy ServiceAccount token clean up](#auto-generated-legacy-serviceaccount-token-clean-up).
{{< /warning >}}

## Delete/invalidate a ServiceAccount token {#delete-token}

If you know the name of the Secret that contains the token you want to remove:
Original file line number Diff line number Diff line change
@@ -131,7 +131,8 @@ For a reference to old feature gates that are removed, please refer to
| `KubeletPodResourcesGet` | `false` | Alpha | 1.27 | |
| `KubeletTracing` | `false` | Alpha | 1.25 | 1.26 |
| `KubeletTracing` | `true` | Beta | 1.27 | |
| `LegacyServiceAccountTokenCleanUp` | `false` | Alpha | 1.28 | |
| `LegacyServiceAccountTokenCleanUp` | `false` | Alpha | 1.28 | 1.28 |
| `LegacyServiceAccountTokenCleanUp` | `true` | Beta | 1.29 | |
| `LoadBalancerIPMode` | `false` | Alpha | 1.29 | |
| `LocalStorageCapacityIsolationFSQuotaMonitoring` | `false` | Alpha | 1.15 | - |
| `LogarithmicScaleDown` | `false` | Alpha | 1.21 | 1.21 |
@@ -602,9 +603,11 @@ Each feature gate is designed for enabling/disabling a specific feature:
See [Traces for Kubernetes System Components](/docs/concepts/cluster-administration/system-traces) for more details.
- `LegacyServiceAccountTokenNoAutoGeneration`: Stop auto-generation of Secret-based
[service account tokens](/docs/concepts/security/service-accounts/#get-a-token).
- `LegacyServiceAccountTokenCleanUp`: Enable cleaning up Secret-based
- `LegacyServiceAccountTokenCleanUp`: Enable invalidating auto-generated Secret-based
[service account tokens](/docs/concepts/security/service-accounts/#get-a-token)
when they are not used in a specified time (default to be one year).
when they have not been used in a specified time (defaults to one year). Clean up
the auto-generated Secret-based tokens if they have been invalidated for a specified time
(defaults to one year).
- `LegacyServiceAccountTokenTracking`: Track usage of Secret-based
[service account tokens](/docs/concepts/security/service-accounts/#get-a-token).
- `LoadBalancerIPMode`: Allows setting `ipMode` for Services where `type` is set to `LoadBalancer`.
17 changes: 17 additions & 0 deletions content/en/docs/reference/labels-annotations-taints/_index.md
Original file line number Diff line number Diff line change
@@ -1028,6 +1028,23 @@ last saw a request where the client authenticated using the service account toke
If a legacy token was last used before the cluster gained the feature (added in Kubernetes v1.26),
then the label isn't set.

### kubernetes.io/legacy-token-invalid-since

Type: Label

Example: `kubernetes.io/legacy-token-invalid-since: 2023-10-27`

Used on: Secret

The control plane automatically adds this label to auto-generated Secrets that
have the type `kubernetes.io/service-account-token`, provided that you have the
`LegacyServiceAccountTokenCleanUp` [feature gate](/docs/reference/command-line-tools-reference/feature-gates/)
enabled. Kubernetes {{< skew currentVersion >}} enables that behavior by default.
This label marks the Secret-based token as invalid for authentication. The value
of this label records the date (ISO 8601 format, UTC time zone) when the control
plane detects that the auto-generated Secret has not been used for a specified
duration (defaults to one year).

### endpointslice.kubernetes.io/managed-by {#endpointslicekubernetesiomanaged-by}

Type: Label

0 comments on commit e652da3

Please sign in to comment.