Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Mutable #40060

Closed
wants to merge 20 commits into from
Closed

Mutable #40060

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
20 commits
Select commit Hold shift + click to select a range
8d49910
Tracking commit for v1.27 docs
krol3 Dec 17, 2022
ca16a71
Move volume expansion feature gates to removed feature gates list
Dec 19, 2022
0201ddd
Update MinDomainsInPodTopologySpread feature gate default value.
Dec 19, 2022
8618771
Merge pull request #38545 from mengjiao-liu/update-volume-expansion-f…
k8s-ci-robot Dec 19, 2022
8da3787
Merge pull request #38549 from mengjiao-liu/update-MinDomainsInPodTop…
k8s-ci-robot Dec 19, 2022
fed60f3
Rename 'PreemptionByKubeScheduler' to 'PreemptionByScheduler'
Huang-Wei Jan 9, 2023
c1bf88e
Update seccomp docs for v1.27
saschagrunert Jan 18, 2023
f2d1a35
Merge pull request #38849 from Huang-Wei/1.27/PreemptionByScheduler
k8s-ci-robot Jan 27, 2023
23db46b
resolved merge conflict from branch sync from main into dev-1.27
reylejano Feb 22, 2023
aabfcc6
Merge pull request #38987 from saschagrunert/seccomp-dev-1.27
k8s-ci-robot Feb 23, 2023
3a7fab6
Merge pull request #39630 from reylejano/merged-main-dev-1.27
k8s-ci-robot Feb 24, 2023
4480cd3
removed ControllerManagerLeaderMigration in v1.27
pacoxu Mar 1, 2023
a6b1df5
Merge pull request #39740 from pacoxu/ControllerManagerLeaderMigratio…
k8s-ci-robot Mar 3, 2023
fc01961
Doc for Beta feature PodSchedulingReadiness
Huang-Wei Mar 6, 2023
987edf7
Merge remote-tracking branch 'upstream/main' into dev-1.27
mickeyboxell Mar 7, 2023
d842569
doc: graduate matchLabelKeys in podTopologySpread to beta
denkensk Mar 7, 2023
5a97479
Merge pull request #39826 from denkensk/doc-matchLabels-beta-1.27
k8s-ci-robot Mar 7, 2023
d3979d5
Merge pull request #39773 from Huang-Wei/1.27/PodSchedulingReadiness-…
k8s-ci-robot Mar 9, 2023
bd456cf
Merge pull request #39824 from mickeyboxell/merged-main-dev-1.27
k8s-ci-robot Mar 10, 2023
c2b4f84
Graduate JobMutableNodeSchedulingDirectives to GA
ahg-g Mar 16, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -55,7 +55,7 @@ kubectl get pod test-pod -o jsonpath='{.spec.schedulingGates}'
The output is:

```none
[{"name":"foo"},{"name":"bar"}]
[{"name":"example.com/foo"},{"name":"example.com/bar"}]
```

To inform scheduler this Pod is ready for scheduling, you can remove its `schedulingGates` entirely
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -64,7 +64,7 @@ spec:
topologyKey: <string>
whenUnsatisfiable: <string>
labelSelector: <object>
matchLabelKeys: <list> # optional; alpha since v1.25
matchLabelKeys: <list> # optional; beta since v1.27
nodeAffinityPolicy: [Honor|Ignore] # optional; beta since v1.26
nodeTaintsPolicy: [Honor|Ignore] # optional; beta since v1.26
### other Pod fields go here
Expand Down Expand Up @@ -144,9 +144,8 @@ your cluster. Those fields are:
```

{{< note >}}
The `matchLabelKeys` field is an alpha field added in 1.25. You have to enable the
`MatchLabelKeysInPodTopologySpread` [feature gate](/docs/reference/command-line-tools-reference/feature-gates/)
in order to use it.
The `matchLabelKeys` field is a beta-level field and enabled by default in 1.27. You can disable it by disabling the
`MatchLabelKeysInPodTopologySpread` [feature gate](/docs/reference/command-line-tools-reference/feature-gates/).
{{< /note >}}

- **nodeAffinityPolicy** indicates how we will treat Pod's nodeAffinity/nodeSelector
Expand Down
11 changes: 2 additions & 9 deletions content/en/docs/concepts/workloads/controllers/job.md
Original file line number Diff line number Diff line change
Expand Up @@ -631,14 +631,7 @@ as soon as the Job was resumed.
### Mutable Scheduling Directives
{{< feature-state for_k8s_version="v1.23" state="beta" >}}
{{< note >}}
In order to use this behavior, you must enable the `JobMutableNodeSchedulingDirectives`
[feature gate](/docs/reference/command-line-tools-reference/feature-gates/)
on the [API server](/docs/reference/command-line-tools-reference/kube-apiserver/).
It is enabled by default.
{{< /note >}}
{{< feature-state for_k8s_version="v1.27" state="stable" >}}
In most cases a parallel job will want the pods to run with constraints,
like all in the same zone, or all either on GPU model x or y but not a mix of both.
Expand All @@ -653,7 +646,7 @@ pod-to-node assignment to kube-scheduler. This is allowed only for suspended Job
been unsuspended before.
The fields in a Job's pod template that can be updated are node affinity, node selector,
tolerations, labels and annotations.
tolerations, labels, annotations and scheduling gates.
### Specifying your own Pod selector
Expand Down
2 changes: 1 addition & 1 deletion content/en/docs/concepts/workloads/pods/disruptions.md
Original file line number Diff line number Diff line change
Expand Up @@ -247,7 +247,7 @@ that the Pod is about to be deleted due to a {{<glossary_tooltip term_id="disrup
The `reason` field of the condition additionally
indicates one of the following reasons for the Pod termination:

`PreemptionByKubeScheduler`
`PreemptionByScheduler`
: Pod is due to be {{<glossary_tooltip term_id="preemption" text="preempted">}} by a scheduler in order to accommodate a new Pod with a higher priority. For more information, see [Pod priority preemption](/docs/concepts/scheduling-eviction/pod-priority-preemption/).

`DeletionByTaintManager`
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -93,6 +93,9 @@ In the following table:
| `CronJobControllerV2` | `true` | GA | 1.22 | 1.23 |
| `CSRDuration` | `true` | Beta | 1.22 | 1.23 |
| `CSRDuration` | `true` | GA | 1.24 | 1.25 |
| `ControllerManagerLeaderMigration` | `false` | Alpha | 1.21 | 1.21 |
| `ControllerManagerLeaderMigration` | `true` | Beta | 1.22 | 1.23 |
| `ControllerManagerLeaderMigration` | `true` | GA | 1.24 | 1.26 |
| `CustomPodDNS` | `false` | Alpha | 1.9 | 1.9 |
| `CustomPodDNS` | `true` | Beta| 1.10 | 1.13 |
| `CustomPodDNS` | `true` | GA | 1.14 | 1.16 |
Expand Down Expand Up @@ -138,6 +141,15 @@ In the following table:
| `EvenPodsSpread` | `false` | Alpha | 1.16 | 1.17 |
| `EvenPodsSpread` | `true` | Beta | 1.18 | 1.18 |
| `EvenPodsSpread` | `true` | GA | 1.19 | 1.21 |
| `ExpandCSIVolumes` | `false` | Alpha | 1.14 | 1.15 |
| `ExpandCSIVolumes` | `true` | Beta | 1.16 | 1.23 |
| `ExpandCSIVolumes` | `true` | GA | 1.24 | 1.27 |
| `ExpandInUsePersistentVolumes` | `false` | Alpha | 1.11 | 1.14 |
| `ExpandInUsePersistentVolumes` | `true` | Beta | 1.15 | 1.23 |
| `ExpandInUsePersistentVolumes` | `true` | GA | 1.24 | 1.27 |
| `ExpandPersistentVolumes` | `false` | Alpha | 1.8 | 1.10 |
| `ExpandPersistentVolumes` | `true` | Beta | 1.11 | 1.23 |
| `ExpandPersistentVolumes` | `true` | GA | 1.24 | 1.27 |
| `ExperimentalCriticalPodAnnotation` | `false` | Alpha | 1.5 | 1.12 |
| `ExperimentalCriticalPodAnnotation` | `false` | Deprecated | 1.13 | 1.16 |
| `ExternalPolicyForExternalIP` | `true` | GA | 1.18 | 1.22 |
Expand Down Expand Up @@ -468,6 +480,13 @@ In the following table:
{{< glossary_tooltip text="CronJob" term_id="cronjob" >}} controller. Otherwise,
version 1 of the same controller is selected.

- `ControllerManagerLeaderMigration`: Enables Leader Migration for
[kube-controller-manager](/docs/tasks/administer-cluster/controller-manager-leader-migration/#initial-leader-migration-configuration) and
[cloud-controller-manager](/docs/tasks/administer-cluster/controller-manager-leader-migration/#deploy-cloud-controller-manager)
which allows a cluster operator to live migrate
controllers from the kube-controller-manager into an external controller-manager
(e.g. the cloud-controller-manager) in an HA cluster without downtime.

- `CustomPodDNS`: Enable customizing the DNS settings for a Pod using its `dnsConfig` property.
Check [Pod's DNS Config](/docs/concepts/services-networking/dns-pod-service/#pods-dns-config)
for more details.
Expand Down Expand Up @@ -520,6 +539,14 @@ In the following table:
- `EvenPodsSpread`: Enable pods to be scheduled evenly across topology domains. See
[Pod Topology Spread Constraints](/docs/concepts/scheduling-eviction/topology-spread-constraints/).

- `ExpandCSIVolumes`: Enable the expanding of CSI volumes.

- `ExpandInUsePersistentVolumes`: Enable expanding in-use PVCs. See
[Resizing an in-use PersistentVolumeClaim](/docs/concepts/storage/persistent-volumes/#resizing-an-in-use-persistentvolumeclaim).

- `ExpandPersistentVolumes`: Enable the expanding of persistent volumes. See
[Expanding Persistent Volumes Claims](/docs/concepts/storage/persistent-volumes/#expanding-persistent-volumes-claims).

- `ExperimentalCriticalPodAnnotation`: Enable annotating specific pods as *critical*
so that their [scheduling is guaranteed](/docs/tasks/administer-cluster/guaranteed-scheduling-critical-addon-pods/).
This feature is deprecated by Pod Priority and Preemption as of v1.13.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -116,7 +116,6 @@ For a reference to old feature gates that are removed, please refer to
| `InTreePluginPortworxUnregister` | `false` | Alpha | 1.23 | |
| `InTreePluginRBDUnregister` | `false` | Alpha | 1.23 | |
| `InTreePluginvSphereUnregister` | `false` | Alpha | 1.21 | |
| `JobMutableNodeSchedulingDirectives` | `true` | Beta | 1.23 | |
| `JobPodFailurePolicy` | `false` | Alpha | 1.25 | 1.25 |
| `JobPodFailurePolicy` | `true` | Beta | 1.26 | |
| `JobReadyPods` | `false` | Alpha | 1.23 | 1.23 |
Expand All @@ -134,12 +133,17 @@ For a reference to old feature gates that are removed, please refer to
| `LogarithmicScaleDown` | `true` | Beta | 1.22 | |
| `LoggingAlphaOptions` | `false` | Alpha | 1.24 | - |
| `LoggingBetaOptions` | `true` | Beta | 1.24 | - |
| `MatchLabelKeysInPodTopologySpread` | `false` | Alpha | 1.25 | |
| `MatchLabelKeysInPodTopologySpread` | `false` | Alpha | 1.25 | 1.26 |
| `MatchLabelKeysInPodTopologySpread` | `true` | Beta | 1.27 | - |
| `MaxUnavailableStatefulSet` | `false` | Alpha | 1.24 | |
| `MemoryManager` | `false` | Alpha | 1.21 | 1.21 |
| `MemoryManager` | `true` | Beta | 1.22 | |
| `MemoryQoS` | `false` | Alpha | 1.22 | |
| `MinDomainsInPodTopologySpread` | `false` | Alpha | 1.24 | 1.24 |
| `MinDomainsInPodTopologySpread` | `false` | Beta | 1.25 | 1.26 |
| `MinDomainsInPodTopologySpread` | `true` | Beta | 1.27 | |
| `MixedProtocolLBService` | `false` | Alpha | 1.20 | 1.23 |
| `MixedProtocolLBService` | `true` | Beta | 1.24 | |
| `MinDomainsInPodTopologySpread` | `false` | Beta | 1.25 | |
| `MinimizeIPTablesRestore` | `false` | Alpha | 1.26 | - |
| `MultiCIDRRangeAllocator` | `false` | Alpha | 1.25 | |
Expand All @@ -160,7 +164,8 @@ For a reference to old feature gates that are removed, please refer to
| `PodDisruptionConditions` | `false` | Alpha | 1.25 | 1.25 |
| `PodDisruptionConditions` | `true` | Beta | 1.26 | |
| `PodHasNetworkCondition` | `false` | Alpha | 1.25 | |
| `PodSchedulingReadiness` | `false` | Alpha | 1.26 | |
| `PodSchedulingReadiness` | `false` | Alpha | 1.26 | 1.26 |
| `PodSchedulingReadiness` | `true` | Beta | 1.27 | |
| `ProbeTerminationGracePeriod` | `false` | Alpha | 1.21 | 1.21 |
| `ProbeTerminationGracePeriod` | `false` | Beta | 1.22 | 1.24 |
| `ProbeTerminationGracePeriod` | `true` | Beta | 1.25 | |
Expand Down Expand Up @@ -247,9 +252,6 @@ For a reference to old feature gates that are removed, please refer to
| `CSIStorageCapacity` | `true` | Beta | 1.21 | 1.23 |
| `CSIStorageCapacity` | `true` | GA | 1.24 | - |
| `ConsistentHTTPGetHandlers` | `true` | GA | 1.25 | - |
| `ControllerManagerLeaderMigration` | `false` | Alpha | 1.21 | 1.21 |
| `ControllerManagerLeaderMigration` | `true` | Beta | 1.22 | 1.23 |
| `ControllerManagerLeaderMigration` | `true` | GA | 1.24 | - |
| `DaemonSetUpdateSurge` | `false` | Alpha | 1.21 | 1.21 |
| `DaemonSetUpdateSurge` | `true` | Beta | 1.22 | 1.24 |
| `DaemonSetUpdateSurge` | `true` | GA | 1.25 | - |
Expand All @@ -275,18 +277,11 @@ For a reference to old feature gates that are removed, please refer to
| `EphemeralContainers` | `true` | Beta | 1.23 | 1.24 |
| `EphemeralContainers` | `true` | GA | 1.25 | - |
| `ExecProbeTimeout` | `true` | GA | 1.20 | - |
| `ExpandCSIVolumes` | `false` | Alpha | 1.14 | 1.15 |
| `ExpandCSIVolumes` | `true` | Beta | 1.16 | 1.23 |
| `ExpandCSIVolumes` | `true` | GA | 1.24 | - |
| `ExpandInUsePersistentVolumes` | `false` | Alpha | 1.11 | 1.14 |
| `ExpandInUsePersistentVolumes` | `true` | Beta | 1.15 | 1.23 |
| `ExpandInUsePersistentVolumes` | `true` | GA | 1.24 | - |
| `ExpandPersistentVolumes` | `false` | Alpha | 1.8 | 1.10 |
| `ExpandPersistentVolumes` | `true` | Beta | 1.11 | 1.23 |
| `ExpandPersistentVolumes` | `true` | GA | 1.24 |- |
| `IdentifyPodOS` | `false` | Alpha | 1.23 | 1.23 |
| `IdentifyPodOS` | `true` | Beta | 1.24 | 1.24 |
| `IdentifyPodOS` | `true` | GA | 1.25 | - |
| `JobMutableNodeSchedulingDirectives` | `true` | Beta | 1.23 | |
| `JobMutableNodeSchedulingDirectives` | `true` | GA | 1.27 | |
| `JobTrackingWithFinalizers` | `false` | Alpha | 1.22 | 1.22 |
| `JobTrackingWithFinalizers` | `false` | Beta | 1.23 | 1.24 |
| `JobTrackingWithFinalizers` | `true` | Beta | 1.25 | 1.25 |
Expand Down Expand Up @@ -395,12 +390,6 @@ Each feature gate is designed for enabling/disabling a specific feature:
See [AppArmor Tutorial](/docs/tutorials/security/apparmor/) for more details.
- `ContainerCheckpoint`: Enables the kubelet `checkpoint` API.
See [Kubelet Checkpoint API](/docs/reference/node/kubelet-checkpoint-api/) for more details.
- `ControllerManagerLeaderMigration`: Enables Leader Migration for
[kube-controller-manager](/docs/tasks/administer-cluster/controller-manager-leader-migration/#initial-leader-migration-configuration) and
[cloud-controller-manager](/docs/tasks/administer-cluster/controller-manager-leader-migration/#deploy-cloud-controller-manager)
which allows a cluster operator to live migrate
controllers from the kube-controller-manager into an external controller-manager
(e.g. the cloud-controller-manager) in an HA cluster without downtime.
- `CPUManager`: Enable container level CPU affinity support, see
[CPU Management Policies](/docs/tasks/administer-cluster/cpu-management-policies/).
- `CPUManagerPolicyAlphaOptions`: This allows fine-tuning of CPUManager policies,
Expand Down Expand Up @@ -472,8 +461,6 @@ Each feature gate is designed for enabling/disabling a specific feature:
handlers with probers.
- `ContextualLogging`: When you enable this feature gate, Kubernetes components that support
contextual logging add extra detail to log output.
- `ControllerManagerLeaderMigration`: Enables leader migration for
`kube-controller-manager` and `cloud-controller-manager`.
- `CronJobTimeZone`: Allow the use of the `timeZone` optional field in [CronJobs](/docs/concepts/workloads/controllers/cron-jobs/)
- `CrossNamespaceVolumeDataSource`: Enable the usage of cross namespace volume data source
to allow you to specify a source namespace in the `dataSourceRef` field of a
Expand Down Expand Up @@ -523,15 +510,10 @@ Each feature gate is designed for enabling/disabling a specific feature:
This feature gate exists in case any of your existing workloads depend on a
now-corrected fault where Kubernetes ignored exec probe timeouts. See
[readiness probes](/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/#configure-probes).
- `ExpandCSIVolumes`: Enable the expanding of CSI volumes.
- `ExpandedDNSConfig`: Enable kubelet and kube-apiserver to allow more DNS
search paths and longer list of DNS search paths. This feature requires container
runtime support(Containerd: v1.5.6 or higher, CRI-O: v1.22 or higher). See
[Expanded DNS Configuration](/docs/concepts/services-networking/dns-pod-service/#expanded-dns-configuration).
- `ExpandInUsePersistentVolumes`: Enable expanding in-use PVCs. See
[Resizing an in-use PersistentVolumeClaim](/docs/concepts/storage/persistent-volumes/#resizing-an-in-use-persistentvolumeclaim).
- `ExpandPersistentVolumes`: Enable the expanding of persistent volumes. See
[Expanding Persistent Volumes Claims](/docs/concepts/storage/persistent-volumes/#expanding-persistent-volumes-claims).
- `ExperimentalHostUserNamespaceDefaulting`: Enabling the defaulting user
namespace to host. This is for containers that are using other host namespaces,
host mounts, or containers that are privileged or using specific non-namespaced
Expand Down
30 changes: 13 additions & 17 deletions content/en/docs/reference/labels-annotations-taints/_index.md
Original file line number Diff line number Diff line change
Expand Up @@ -848,23 +848,19 @@ When the PodSecurityPolicy admission controller admitted a Pod, the admission co
modified the Pod to have this annotation.
The value of the annotation was the name of the PodSecurityPolicy that was used for validation.

### seccomp.security.alpha.kubernetes.io/pod (deprecated) {#seccomp-security-alpha-kubernetes-io-pod}

This annotation has been deprecated since Kubernetes v1.19 and will become non-functional in a future release.
please use the corresponding pod or container `securityContext.seccompProfile` field instead.
To specify security settings for a Pod, include the `securityContext` field in the Pod specification.
The [`securityContext`](/docs/reference/kubernetes-api/workload-resources/pod-v1/#security-context) field within a Pod's `.spec` defines pod-level security attributes.
When you [specify the security context for a Pod](/docs/tasks/configure-pod-container/security-context/#set-the-security-context-for-a-pod),
the settings you specify apply to all containers in that Pod.

### container.seccomp.security.alpha.kubernetes.io/[NAME] (deprecated) {#container-seccomp-security-alpha-kubernetes-io}

This annotation has been deprecated since Kubernetes v1.19 and will become non-functional in a future release.
please use the corresponding pod or container `securityContext.seccompProfile` field instead.
The tutorial [Restrict a Container's Syscalls with seccomp](/docs/tutorials/security/seccomp/) takes
you through the steps you follow to apply a seccomp profile to a Pod or to one of
its containers. That tutorial covers the supported mechanism for configuring seccomp in Kubernetes,
based on setting `securityContext` within the Pod's `.spec`.
### seccomp.security.alpha.kubernetes.io/pod (non-functional) {#seccomp-security-alpha-kubernetes-io-pod}

Older versions of Kubernetes allowed you to configure seccomp
behavior using this {{< glossary_tooltip text="annotation" term_id="annotation" >}}.
See [Restrict a Container's Syscalls with seccomp](/docs/tutorials/security/seccomp/) to
learn the supported way to specify seccomp restrictions for a Pod.

### container.seccomp.security.alpha.kubernetes.io/[NAME] (non-functional) {#container-seccomp-security-alpha-kubernetes-io}

Older versions of Kubernetes allowed you to configure seccomp
behavior using this {{< glossary_tooltip text="annotation" term_id="annotation" >}}.
See [Restrict a Container's Syscalls with seccomp](/docs/tutorials/security/seccomp/) to
learn the supported way to specify seccomp restrictions for a Pod.

### snapshot.storage.kubernetes.io/allow-volume-mode-change

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -10,8 +10,6 @@ weight: 250

<!-- overview -->

{{< feature-state for_k8s_version="v1.24" state="stable" >}}

{{< glossary_definition term_id="cloud-controller-manager" length="all" prepend="The cloud-controller-manager is">}}

## Background
Expand Down
16 changes: 5 additions & 11 deletions content/en/docs/tutorials/security/seccomp.md
Original file line number Diff line number Diff line change
Expand Up @@ -313,17 +313,11 @@ Here's a manifest for that Pod:
{{< codenew file="pods/security/seccomp/ga/audit-pod.yaml" >}}

{{< note >}}
The functional support for the already deprecated seccomp annotations
`seccomp.security.alpha.kubernetes.io/pod` (for the whole pod) and
`container.seccomp.security.alpha.kubernetes.io/[name]` (for a single container)
is going to be removed with a future release of Kubernetes. Please always use
the native API fields in favor of the annotations.

Since Kubernetes v1.25, kubelets no longer support the annotations, use of the
annotations in static pods is no longer supported, and the seccomp annotations
are no longer auto-populated when pods with seccomp fields are created.
Auto-population of the seccomp fields from the annotations is planned to be
removed in a future release.
Older versions of Kubernetes allowed you to configure seccomp
behavior using {{< glossary_tooltip text="annotations" term_id="annotation" >}}.
Kubernetes {{< skew currentVersion >}} only supports using fields within
`.spec.securityContext` to configure seccomp, and this tutorial explains that
approach.
{{< /note >}}

Create the Pod in the cluster:
Expand Down
4 changes: 2 additions & 2 deletions content/en/examples/pods/pod-with-scheduling-gates.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -4,8 +4,8 @@ metadata:
name: test-pod
spec:
schedulingGates:
- name: foo
- name: bar
- name: example.com/foo
- name: example.com/bar
containers:
- name: pause
image: registry.k8s.io/pause:3.6