diff --git a/content/en/docs/concepts/extend-kubernetes/compute-storage-net/device-plugins.md b/content/en/docs/concepts/extend-kubernetes/compute-storage-net/device-plugins.md index 7722d9ffa8ef3..872f1eb6dbdc0 100644 --- a/content/en/docs/concepts/extend-kubernetes/compute-storage-net/device-plugins.md +++ b/content/en/docs/concepts/extend-kubernetes/compute-storage-net/device-plugins.md @@ -195,7 +195,7 @@ of the device allocations during the upgrade. ## Monitoring device plugin resources -{{< feature-state for_k8s_version="v1.15" state="beta" >}} +{{< feature-state for_k8s_version="v1.28" state="stable" >}} In order to monitor resources provided by device plugins, monitoring agents need to be able to discover the set of devices that are in-use on the node and obtain metadata to describe which diff --git a/content/en/docs/concepts/scheduling-eviction/dynamic-resource-allocation.md b/content/en/docs/concepts/scheduling-eviction/dynamic-resource-allocation.md index cd7f5a0373a87..f08eb3cf5313d 100644 --- a/content/en/docs/concepts/scheduling-eviction/dynamic-resource-allocation.md +++ b/content/en/docs/concepts/scheduling-eviction/dynamic-resource-allocation.md @@ -11,7 +11,7 @@ weight: 65 {{< feature-state for_k8s_version="v1.27" state="alpha" >}} -Dynamic resource allocation is a new API for requesting and sharing resources +Dynamic resource allocation is an API for requesting and sharing resources between pods and containers inside a pod. It is a generalization of the persistent volumes API for generic resources. Third-party resource drivers are responsible for tracking and allocating resources. Different kinds of @@ -32,7 +32,7 @@ check the documentation for that version of Kubernetes. ## API The `resource.k8s.io/v1alpha2` {{< glossary_tooltip text="API group" -term_id="api-group" >}} provides four new types: +term_id="api-group" >}} provides four types: ResourceClass : Defines which resource driver handles a certain kind of @@ -61,7 +61,7 @@ typically using the type defined by a {{< glossary_tooltip term_id="CustomResourceDefinition" text="CRD" >}} that was created when installing a resource driver. -The `core/v1` `PodSpec` defines ResourceClaims that are needed for a Pod in a new +The `core/v1` `PodSpec` defines ResourceClaims that are needed for a Pod in a `resourceClaims` field. Entries in that list reference either a ResourceClaim or a ResourceClaimTemplate. When referencing a ResourceClaim, all Pods using this PodSpec (for example, inside a Deployment or StatefulSet) share the same diff --git a/content/en/docs/concepts/services-networking/dns-pod-service.md b/content/en/docs/concepts/services-networking/dns-pod-service.md index d5cb855626d67..731e593628ed9 100644 --- a/content/en/docs/concepts/services-networking/dns-pod-service.md +++ b/content/en/docs/concepts/services-networking/dns-pod-service.md @@ -292,7 +292,7 @@ Below are the properties a user can specify in the `dnsConfig` field: This property is optional. When specified, the provided list will be merged into the base search domain names generated from the chosen DNS policy. Duplicate domain names are removed. - Kubernetes allows for at most 6 search domains. + Kubernetes allows up to 32 search domains. - `options`: an optional list of objects where each object may have a `name` property (required) and a `value` property (optional). The contents in this property will be merged to the options generated from the specified DNS policy. diff --git a/content/en/docs/concepts/services-networking/service.md b/content/en/docs/concepts/services-networking/service.md index b96ae64ffeea0..f093c2ceb0529 100644 --- a/content/en/docs/concepts/services-networking/service.md +++ b/content/en/docs/concepts/services-networking/service.md @@ -516,7 +516,7 @@ spec: #### Reserve Nodeport Ranges to avoid collisions when port assigning -{{< feature-state for_k8s_version="v1.27" state="alpha" >}} +{{< feature-state for_k8s_version="v1.28" state="beta" >}} The policy for assigning ports to NodePort services applies to both the auto-assignment and the manual assignment scenarios. When a user wants to create a NodePort service that diff --git a/content/en/docs/concepts/workloads/pods/pod-lifecycle.md b/content/en/docs/concepts/workloads/pods/pod-lifecycle.md index 1636fe3a9db83..fa0131eede351 100644 --- a/content/en/docs/concepts/workloads/pods/pod-lifecycle.md +++ b/content/en/docs/concepts/workloads/pods/pod-lifecycle.md @@ -164,7 +164,7 @@ through which the Pod has or has not passed. Kubelet manages the following PodConditions: * `PodScheduled`: the Pod has been scheduled to a node. -* `PodHasNetwork`: (alpha feature; must be [enabled explicitly](#pod-has-network)) the +* `PodReadyToStartContainers`: (alpha feature; must be [enabled explicitly](#pod-has-network)) the Pod sandbox has been successfully created and networking configured. * `ContainersReady`: all containers in the Pod are ready. * `Initialized`: all [init containers](/docs/concepts/workloads/pods/init-containers/) @@ -244,15 +244,19 @@ When a Pod's containers are Ready but at least one custom condition is missing o {{< feature-state for_k8s_version="v1.25" state="alpha" >}} +{{< note >}} +This condition was renamed from PodHasNetwork to PodReadyToStartContainers. +{{< /note >}} + After a Pod gets scheduled on a node, it needs to be admitted by the Kubelet and have any volumes mounted. Once these phases are complete, the Kubelet works with a container runtime (using {{< glossary_tooltip term_id="cri" >}}) to set up a runtime sandbox and configure networking for the Pod. If the -`PodHasNetworkCondition` [feature gate](/docs/reference/command-line-tools-reference/feature-gates/) is enabled, +`PodReadyToStartContainersCondition` [feature gate](/docs/reference/command-line-tools-reference/feature-gates/) is enabled, Kubelet reports whether a pod has reached this initialization milestone through -the `PodHasNetwork` condition in the `status.conditions` field of a Pod. +the `PodReadyToStartContainers` condition in the `status.conditions` field of a Pod. -The `PodHasNetwork` condition is set to `False` by the Kubelet when it detects a +The `PodReadyToStartContainers` condition is set to `False` by the Kubelet when it detects a Pod does not have a runtime sandbox with networking configured. This occurs in the following scenarios: * Early in the lifecycle of the Pod, when the kubelet has not yet begun to set up a sandbox for the Pod using the container runtime. @@ -262,10 +266,10 @@ the following scenarios: * for container runtimes that use virtual machines for isolation, the Pod sandbox virtual machine rebooting, which then requires creating a new sandbox and fresh container network configuration. -The `PodHasNetwork` condition is set to `True` by the kubelet after the +The `PodReadyToStartContainers` condition is set to `True` by the kubelet after the successful completion of sandbox creation and network configuration for the Pod by the runtime plugin. The kubelet can start pulling container images and create -containers after `PodHasNetwork` condition has been set to `True`. +containers after `PodReadyToStartContainers` condition has been set to `True`. For a Pod with init containers, the kubelet sets the `Initialized` condition to `True` after the init containers have successfully completed (which happens diff --git a/content/en/docs/reference/access-authn-authz/authentication.md b/content/en/docs/reference/access-authn-authz/authentication.md index 2a2f937bf2559..960fa736a408f 100644 --- a/content/en/docs/reference/access-authn-authz/authentication.md +++ b/content/en/docs/reference/access-authn-authz/authentication.md @@ -1220,7 +1220,7 @@ The following `ExecCredential` manifest describes a cluster information sample. ## API access to authentication information for a client {#self-subject-review} -{{< feature-state for_k8s_version="v1.27" state="beta" >}} +{{< feature-state for_k8s_version="v1.28" state="stable" >}} If your cluster has the API enabled, you can use the `SelfSubjectReview` API to find out how your Kubernetes cluster maps your authentication information to identify you as a client. This works whether you are authenticating as a user (typically representing @@ -1230,11 +1230,11 @@ a real person) or as a ServiceAccount. Request example (the body would be a `SelfSubjectReview`): ``` -POST /apis/authentication.k8s.io/v1beta1/selfsubjectreviews +POST /apis/authentication.k8s.io/v1/selfsubjectreviews ``` ```json { - "apiVersion": "authentication.k8s.io/v1beta1", + "apiVersion": "authentication.k8s.io/v1", "kind": "SelfSubjectReview" } ``` @@ -1242,7 +1242,7 @@ Response example: ```json { - "apiVersion": "authentication.k8s.io/v1beta1", + "apiVersion": "authentication.k8s.io/v1", "kind": "SelfSubjectReview", "status": { "userInfo": { @@ -1285,7 +1285,7 @@ By providing the output flag, it is also possible to print the JSON or YAML repr {{% tab name="JSON" %}} ```json { - "apiVersion": "authentication.k8s.io/v1alpha1", + "apiVersion": "authentication.k8s.io/v1", "kind": "SelfSubjectReview", "status": { "userInfo": { @@ -1314,7 +1314,7 @@ By providing the output flag, it is also possible to print the JSON or YAML repr {{% tab name="YAML" %}} ```yaml -apiVersion: authentication.k8s.io/v1alpha1 +apiVersion: authentication.k8s.io/v1 kind: SelfSubjectReview status: userInfo: @@ -1351,8 +1351,10 @@ By default, all authenticated users can create `SelfSubjectReview` objects when You can only make `SelfSubjectReview` requests if: * the `APISelfSubjectReview` [feature gate](/docs/reference/command-line-tools-reference/feature-gates/) - is enabled for your cluster (enabled by default after reaching Beta) -* the API server for your cluster has the `authentication.k8s.io/v1alpha1` or `authentication.k8s.io/v1beta1` + is enabled for your cluster (not needed for Kubernetes {{< skew currentVersion >}}, but older + Kubernetes versions might not offer this feature gate, or might default it to be off) +* (if you are running a version of Kubernetes older than v1.28) the API server for your + cluster has the `authentication.k8s.io/v1alpha1` or `authentication.k8s.io/v1beta1` {{< glossary_tooltip term_id="api-group" text="API group" >}} enabled. {{< /note >}} diff --git a/content/en/docs/reference/command-line-tools-reference/feature-gates-removed.md b/content/en/docs/reference/command-line-tools-reference/feature-gates-removed.md index 99026dadec563..1e55b9e10de42 100644 --- a/content/en/docs/reference/command-line-tools-reference/feature-gates-removed.md +++ b/content/en/docs/reference/command-line-tools-reference/feature-gates-removed.md @@ -32,6 +32,9 @@ In the following table: |---------|---------|-------|-------|-------| | `Accelerators` | `false` | Alpha | 1.6 | 1.10 | | `Accelerators` | - | Deprecated | 1.11 | 1.11 | +| `AdvancedAuditing` | `false` | Alpha | 1.7 | 1.7 | +| `AdvancedAuditing` | `true` | Beta | 1.8 | 1.11 | +| `AdvancedAuditing` | `true` | GA | 1.12 | 1.27 | | `AffinityInAnnotations` | `false` | Alpha | 1.6 | 1.7 | | `AffinityInAnnotations` | - | Deprecated | 1.8 | 1.8 | | `AllowExtTrafficLocalEndpoints` | `false` | Beta | 1.4 | 1.6 | @@ -134,6 +137,12 @@ In the following table: | `DefaultPodTopologySpread` | `false` | Alpha | 1.19 | 1.19 | | `DefaultPodTopologySpread` | `true` | Beta | 1.20 | 1.23 | | `DefaultPodTopologySpread` | `true` | GA | 1.24 | 1.25 | +| `DelegateFSGroupToCSIDriver` | `false` | Alpha | 1.22 | 1.22 | +| `DelegateFSGroupToCSIDriver` | `true` | Beta | 1.23 | 1.25 | +| `DelegateFSGroupToCSIDriver` | `true` | GA | 1.26 | 1.27 | +| `DevicePlugins` | `false` | Alpha | 1.8 | 1.9 | +| `DevicePlugins` | `true` | Beta | 1.10 | 1.25 | +| `DevicePlugins` | `true` | GA | 1.26 | 1.27 | | `DynamicAuditing` | `false` | Alpha | 1.13 | 1.18 | | `DynamicAuditing` | - | Deprecated | 1.19 | 1.19 | | `DynamicKubeletConfig` | `false` | Alpha | 1.4 | 1.10 | @@ -244,6 +253,7 @@ In the following table: | `PodDisruptionBudget` | `false` | Alpha | 1.3 | 1.4 | | `PodDisruptionBudget` | `true` | Beta | 1.5 | 1.20 | | `PodDisruptionBudget` | `true` | GA | 1.21 | 1.25 | +| `PodHasNetworkCondition` | `false` | Alpha | 1.25 | 1.27 | | `PodOverhead` | `false` | Alpha | 1.16 | 1.17 | | `PodOverhead` | `true` | Beta | 1.18 | 1.23 | | `PodOverhead` | `true` | GA | 1.24 | 1.25 | @@ -389,6 +399,8 @@ In the following table: - `AffinityInAnnotations`: Enable setting [Pod affinity or anti-affinity](/docs/concepts/scheduling-eviction/assign-pod-node/#affinity-and-anti-affinity). +- `AdvancedAuditing`: Enable [advanced auditing](/docs/tasks/debug/debug-cluster/audit/#advanced-audit) + - `AllowExtTrafficLocalEndpoints`: Enable a service to route external requests to node local endpoints. - `AllowInsecureBackendProxy`: Enable the users to skip TLS verification of @@ -564,6 +576,13 @@ In the following table: - `DefaultPodTopologySpread`: Enables the use of `PodTopologySpread` scheduling plugin to do [default spreading](/docs/concepts/scheduling-eviction/topology-spread-constraints/#internal-default-constraints). +- `DelegateFSGroupToCSIDriver`: If supported by the CSI driver, delegates the + role of applying `fsGroup` from a Pod's `securityContext` to the driver by + passing `fsGroup` through the NodeStageVolume and NodePublishVolume CSI calls. + +- `DevicePlugins`: Enable the [device-plugins](/docs/concepts/extend-kubernetes/compute-storage-net/device-plugins/) + based resource provisioning on nodes. + - `DynamicAuditing`: Used to enable dynamic auditing before v1.19. - `DynamicKubeletConfig`: Enable the dynamic configuration of kubelet. The @@ -699,6 +718,8 @@ In the following table: - `PodDisruptionBudget`: Enable the [PodDisruptionBudget](/docs/tasks/run-application/configure-pdb/) feature. +- `PodHasNetwork`: Enable the kubelet to mark the [PodHasNetwork](/docs/concepts/workloads/pods/pod-lifecycle/#pod-has-network) + condition on pods. This was renamed to `PodReadyToStartContainersCondition` in 1.28. - `PodOverhead`: Enable the [PodOverhead](/docs/concepts/scheduling-eviction/pod-overhead/) feature to account for pod overheads. diff --git a/content/en/docs/reference/command-line-tools-reference/feature-gates.md b/content/en/docs/reference/command-line-tools-reference/feature-gates.md index 5aa61750270b5..be6ea04ceef80 100644 --- a/content/en/docs/reference/command-line-tools-reference/feature-gates.md +++ b/content/en/docs/reference/command-line-tools-reference/feature-gates.md @@ -62,8 +62,6 @@ For a reference to old feature gates that are removed, please refer to | `APIPriorityAndFairness` | `true` | Beta | 1.20 | | | `APIResponseCompression` | `false` | Alpha | 1.7 | 1.15 | | `APIResponseCompression` | `true` | Beta | 1.16 | | -| `APISelfSubjectReview` | `false` | Alpha | 1.26 | 1.26 | -| `APISelfSubjectReview` | `true` | Beta | 1.27 | | | `APIServerIdentity` | `false` | Alpha | 1.20 | 1.25 | | `APIServerIdentity` | `true` | Beta | 1.26 | | | `APIServerTracing` | `false` | Alpha | 1.22 | 1.26 | @@ -101,8 +99,6 @@ For a reference to old feature gates that are removed, please refer to | `ElasticIndexedJob` | `true` | Beta` | 1.27 | | | `EventedPLEG` | `false` | Alpha | 1.26 | 1.26 | | `EventedPLEG` | `false` | Beta | 1.27 | - | -| `ExpandedDNSConfig` | `false` | Alpha | 1.22 | 1.25 | -| `ExpandedDNSConfig` | `true` | Beta | 1.26 | | | `ExperimentalHostUserNamespaceDefaulting` | `false` | Beta | 1.5 | | | `GracefulNodeShutdown` | `false` | Alpha | 1.20 | 1.20 | | `GracefulNodeShutdown` | `true` | Beta | 1.21 | | @@ -130,16 +126,13 @@ For a reference to old feature gates that are removed, please refer to | `KMSv2` | `false` | Alpha | 1.25 | 1.26 | | `KMSv2` | `true` | Beta | 1.27 | | | `KubeletInUserNamespace` | `false` | Alpha | 1.22 | | -| `KubeletPodResources` | `false` | Alpha | 1.13 | 1.14 | -| `KubeletPodResources` | `true` | Beta | 1.15 | | | `KubeletPodResourcesDynamicResources` | `false` | Alpha | 1.27 | | | `KubeletPodResourcesGet` | `false` | Alpha | 1.27 | | | `KubeletPodResourcesGetAllocatable` | `false` | Alpha | 1.21 | 1.22 | | `KubeletPodResourcesGetAllocatable` | `true` | Beta | 1.23 | | | `KubeletTracing` | `false` | Alpha | 1.25 | 1.26 | | `KubeletTracing` | `true` | Beta | 1.27 | | -| `LegacyServiceAccountTokenTracking` | `false` | Alpha | 1.26 | 1.26 | -| `LegacyServiceAccountTokenTracking` | `true` | Beta | 1.27 | | +| `LegacyServiceAccountTokenCleanUp` | `false` | Alpha | 1.28 | | | `LocalStorageCapacityIsolationFSQuotaMonitoring` | `false` | Alpha | 1.15 | - | | `LogarithmicScaleDown` | `false` | Alpha | 1.21 | 1.21 | | `LogarithmicScaleDown` | `true` | Beta | 1.22 | | @@ -175,7 +168,7 @@ For a reference to old feature gates that are removed, please refer to | `PodDeletionCost` | `true` | Beta | 1.22 | | | `PodDisruptionConditions` | `false` | Alpha | 1.25 | 1.25 | | `PodDisruptionConditions` | `true` | Beta | 1.26 | | -| `PodHasNetworkCondition` | `false` | Alpha | 1.25 | | +| `PodReadyToStartContainersCondition` | `false` | Alpha | 1.28 | | | `PodSchedulingReadiness` | `false` | Alpha | 1.26 | 1.26 | | `PodSchedulingReadiness` | `true` | Beta | 1.27 | | | `ProbeTerminationGracePeriod` | `false` | Alpha | 1.21 | 1.21 | @@ -197,7 +190,8 @@ For a reference to old feature gates that are removed, please refer to | `SELinuxMountReadWriteOncePod` | `false` | Alpha | 1.25 | 1.26 | | `SELinuxMountReadWriteOncePod` | `true` | Beta | 1.27 | | | `SecurityContextDeny` | `false` | Alpha | 1.27 | | -| `ServiceNodePortStaticSubrange` | `false` | Alpha | 1.27 | | +| `ServiceNodePortStaticSubrange` | `false` | Alpha | 1.27 | 1.27 | +| `ServiceNodePortStaticSubrange` | `true` | Beta | 1.28 | | | `SizeMemoryBackedVolumes` | `false` | Alpha | 1.20 | 1.21 | | `SizeMemoryBackedVolumes` | `true` | Beta | 1.22 | | | `StableLoadBalancerNodeGet` | `true` | Beta | 1.27 | | @@ -230,9 +224,9 @@ For a reference to old feature gates that are removed, please refer to | Feature | Default | Stage | Since | Until | |---------|---------|-------|-------|-------| -| `AdvancedAuditing` | `false` | Alpha | 1.7 | 1.7 | -| `AdvancedAuditing` | `true` | Beta | 1.8 | 1.11 | -| `AdvancedAuditing` | `true` | GA | 1.12 | - | +| `APISelfSubjectReview` | `false` | Alpha | 1.26 | 1.26 | +| `APISelfSubjectReview` | `true` | Beta | 1.27 | 1.27 | +| `APISelfSubjectReview` | `true` | GA | 1.28 | - | | `CPUManager` | `false` | Alpha | 1.8 | 1.9 | | `CPUManager` | `true` | Beta | 1.10 | 1.25 | | `CPUManager` | `true` | GA | 1.26 | - | @@ -255,12 +249,9 @@ For a reference to old feature gates that are removed, please refer to | `CronJobTimeZone` | `false` | Alpha | 1.24 | 1.24 | | `CronJobTimeZone` | `true` | Beta | 1.25 | 1.26 | | `CronJobTimeZone` | `true` | GA | 1.27 | - | -| `DelegateFSGroupToCSIDriver` | `false` | Alpha | 1.22 | 1.22 | -| `DelegateFSGroupToCSIDriver` | `true` | Beta | 1.23 | 1.25 | -| `DelegateFSGroupToCSIDriver` | `true` | GA | 1.26 |-| -| `DevicePlugins` | `false` | Alpha | 1.8 | 1.9 | -| `DevicePlugins` | `true` | Beta | 1.10 | 1.25 | -| `DevicePlugins` | `true` | GA | 1.26 | - | +| `DaemonSetUpdateSurge` | `false` | Alpha | 1.21 | 1.21 | +| `DaemonSetUpdateSurge` | `true` | Beta | 1.22 | 1.24 | +| `DaemonSetUpdateSurge` | `true` | GA | 1.25 | - | | `DisableAcceleratorUsageMetrics` | `false` | Alpha | 1.19 | 1.19 | | `DisableAcceleratorUsageMetrics` | `true` | Beta | 1.20 | 1.24 | | `DisableAcceleratorUsageMetrics` | `true` | GA | 1.25 |- | @@ -278,6 +269,9 @@ For a reference to old feature gates that are removed, please refer to | `EndpointSliceTerminatingCondition` | `true` | Beta | 1.22 | 1.25 | | `EndpointSliceTerminatingCondition` | `true` | GA | 1.26 | | | `ExecProbeTimeout` | `true` | GA | 1.20 | - | +| `ExpandedDNSConfig` | `false` | Alpha | 1.22 | 1.25 | +| `ExpandedDNSConfig` | `true` | Beta | 1.26 | 1.27 | +| `ExpandedDNSConfig` | `true` | GA | 1.28 | - | | `GRPCContainerProbe` | `false` | Alpha | 1.23 | 1.23 | | `GRPCContainerProbe` | `true` | Beta | 1.24 | 1.26 | | `GRPCContainerProbe` | `true` | GA | 1.27 | | @@ -290,8 +284,14 @@ For a reference to old feature gates that are removed, please refer to | `KubeletCredentialProviders` | `false` | Alpha | 1.20 | 1.23 | | `KubeletCredentialProviders` | `true` | Beta | 1.24 | 1.25 | | `KubeletCredentialProviders` | `true` | GA | 1.26 | - | +| `KubeletPodResources` | `false` | Alpha | 1.13 | 1.14 | +| `KubeletPodResources` | `true` | Beta | 1.15 | 1.27 | +| `KubeletPodResources` | `true` | GA | 1.28 | - | | `LegacyServiceAccountTokenNoAutoGeneration` | `true` | Beta | 1.24 | 1.25 | | `LegacyServiceAccountTokenNoAutoGeneration` | `true` | GA | 1.26 | - | +| `LegacyServiceAccountTokenTracking` | `false` | Alpha | 1.26 | 1.26 | +| `LegacyServiceAccountTokenTracking` | `true` | Beta | 1.27 | 1.27 | +| `LegacyServiceAccountTokenTracking` | `true` | GA | 1.28 | - | | `MixedProtocolLBService` | `false` | Alpha | 1.20 | 1.23 | | `MixedProtocolLBService` | `true` | Beta | 1.24 | 1.25 | | `MixedProtocolLBService` | `true` | GA | 1.26 | - | @@ -387,7 +387,6 @@ Each feature gate is designed for enabling/disabling a specific feature: - `APIServerIdentity`: Assign each API server an ID in a cluster, using a [Lease](/docs/concepts/architecture/leases). - `APIServerTracing`: Add support for distributed tracing in the API server. See [Traces for Kubernetes System Components](/docs/concepts/cluster-administration/system-traces) for more details. -- `AdvancedAuditing`: Enable [advanced auditing](/docs/tasks/debug/debug-cluster/audit/#advanced-audit) - `AggregatedDiscoveryEndpoint`: Enable a single HTTP endpoint `/discovery/` which supports native HTTP caching with ETags containing all APIResources known to the API server. - `AnyVolumeDataSource`: Enable use of any custom resource as the `DataSource` of a @@ -465,11 +464,9 @@ Each feature gate is designed for enabling/disabling a specific feature: - `CustomResourceValidationExpressions`: Enable expression language validation in CRD which will validate customer resource based on validation rules written in the `x-kubernetes-validations` extension. -- `DelegateFSGroupToCSIDriver`: If supported by the CSI driver, delegates the - role of applying `fsGroup` from a Pod's `securityContext` to the driver by - passing `fsGroup` through the NodeStageVolume and NodePublishVolume CSI calls. -- `DevicePlugins`: Enable the [device-plugins](/docs/concepts/extend-kubernetes/compute-storage-net/device-plugins/) - based resource provisioning on nodes. +- `DaemonSetUpdateSurge`: Enables the DaemonSet workloads to maintain + availability during update per node. + See [Perform a Rolling Update on a DaemonSet](/docs/tasks/manage-daemon/update-daemon-set/). - `DisableAcceleratorUsageMetrics`: [Disable accelerator metrics collected by the kubelet](/docs/concepts/cluster-administration/system-metrics/#disable-accelerator-metrics). - `DisableCloudProviders`: Disables any functionality in `kube-apiserver`, @@ -584,9 +581,12 @@ Each feature gate is designed for enabling/disabling a specific feature: OpenTelemetry trace spans. See [Traces for Kubernetes System Components](/docs/concepts/cluster-administration/system-traces) for more details. - `LegacyServiceAccountTokenNoAutoGeneration`: Stop auto-generation of Secret-based - [service account tokens](/docs/reference/access-authn-authz/authentication/#service-account-tokens). + [service account tokens](/docs/concepts/security/service-accounts/#get-a-token). +- `LegacyServiceAccountTokenCleanUp`: Enable cleaning up Secret-based + [service account tokens](/docs/concepts/security/service-accounts/#get-a-token) + when they are not used in a specified time (default to be one year). - `LegacyServiceAccountTokenTracking`: Track usage of Secret-based - [service account tokens](/docs/reference/access-authn-authz/authentication/#service-account-tokens). + [service account tokens](/docs/concepts/security/service-accounts/#get-a-token). - `LocalStorageCapacityIsolationFSQuotaMonitoring`: When `LocalStorageCapacityIsolation` is enabled for [local ephemeral storage](/docs/concepts/configuration/manage-resources-containers/) @@ -650,7 +650,8 @@ Each feature gate is designed for enabling/disabling a specific feature: - `PodAndContainerStatsFromCRI`: Configure the kubelet to gather container and pod stats from the CRI container runtime rather than gathering them from cAdvisor. As of 1.26, this also includes gathering metrics from CRI and emitting them over `/metrics/cadvisor` (rather than having cAdvisor emit them directly). - `PodDisruptionConditions`: Enables support for appending a dedicated pod condition indicating that the pod is being deleted due to a disruption. -- `PodHasNetworkCondition`: Enable the kubelet to mark the [PodHasNetwork](/docs/concepts/workloads/pods/pod-lifecycle/#pod-has-network) condition on pods. +- `PodReadyToStartContainersCondition`: Enable the kubelet to mark the [PodReadyToStartContainers](/docs/concepts/workloads/pods/pod-lifecycle/#pod-has-network) + condition on pods. This was previously (1.25-1.27) known as `PodHasNetworkCondition`. - `PodSchedulingReadiness`: Enable setting `schedulingGates` field to control a Pod's [scheduling readiness](/docs/concepts/scheduling-eviction/pod-scheduling-readiness). - `PodSecurity`: Enables the `PodSecurity` admission plugin. - `ProbeTerminationGracePeriod`: Enable [setting probe-level diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_config_validate.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_config_validate.md new file mode 100644 index 0000000000000..c96ae18fb061c --- /dev/null +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_config_validate.md @@ -0,0 +1,77 @@ + + +PLACEHOLDER + +### Synopsis + +PLACEHOLDER + +``` +kubeadm config validate [flags] +``` + +### Options + + ++++ + + + + + + + + + + + + + + + + + +
-h, --help

help for validate

--config string

PLACEHOLDER

+ + + +### Options inherited from parent commands + + ++++ + + + + + + + + + + + + + + + + + +
--kubeconfig string     Default: "/etc/kubernetes/admin.conf"

The kubeconfig file to use when talking to the cluster. If the flag is not set, a set of standard locations can be searched for an existing kubeconfig file.

--rootfs string

[EXPERIMENTAL] The path to the 'real' host root filesystem.

+ + + diff --git a/content/en/docs/reference/setup-tools/kubeadm/kubeadm-config.md b/content/en/docs/reference/setup-tools/kubeadm/kubeadm-config.md index b9908744e0707..e21f6a00fd161 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/kubeadm-config.md +++ b/content/en/docs/reference/setup-tools/kubeadm/kubeadm-config.md @@ -31,6 +31,8 @@ For more information on using the kubeadm configuration API navigate to You can use `kubeadm config migrate` to convert your old configuration files that contain a deprecated API version to a newer, supported API version. +`kubeadm config validate` can be used for validating a configuration file. + `kubeadm config images list` and `kubeadm config images pull` can be used to list and pull the images that kubeadm requires. @@ -51,6 +53,10 @@ that kubeadm requires. {{< include "generated/kubeadm_config_migrate.md" >}} +## kubeadm config validate {#cmd-config-validate} + +{{< include "generated/kubeadm_config_validate.md" >}} + ## kubeadm config images list {#cmd-config-images-list} {{< include "generated/kubeadm_config_images_list.md" >}} diff --git a/content/en/docs/reference/setup-tools/kubeadm/kubeadm-init.md b/content/en/docs/reference/setup-tools/kubeadm/kubeadm-init.md index 4d102d1a15313..bbc1f104ad7fc 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/kubeadm-init.md +++ b/content/en/docs/reference/setup-tools/kubeadm/kubeadm-init.md @@ -189,6 +189,30 @@ or `kubeadm upgrade apply`), kubeadm respects the value of `UnversionedKubeletCo (during `kubeadm join`, `kubeadm reset`, `kubeadm upgrade ...`), kubeadm attempts to use unversioned ConfigMap name first; if that does not succeed, kubeadm falls back to using the legacy (versioned) name for that ConfigMap. +List of deprecated feature gates: + +{{< table caption="kubeadm deprecated feature gates" >}} +Feature | Default +:-------|:-------- +`UpgradeAddonsBeforeControlPlane` | `false` +{{< /table >}} + +Feature gate descriptions: + +`UpgradeAddonsBeforeControlPlane` +: This is as a **disabled** feature gate that was introduced for Kubernetes v1.28, in order to allow reactivating a legacy +and deprecated behavior during cluster upgrade. For kubeadm versions prior to v1.28, kubeadm upgrades cluster addons (including +CoreDNS and kube-proxy) immediately during `kubeadm upgrade apply`, regardless of whether there are other control plane +instances that have not been upgraded. This may cause compatibility problems. Since v1.28, kubeadm defaults to a mode that +always checks whether all the control plane instances have been upgraded before starting to upgrade the addons. This behavior +is applied to both `kubeadm upgrade apply` and `kubeadm upgrade node`. kubeadm determines whether a control plane instance +has been upgraded by checking whether the image of the kube-apiserver Pod has been upgraded. You must perform control plane +instances upgrade sequentially or at least ensure that the last control plane instance upgrade is not started until all the +other control plane instances have been upgraded completely, and the addons upgrade will be performed after the last control plane +instance is upgraded. The deprecated `UpgradeAddonsBeforeControlPlane` feature gate gives you a chance to keep the old upgrade +behavior. You should not need this old behavior; if you do, you should consider changing your cluster or upgrade processes, as this +feature gate will be removed in a future release. + ### Adding kube-proxy parameters {#kube-proxy} For information about kube-proxy parameters in the kubeadm configuration see: diff --git a/content/en/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade.md b/content/en/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade.md index 82bbecca0583c..9db7e3e7df3d6 100644 --- a/content/en/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade.md +++ b/content/en/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade.md @@ -152,6 +152,20 @@ Pick a control plane node that you wish to upgrade first. It must have the `/etc [upgrade/kubelet] Now that your control plane is upgraded, please proceed with upgrading your kubelets if you haven't already done so. ``` + {{< note >}} + For versions earlier than v1.28, kubeadm defaulted to a mode that upgrades the addons (including CoreDNS and kube-proxy) + immediately during `kubeadm upgrade apply`, regardless of whether there are other control plane instances that have not + been upgraded. This may cause compatibility problems. Since v1.28, kubeadm defaults to a mode that checks whether all + the control plane instances have been upgraded before starting to upgrade the addons. You must perform control plane + instances upgrade sequentially or at least ensure that the last control plane instance upgrade is not started until all + the other control plane instances have been upgraded completely, and the addons upgrade will be performed after the last + control plane instance is upgraded. If you want to keep the old upgrade behavior, please enable the `UpgradeAddonsBeforeControlPlane` + feature gate by `kubeadm upgrade apply --feature-gates=UpgradeAddonsBeforeControlPlane=true`. The Kubernetes project does + not in general recommend enabling this feature gate, you should instead change your upgrade process or cluster addons so + that you do not need to enable the legacy behavior. The `UpgradeAddonsBeforeControlPlane` feature gate will be removed in + a future release. + {{}} + 1. Manually upgrade your CNI provider plugin. Your Container Network Interface (CNI) provider may have its own upgrade instructions to follow. diff --git a/content/en/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definitions.md b/content/en/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definitions.md index edade8f8258fd..45095d6b5311d 100644 --- a/content/en/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definitions.md +++ b/content/en/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definitions.md @@ -1066,6 +1066,33 @@ message will be used instead. `messageExpression` is a CEL expression, so the restrictions listed in [Resource use by validation functions](#resource-use-by-validation-functions) apply. If evaluation halts due to resource constraints during `messageExpression` execution, then no further validation rules will be executed. +#### The reason and fieldPath fields + +Besides the `message` and `messageExpression` fields, which defines the string reported for a validation rule failure, +we have also added two more fields under `validation`: +- `reason` field which allows user to specify a machine-readable validation failure reason when a request fails this validation rule. +- `fieldPath` field which specify the field path returned when the validation fails. + +For example: + +```yaml +x-kubernetes-validations: +- rule: "self.x <= self.maxLimit" + messageExpression: '"x exceeded max limit of " + string(self.maxLimit)' + reason: "FieldValueInvalid" + fieldPath: ".x" +``` + +The HTTP status code returned to the caller will match the reason of the reason of the first failed validation rule. +The currently supported reasons are: "FieldValueInvalid", "FieldValueForbidden", "FieldValueRequired", "FieldValueDuplicate". +If not set, default to use "FieldValueInvalid". + +For `fieldPath`, It must be a relative JSON path scoped to the location of this x-kubernetes-validations extension in the schema and refer to an existing field. +For example when validation checks if a specific attribute `foo` under a map `testMap`, the fieldPath could be set to `".testMap.foo"` or `.tetsMap['foo']'`. +If the validation checks two lists must have unique attributes, the fieldPath could be set to either of the list: e.g. `.testList1` or `.testList2`. +It supports child operation to refer to an existing field currently. Refer to [JSONPath support in Kubernetes](https://kubernetes.io/docs/reference/kubectl/jsonpath/) for more info. +And it does not support numeric index of array. + #### Validation functions {#available-validation-functions} Functions available include: diff --git a/hugo.toml b/hugo.toml index 43503425e215a..87fe12373778b 100644 --- a/hugo.toml +++ b/hugo.toml @@ -138,12 +138,12 @@ time_format_default = "January 02, 2006 at 3:04 PM PST" description = "Production-Grade Container Orchestration" showedit = true -latest = "v1.27" +latest = "v1.28" -fullversion = "v1.27.0" # legacy; use {{< skew currentPatchVersion >}} instead +fullversion = "v1.28.0" # legacy; use {{< skew currentPatchVersion >}} instead # retain until all localizations have migrated -version = "v1.27" +version = "v1.28" githubbranch = "main" docsbranch = "main" deprecated = false @@ -182,13 +182,21 @@ js = [ "script" ] +[[params.versions]] +fullversion = "v1.28.0" # legacy; use {{< skew currentPatchVersion >}} instead + # retain until all localizations have migrated +version = "v1.28" +githubbranch = "v1.28.0" +docsbranch = "main" +url = "https://kubernetes.io" + [[params.versions]] fullversion = "v1.27.0" # legacy; use {{< skew currentPatchVersion >}} instead # retain until all localizations have migrated version = "v1.27" githubbranch = "v1.27.0" -docsbranch = "main" -url = "https://kubernetes.io" +docsbranch = "release-1.27" +url = "https://v1-27.docs.kubernetes.io" [[params.versions]] fullversion = "v1.26.3" # legacy; use {{< skew currentPatchVersion >}} instead @@ -214,14 +222,6 @@ githubbranch = "v1.24.12" docsbranch = "release-1.24" url = "https://v1-24.docs.kubernetes.io" -[[params.versions]] -fullversion = "v1.23.17" # legacy; use {{< skew currentPatchVersion >}} instead - # retain until all localizations have migrated -version = "v1.23" -githubbranch = "v1.23.17" -docsbranch = "release-1.23" -url = "https://v1-23.docs.kubernetes.io" - # User interface configuration [params.ui] # Enable to show the side bar menu in its compact state.