From b12c545697f5fc0e126bd0f8ef3e314b57e2872c Mon Sep 17 00:00:00 2001 From: Heba Elayoty Date: Mon, 20 Oct 2025 15:17:53 -0700 Subject: [PATCH 1/6] Add docs for extended tolerations operators Signed-off-by: Heba Elayoty --- .../taint-and-toleration.md | 130 ++++++++++++++---- .../TaintTolerationComparisonOperators.md | 17 +++ .../pods/pod-with-numeric-toleration.yaml | 16 +++ 3 files changed, 135 insertions(+), 28 deletions(-) create mode 100644 content/en/docs/reference/command-line-tools-reference/feature-gates/TaintTolerationComparisonOperators.md create mode 100644 content/en/examples/pods/pod-with-numeric-toleration.yaml diff --git a/content/en/docs/concepts/scheduling-eviction/taint-and-toleration.md b/content/en/docs/concepts/scheduling-eviction/taint-and-toleration.md index c51e01197e84e..e0b45e0a7b9ce 100644 --- a/content/en/docs/concepts/scheduling-eviction/taint-and-toleration.md +++ b/content/en/docs/concepts/scheduling-eviction/taint-and-toleration.md @@ -11,7 +11,7 @@ weight: 50 [_Node affinity_](/docs/concepts/scheduling-eviction/assign-pod-node/#affinity-and-anti-affinity) -is a property of {{< glossary_tooltip text="Pods" term_id="pod" >}} that *attracts* them to +is a property of {{< glossary_tooltip text="Pods" term_id="pod" >}} that _attracts_ them to a set of {{< glossary_tooltip text="nodes" term_id="node" >}} (either as a preference or a hard requirement). _Taints_ are the opposite -- they allow a node to repel a set of pods. @@ -39,6 +39,7 @@ places a taint on node `node1`. The taint has key `key1`, value `value1`, and ta This means that no pod will be able to schedule onto `node1` unless it has a matching toleration. To remove the taint added by the command above, you can run: + ```shell kubectl taint nodes node1 key1=value1:NoSchedule- ``` @@ -81,6 +82,24 @@ A toleration "matches" a taint if the keys are the same and the effects are the * the `operator` is `Exists` (in which case no `value` should be specified), or * the `operator` is `Equal` and the values should be equal. +{{< feature-state feature_gate_name="TaintTolerationComparisonOperators" >}} + +You can also use numeric comparison operators for threshold-based matching: + +* the `operator` is `Gt` (greater than) and the toleration value is greater than the taint value, or +* the `operator` is `Lt` (less than) and the toleration value is less than the taint value. + +For numeric operators, both the toleration and taint values must be valid integers. +If either value cannot be parsed as an integer, the toleration does not match. + +{{< note >}} +When you create a Pod that uses `Gt` or `Lt` tolerations operators, the API server +validates that the toleration values are valid integers. Taint values on nodes are not +validated at node registration time. If a node has a non-numeric taint value +(for example, `node.kubernetes.io/sla=high:NoSchedule`), pods with numeric comparison +operators will not match that taint and cannot schedule on that node. +{{< /note >}} + {{< note >}} There are two special cases: @@ -93,15 +112,15 @@ An empty `effect` matches all effects with key `key1`. The above example used the `effect` of `NoSchedule`. Alternatively, you can use the `effect` of `PreferNoSchedule`. - The allowed values for the `effect` field are: `NoExecute` : This affects pods that are already running on the node as follows: - * Pods that do not tolerate the taint are evicted immediately - * Pods that tolerate the taint without specifying `tolerationSeconds` in + +* Pods that do not tolerate the taint are evicted immediately +* Pods that tolerate the taint without specifying `tolerationSeconds` in their toleration specification remain bound forever - * Pods that tolerate the taint with a specified `tolerationSeconds` remain +* Pods that tolerate the taint with a specified `tolerationSeconds` remain bound for the specified amount of time. After that time elapses, the node lifecycle controller evicts the Pods from the node. @@ -111,7 +130,7 @@ The allowed values for the `effect` field are: `PreferNoSchedule` : `PreferNoSchedule` is a "preference" or "soft" version of `NoSchedule`. - The control plane will *try* to avoid placing a Pod that does not tolerate + The control plane will _try_ to avoid placing a Pod that does not tolerate the taint on the node, but it is not guaranteed. You can put multiple taints on the same node and multiple tolerations on the same pod. @@ -122,7 +141,7 @@ remaining un-ignored taints have the indicated effects on the pod. In particular * if there is at least one un-ignored taint with effect `NoSchedule` then Kubernetes will not schedule the pod onto that node * if there is no un-ignored taint with effect `NoSchedule` but there is at least one un-ignored taint with -effect `PreferNoSchedule` then Kubernetes will *try* to not schedule the pod onto the node +effect `PreferNoSchedule` then Kubernetes will _try_ to not schedule the pod onto the node * if there is at least one un-ignored taint with effect `NoExecute` then the pod will be evicted from the node (if it is already running on the node), and will not be scheduled onto the node (if it is not yet running on the node). @@ -173,9 +192,62 @@ means that if this pod is running and a matching taint is added to the node, the the pod will stay bound to the node for 3600 seconds, and then be evicted. If the taint is removed before that time, the pod will not be evicted. +## Numeric comparison operators {#numeric-comparison-operators} + +{{< feature-state feature_gate_name="TaintTolerationComparisonOperators" >}} + +In addition to the `Equal` and `Exists` operators, you can use numeric comparison +operators (`Gt` and `Lt`) to match taints with integer values. This is useful for +threshold-based scheduling scenarios, such as matching nodes based on reliability +levels or SLA requirements. + +For example, if nodes are tainted with an SLA value: + +```shell +kubectl taint nodes node1 node.kubernetes.io/sla=950:NoSchedule +``` + +A pod can tolerate nodes with SLA greater than 900: + +{{% code_sample file="pods/pod-with-numeric-toleration.yaml" %}} + +This toleration matches the taint on `node1` because `900 < 950` (the toleration +value is less than the taint value for the `Gt` operator). + +Similarly, you can use the `Lt` operator to match taints where the toleration value +is greater than the taint value: + +```yaml +tolerations: +- key: "node.kubernetes.io/sla" + operator: "Lt" + value: "1000" + effect: "NoSchedule" +``` + +{{< note >}} +When using numeric comparison operators: + +* Both the toleration and taint values must be valid integers (signed 64-bit). +* If a value cannot be parsed as an integer, the toleration does not match. +* Numeric operators work with all taint effects: `NoSchedule`, `PreferNoSchedule`, and `NoExecute`. +{{< /note >}} + +{{< warning >}} + +Before disabling the `TaintTolerationComparisonOperators` feature gate, you should identify +all workloads using the `Gt` or `Lt` operators to avoid controller hot-loops. + +Before disabling the feature gate: + +* Update all workload controller templates to use `Equal` or `Exists` operators instead +* Delete any pending pods that use `Gt` or `Lt` operators +* Monitor the `apiserver_request_total` metric for spikes in validation errors +{{< /warning >}} + ## Example Use Cases -Taints and tolerations are a flexible way to steer pods *away* from nodes or evict +Taints and tolerations are a flexible way to steer pods _away_ from nodes or evict pods that shouldn't be running. A few of the use cases are * **Dedicated Nodes**: If you want to dedicate a set of nodes for exclusive use by @@ -184,8 +256,8 @@ a particular set of users, you can add a taint to those nodes (say, toleration to their pods (this would be done most easily by writing a custom [admission controller](/docs/reference/access-authn-authz/admission-controllers/)). The pods with the tolerations will then be allowed to use the tainted (dedicated) nodes as -well as any other nodes in the cluster. If you want to dedicate the nodes to them *and* -ensure they *only* use the dedicated nodes, then you should additionally add a label similar +well as any other nodes in the cluster. If you want to dedicate the nodes to them _and_ +ensure they _only_ use the dedicated nodes, then you should additionally add a label similar to the taint to the same set of nodes (e.g. `dedicated=groupName`), and the admission controller should additionally add a node affinity to require that the pods can only schedule onto nodes labeled with `dedicated=groupName`. @@ -215,25 +287,28 @@ manually add tolerations to your pods. * **Taint based Evictions**: A per-pod-configurable eviction behavior when there are node problems, which is described in the next section. +* **SLA-based Scheduling**: In clusters with mixed node types (i.e. spot instances), +* you can taint nodes with numeric SLA or reliability values. Pods can then +use numeric comparison operators to opt-in to nodes meeting specific reliability thresholds, +while the cluster's default policy keeps most workloads away from lower-SLA nodes. + ## Taint based Evictions {{< feature-state for_k8s_version="v1.18" state="stable" >}} - - The node controller automatically taints a Node when certain conditions are true. The following taints are built in: - * `node.kubernetes.io/not-ready`: Node is not ready. This corresponds to +* `node.kubernetes.io/not-ready`: Node is not ready. This corresponds to the NodeCondition `Ready` being "`False`". - * `node.kubernetes.io/unreachable`: Node is unreachable from the node +* `node.kubernetes.io/unreachable`: Node is unreachable from the node controller. This corresponds to the NodeCondition `Ready` being "`Unknown`". - * `node.kubernetes.io/memory-pressure`: Node has memory pressure. - * `node.kubernetes.io/disk-pressure`: Node has disk pressure. - * `node.kubernetes.io/pid-pressure`: Node has PID pressure. - * `node.kubernetes.io/network-unavailable`: Node's network is unavailable. - * `node.kubernetes.io/unschedulable`: Node is unschedulable. - * `node.cloudprovider.kubernetes.io/uninitialized`: When the kubelet is started +* `node.kubernetes.io/memory-pressure`: Node has memory pressure. +* `node.kubernetes.io/disk-pressure`: Node has disk pressure. +* `node.kubernetes.io/pid-pressure`: Node has PID pressure. +* `node.kubernetes.io/network-unavailable`: Node's network is unavailable. +* `node.kubernetes.io/unschedulable`: Node is unschedulable. +* `node.cloudprovider.kubernetes.io/uninitialized`: When the kubelet is started with an "external" cloud provider, this taint is set on a node to mark it as unusable. After a controller from the cloud-controller-manager initializes this node, the kubelet removes this taint. @@ -284,8 +359,8 @@ Nodes for 5 minutes after one of these problems is detected. [DaemonSet](/docs/concepts/workloads/controllers/daemonset/) pods are created with `NoExecute` tolerations for the following taints with no `tolerationSeconds`: - * `node.kubernetes.io/unreachable` - * `node.kubernetes.io/not-ready` +* `node.kubernetes.io/unreachable` +* `node.kubernetes.io/not-ready` This ensures that DaemonSet pods are never evicted due to these problems. @@ -320,11 +395,11 @@ onto the affected node. The DaemonSet controller automatically adds the following `NoSchedule` tolerations to all daemons, to prevent DaemonSets from breaking. - * `node.kubernetes.io/memory-pressure` - * `node.kubernetes.io/disk-pressure` - * `node.kubernetes.io/pid-pressure` (1.14 or later) - * `node.kubernetes.io/unschedulable` (1.10 or later) - * `node.kubernetes.io/network-unavailable` (*host network only*) +* `node.kubernetes.io/memory-pressure` +* `node.kubernetes.io/disk-pressure` +* `node.kubernetes.io/pid-pressure` (1.14 or later) +* `node.kubernetes.io/unschedulable` (1.10 or later) +* `node.kubernetes.io/network-unavailable` (_host network only_) Adding these tolerations ensures backward compatibility. You can also add arbitrary tolerations to DaemonSets. @@ -343,4 +418,3 @@ devices. Like taints they apply to all pods which share the same allocated devic and how you can configure it * Read about [Pod Priority](/docs/concepts/scheduling-eviction/pod-priority-preemption/) * Read about [device taints and tolerations](/docs/concepts/scheduling-eviction/dynamic-resource-allocation#device-taints-and-tolerations) - diff --git a/content/en/docs/reference/command-line-tools-reference/feature-gates/TaintTolerationComparisonOperators.md b/content/en/docs/reference/command-line-tools-reference/feature-gates/TaintTolerationComparisonOperators.md new file mode 100644 index 0000000000000..f879ff7a01758 --- /dev/null +++ b/content/en/docs/reference/command-line-tools-reference/feature-gates/TaintTolerationComparisonOperators.md @@ -0,0 +1,17 @@ +--- +title: TaintTolerationComparisonOperators +content_type: feature_gate +_build: + list: never + render: false + +stages: + - stage: alpha + defaultValue: false + fromVersion: "1.35" +--- +Enables numeric comparison operators (`Lt` and `Gt`) for +[tolerations](/docs/concepts/scheduling-eviction/taint-and-toleration/), +allowing pods to match taints using threshold-based comparisons. +This is useful for scenarios like SLA-based scheduling where nodes are +tainted with numeric values representing reliability levels. diff --git a/content/en/examples/pods/pod-with-numeric-toleration.yaml b/content/en/examples/pods/pod-with-numeric-toleration.yaml new file mode 100644 index 0000000000000..a4e8f3bb4252d --- /dev/null +++ b/content/en/examples/pods/pod-with-numeric-toleration.yaml @@ -0,0 +1,16 @@ +apiVersion: v1 +kind: Pod +metadata: + name: nginx-numeric-toleration + labels: + env: test +spec: + containers: + - name: nginx + image: nginx + imagePullPolicy: IfNotPresent + tolerations: + - key: "node.kubernetes.io/sla" + operator: "Gt" + value: "900" + effect: "NoSchedule" From ad6f69e0301895f552e7d883ff8e407e04f6c88b Mon Sep 17 00:00:00 2001 From: Heba Elayoty Date: Tue, 21 Oct 2025 19:58:04 -0700 Subject: [PATCH 2/6] address review feedback Signed-off-by: Heba Elayoty --- .../scheduling-eviction/taint-and-toleration.md | 13 ++++--------- .../TaintTolerationComparisonOperators.md | 5 +---- .../examples/pods/pod-with-numeric-toleration.yaml | 2 +- 3 files changed, 6 insertions(+), 14 deletions(-) diff --git a/content/en/docs/concepts/scheduling-eviction/taint-and-toleration.md b/content/en/docs/concepts/scheduling-eviction/taint-and-toleration.md index e0b45e0a7b9ce..64196313583e2 100644 --- a/content/en/docs/concepts/scheduling-eviction/taint-and-toleration.md +++ b/content/en/docs/concepts/scheduling-eviction/taint-and-toleration.md @@ -96,7 +96,7 @@ If either value cannot be parsed as an integer, the toleration does not match. When you create a Pod that uses `Gt` or `Lt` tolerations operators, the API server validates that the toleration values are valid integers. Taint values on nodes are not validated at node registration time. If a node has a non-numeric taint value -(for example, `node.kubernetes.io/sla=high:NoSchedule`), pods with numeric comparison +(for example, `servicelevel.organization.example/agreed-service-level=high:NoSchedule`), pods with numeric comparison operators will not match that taint and cannot schedule on that node. {{< /note >}} @@ -201,10 +201,10 @@ operators (`Gt` and `Lt`) to match taints with integer values. This is useful fo threshold-based scheduling scenarios, such as matching nodes based on reliability levels or SLA requirements. -For example, if nodes are tainted with an SLA value: +For example, if nodes are tainted with a value representing a service level agreement (SLA): ```shell -kubectl taint nodes node1 node.kubernetes.io/sla=950:NoSchedule +kubectl taint nodes node1 servicelevel.organization.example/agreed-service-level=950:NoSchedule ``` A pod can tolerate nodes with SLA greater than 900: @@ -219,7 +219,7 @@ is greater than the taint value: ```yaml tolerations: -- key: "node.kubernetes.io/sla" +- key: "servicelevel.organization.example/agreed-service-level" operator: "Lt" value: "1000" effect: "NoSchedule" @@ -287,11 +287,6 @@ manually add tolerations to your pods. * **Taint based Evictions**: A per-pod-configurable eviction behavior when there are node problems, which is described in the next section. -* **SLA-based Scheduling**: In clusters with mixed node types (i.e. spot instances), -* you can taint nodes with numeric SLA or reliability values. Pods can then -use numeric comparison operators to opt-in to nodes meeting specific reliability thresholds, -while the cluster's default policy keeps most workloads away from lower-SLA nodes. - ## Taint based Evictions {{< feature-state for_k8s_version="v1.18" state="stable" >}} diff --git a/content/en/docs/reference/command-line-tools-reference/feature-gates/TaintTolerationComparisonOperators.md b/content/en/docs/reference/command-line-tools-reference/feature-gates/TaintTolerationComparisonOperators.md index f879ff7a01758..316fed8278dbf 100644 --- a/content/en/docs/reference/command-line-tools-reference/feature-gates/TaintTolerationComparisonOperators.md +++ b/content/en/docs/reference/command-line-tools-reference/feature-gates/TaintTolerationComparisonOperators.md @@ -11,7 +11,4 @@ stages: fromVersion: "1.35" --- Enables numeric comparison operators (`Lt` and `Gt`) for -[tolerations](/docs/concepts/scheduling-eviction/taint-and-toleration/), -allowing pods to match taints using threshold-based comparisons. -This is useful for scenarios like SLA-based scheduling where nodes are -tainted with numeric values representing reliability levels. +[tolerations](/docs/concepts/scheduling-eviction/taint-and-toleration/). diff --git a/content/en/examples/pods/pod-with-numeric-toleration.yaml b/content/en/examples/pods/pod-with-numeric-toleration.yaml index a4e8f3bb4252d..cfea2b3245e1a 100644 --- a/content/en/examples/pods/pod-with-numeric-toleration.yaml +++ b/content/en/examples/pods/pod-with-numeric-toleration.yaml @@ -10,7 +10,7 @@ spec: image: nginx imagePullPolicy: IfNotPresent tolerations: - - key: "node.kubernetes.io/sla" + - key: "servicelevel.organization.example/agreed-service-level" operator: "Gt" value: "900" effect: "NoSchedule" From 57a6ba435c46ad8255ac6b826b7eb84742f57285 Mon Sep 17 00:00:00 2001 From: Heba Elayoty Date: Mon, 17 Nov 2025 20:46:21 -0800 Subject: [PATCH 3/6] add final updates Signed-off-by: Heba Elayoty --- .../taint-and-toleration.md | 27 ++++++------------- 1 file changed, 8 insertions(+), 19 deletions(-) diff --git a/content/en/docs/concepts/scheduling-eviction/taint-and-toleration.md b/content/en/docs/concepts/scheduling-eviction/taint-and-toleration.md index 64196313583e2..2d6da8502ce8a 100644 --- a/content/en/docs/concepts/scheduling-eviction/taint-and-toleration.md +++ b/content/en/docs/concepts/scheduling-eviction/taint-and-toleration.md @@ -89,14 +89,10 @@ You can also use numeric comparison operators for threshold-based matching: * the `operator` is `Gt` (greater than) and the toleration value is greater than the taint value, or * the `operator` is `Lt` (less than) and the toleration value is less than the taint value. -For numeric operators, both the toleration and taint values must be valid integers. -If either value cannot be parsed as an integer, the toleration does not match. +For numeric operators, both the toleration and taint values must be valid integers. If either value cannot be parsed as an integer, the toleration does not match. {{< note >}} -When you create a Pod that uses `Gt` or `Lt` tolerations operators, the API server -validates that the toleration values are valid integers. Taint values on nodes are not -validated at node registration time. If a node has a non-numeric taint value -(for example, `servicelevel.organization.example/agreed-service-level=high:NoSchedule`), pods with numeric comparison +When you create a Pod that uses `Gt` or `Lt` tolerations operators, the API server validates that the toleration values are valid integers. Taint values on nodes are not validated at node registration time. If a node has a non-numeric taint value (for example, `servicelevel.organization.example/agreed-service-level=high:NoSchedule`), pods with numeric comparison operators will not match that taint and cannot schedule on that node. {{< /note >}} @@ -196,10 +192,7 @@ taint is removed before that time, the pod will not be evicted. {{< feature-state feature_gate_name="TaintTolerationComparisonOperators" >}} -In addition to the `Equal` and `Exists` operators, you can use numeric comparison -operators (`Gt` and `Lt`) to match taints with integer values. This is useful for -threshold-based scheduling scenarios, such as matching nodes based on reliability -levels or SLA requirements. +In addition to the `Equal` and `Exists` operators, you can use numeric comparison operators (`Gt` and `Lt`) to match taints with integer values. This is useful for threshold-based scheduling scenarios, such as matching nodes based on reliability levels or SLA requirements. For example, if nodes are tainted with a value representing a service level agreement (SLA): @@ -211,11 +204,9 @@ A pod can tolerate nodes with SLA greater than 900: {{% code_sample file="pods/pod-with-numeric-toleration.yaml" %}} -This toleration matches the taint on `node1` because `900 < 950` (the toleration -value is less than the taint value for the `Gt` operator). +This toleration matches the taint on `node1` because `900 < 950` (the toleration value is less than the taint value for the `Gt` operator). -Similarly, you can use the `Lt` operator to match taints where the toleration value -is greater than the taint value: +Similarly, you can use the `Lt` operator to match taints where the toleration value is greater than the taint value: ```yaml tolerations: @@ -228,18 +219,16 @@ tolerations: {{< note >}} When using numeric comparison operators: -* Both the toleration and taint values must be valid integers (signed 64-bit). +* Both the toleration and taint values must be valid signed 64-bit integers (zero leading numbers (e.g., "0550") are not allowed). * If a value cannot be parsed as an integer, the toleration does not match. * Numeric operators work with all taint effects: `NoSchedule`, `PreferNoSchedule`, and `NoExecute`. {{< /note >}} {{< warning >}} -Before disabling the `TaintTolerationComparisonOperators` feature gate, you should identify -all workloads using the `Gt` or `Lt` operators to avoid controller hot-loops. - -Before disabling the feature gate: +Before disabling the `TaintTolerationComparisonOperators` feature gate: +* You should identify all workloads using the `Gt` or `Lt` operators to avoid controller hot-loops. * Update all workload controller templates to use `Equal` or `Exists` operators instead * Delete any pending pods that use `Gt` or `Lt` operators * Monitor the `apiserver_request_total` metric for spikes in validation errors From ded370ab549273009c0faa32091267a9df0730e3 Mon Sep 17 00:00:00 2001 From: Heba Elayoty Date: Tue, 18 Nov 2025 10:34:24 -0800 Subject: [PATCH 4/6] apply sembr lint Signed-off-by: Heba Elayoty --- .../taint-and-toleration.md | 27 ++++++++++++------- 1 file changed, 18 insertions(+), 9 deletions(-) diff --git a/content/en/docs/concepts/scheduling-eviction/taint-and-toleration.md b/content/en/docs/concepts/scheduling-eviction/taint-and-toleration.md index 2d6da8502ce8a..fbc200585759f 100644 --- a/content/en/docs/concepts/scheduling-eviction/taint-and-toleration.md +++ b/content/en/docs/concepts/scheduling-eviction/taint-and-toleration.md @@ -89,18 +89,23 @@ You can also use numeric comparison operators for threshold-based matching: * the `operator` is `Gt` (greater than) and the toleration value is greater than the taint value, or * the `operator` is `Lt` (less than) and the toleration value is less than the taint value. -For numeric operators, both the toleration and taint values must be valid integers. If either value cannot be parsed as an integer, the toleration does not match. +For numeric operators, both the toleration and taint values must be valid integers. +If either value cannot be parsed as an integer, the toleration does not match. {{< note >}} -When you create a Pod that uses `Gt` or `Lt` tolerations operators, the API server validates that the toleration values are valid integers. Taint values on nodes are not validated at node registration time. If a node has a non-numeric taint value (for example, `servicelevel.organization.example/agreed-service-level=high:NoSchedule`), pods with numeric comparison -operators will not match that taint and cannot schedule on that node. +When you create a Pod that uses `Gt` or `Lt` tolerations operators, the API server validates that +the toleration values are valid integers. Taint values on nodes are not validated at node +registration time. If a node has a non-numeric taint value (for example, +`servicelevel.organization.example/agreed-service-level=high:NoSchedule`), +pods with numeric comparison operators will not match that taint and cannot schedule on that node. {{< /note >}} {{< note >}} There are two special cases: -If the `key` is empty, then the `operator` must be `Exists`, which matches all keys and values. Note that the `effect` still needs to be matched at the same time. +If the `key` is empty, then the `operator` must be `Exists`, which matches all keys and values. +Note that the `effect` still needs to be matched at the same time. An empty `effect` matches all effects with key `key1`. @@ -192,7 +197,9 @@ taint is removed before that time, the pod will not be evicted. {{< feature-state feature_gate_name="TaintTolerationComparisonOperators" >}} -In addition to the `Equal` and `Exists` operators, you can use numeric comparison operators (`Gt` and `Lt`) to match taints with integer values. This is useful for threshold-based scheduling scenarios, such as matching nodes based on reliability levels or SLA requirements. +In addition to the `Equal` and `Exists` operators, you can use numeric comparison operators +(`Gt` and `Lt`) to match taints with integer values. This is useful for threshold-based scheduling +scenarios, such as matching nodes based on reliability levels or SLA requirements. For example, if nodes are tainted with a value representing a service level agreement (SLA): @@ -204,9 +211,10 @@ A pod can tolerate nodes with SLA greater than 900: {{% code_sample file="pods/pod-with-numeric-toleration.yaml" %}} -This toleration matches the taint on `node1` because `900 < 950` (the toleration value is less than the taint value for the `Gt` operator). - -Similarly, you can use the `Lt` operator to match taints where the toleration value is greater than the taint value: +This toleration matches the taint on `node1` because `900 < 950` (the toleration value +is less than the taint value for the `Gt` operator). +Similarly, you can use the `Lt` operator to match taints where the toleration value is +greater than the taint value: ```yaml tolerations: @@ -219,7 +227,8 @@ tolerations: {{< note >}} When using numeric comparison operators: -* Both the toleration and taint values must be valid signed 64-bit integers (zero leading numbers (e.g., "0550") are not allowed). +* Both the toleration and taint values must be valid signed 64-bit integers + (zero leading numbers (e.g., "0550") are not allowed). * If a value cannot be parsed as an integer, the toleration does not match. * Numeric operators work with all taint effects: `NoSchedule`, `PreferNoSchedule`, and `NoExecute`. {{< /note >}} From 3216dbcdbb1977cf517334920127f8bd8514ebb3 Mon Sep 17 00:00:00 2001 From: Heba Elayoty Date: Tue, 25 Nov 2025 13:39:03 -0800 Subject: [PATCH 5/6] Address review feedback Signed-off-by: Heba Elayoty --- .../taint-and-toleration.md | 23 +++++++++++-------- 1 file changed, 13 insertions(+), 10 deletions(-) diff --git a/content/en/docs/concepts/scheduling-eviction/taint-and-toleration.md b/content/en/docs/concepts/scheduling-eviction/taint-and-toleration.md index fbc200585759f..4dd15c5c7682f 100644 --- a/content/en/docs/concepts/scheduling-eviction/taint-and-toleration.md +++ b/content/en/docs/concepts/scheduling-eviction/taint-and-toleration.md @@ -118,12 +118,12 @@ The allowed values for the `effect` field are: `NoExecute` : This affects pods that are already running on the node as follows: -* Pods that do not tolerate the taint are evicted immediately -* Pods that tolerate the taint without specifying `tolerationSeconds` in - their toleration specification remain bound forever -* Pods that tolerate the taint with a specified `tolerationSeconds` remain - bound for the specified amount of time. After that time elapses, the node - lifecycle controller evicts the Pods from the node. + * Pods that do not tolerate the taint are evicted immediately + * Pods that tolerate the taint without specifying `tolerationSeconds` in + their toleration specification remain bound forever + * Pods that tolerate the taint with a specified `tolerationSeconds` remain + bound for the specified amount of time. After that time elapses, the node + lifecycle controller evicts the Pods from the node. `NoSchedule` : No new Pods will be scheduled on the tainted node unless they have a matching @@ -211,10 +211,10 @@ A pod can tolerate nodes with SLA greater than 900: {{% code_sample file="pods/pod-with-numeric-toleration.yaml" %}} -This toleration matches the taint on `node1` because `900 < 950` (the toleration value -is less than the taint value for the `Gt` operator). -Similarly, you can use the `Lt` operator to match taints where the toleration value is -greater than the taint value: +This toleration matches the taint on `node1` because `950 > 900` (the taint value +is greater than the toleration value for the `Gt` operator). +Similarly, you can use the `Lt` operator to match taints where the taint value is +less than the toleration value: ```yaml tolerations: @@ -231,6 +231,9 @@ When using numeric comparison operators: (zero leading numbers (e.g., "0550") are not allowed). * If a value cannot be parsed as an integer, the toleration does not match. * Numeric operators work with all taint effects: `NoSchedule`, `PreferNoSchedule`, and `NoExecute`. +* For `PreferNoSchedule` with numeric operators: if a pod's toleration doesn't satisfy the numeric comparison + (e.g., toleration value < taint value when using `Gt`), the scheduler gives the node a lower priority + but may still schedule there if no better options exist. {{< /note >}} {{< warning >}} From e284f1356430617d835ed648fe27a2f2306ce0cd Mon Sep 17 00:00:00 2001 From: Heba <31887807+helayoty@users.noreply.github.com> Date: Wed, 26 Nov 2025 13:50:00 -0800 Subject: [PATCH 6/6] Update content/en/docs/concepts/scheduling-eviction/taint-and-toleration.md MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-authored-by: Maciej Skoczeń <87243939+macsko@users.noreply.github.com> --- .../docs/concepts/scheduling-eviction/taint-and-toleration.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/en/docs/concepts/scheduling-eviction/taint-and-toleration.md b/content/en/docs/concepts/scheduling-eviction/taint-and-toleration.md index 4dd15c5c7682f..a7de1b0199852 100644 --- a/content/en/docs/concepts/scheduling-eviction/taint-and-toleration.md +++ b/content/en/docs/concepts/scheduling-eviction/taint-and-toleration.md @@ -232,7 +232,7 @@ When using numeric comparison operators: * If a value cannot be parsed as an integer, the toleration does not match. * Numeric operators work with all taint effects: `NoSchedule`, `PreferNoSchedule`, and `NoExecute`. * For `PreferNoSchedule` with numeric operators: if a pod's toleration doesn't satisfy the numeric comparison - (e.g., toleration value < taint value when using `Gt`), the scheduler gives the node a lower priority + (e.g., toleration value < taint value when using `Gt`), the `TaintToleration` plugin gives the node a lower score but may still schedule there if no better options exist. {{< /note >}}