generated from kubernetes/kubernetes-template-project
-
Notifications
You must be signed in to change notification settings - Fork 275
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
add docs for workloadPriority (#1170)
- Loading branch information
Showing
6 changed files
with
221 additions
and
11 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
114 changes: 114 additions & 0 deletions
114
site/content/en/docs/concepts/workload_priority_class.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,114 @@ | ||
| --- | ||
| title: "Workload Priority Class" | ||
| date: 2023-10-02 | ||
| weight: 6 | ||
| description: > | ||
| A priority class which value is utilized by Kueue controller and is independent from pod's priority. | ||
| --- | ||
|
|
||
| A `WorkloadPriorityClass` allows you to control the [`Workload`'s](/docs/concepts/workload) priority without affecting the pod's priority. | ||
| This feature is useful for these cases: | ||
| - want to prioritize workloads that remain inactive for a specific duration | ||
| - want to set a lower priority for development workloads and higher priority for production workloads | ||
|
|
||
| A sample WorkloadPriorityClass looks like the following: | ||
|
|
||
| ```yaml | ||
| apiVersion: kueue.x-k8s.io/v1beta1 | ||
| kind: WorkloadPriorityClass | ||
| metadata: | ||
| name: sample-priority | ||
| value: 10000 | ||
| description: "Sample priority" | ||
| ``` | ||
| `WorkloadPriorityClass` objects are cluster scoped, so they can be used by a job in any namespace. | ||
|
|
||
| ## How to use WorkloadPriorityClass on Jobs | ||
|
|
||
| You can specify the `WorkloadPriorityClass` by setting the label `kueue.x-k8s.io/priority-class`. | ||
|
|
||
| ```yaml | ||
| apiVersion: batch/v1 | ||
| kind: Job | ||
| metadata: | ||
| name: sample-job | ||
| labels: | ||
| kueue.x-k8s.io/queue-name: user-queue | ||
| kueue.x-k8s.io/priority-class: sample-priority | ||
| spec: | ||
| ... | ||
| ``` | ||
|
|
||
| Kueue generates the following `Workload` for the Job above. | ||
| The `PriorityClassName` field can accept either `PriorityClass` or | ||
| `WorkloadPriorityClass` name as a value. To distinguish, when using `WorkloadPriorityClass`, | ||
| a `priorityClassSource` field has the `kueue.x-k8s.io/workloadpriorityclass` value. | ||
| When using `PriorityClass`, a `priorityClassSource` field has the `scheduling.k8s.io/priorityclass` value. | ||
|
|
||
| ```yaml | ||
| apiVersion: kueue.x-k8s.io/v1beta1 | ||
| kind: Workload | ||
| metadata: | ||
| name: job-sample-job-7f173 | ||
| spec: | ||
| priorityClassSource: kueue.x-k8s.io/workloadpriorityclass | ||
| priorityClassName: sample-priority | ||
| priority: 10000 | ||
| queueName: user-queue | ||
| ... | ||
| ``` | ||
|
|
||
| For other job frameworks, you can set `WorkloadPriorityClass` using the same label. | ||
| The Following is an example of `MPIJob`. | ||
|
|
||
| ```yaml | ||
| apiVersion: kubeflow.org/v2beta1 | ||
| kind: MPIJob | ||
| metadata: | ||
| name: pi | ||
| labels: | ||
| kueue.x-k8s.io/queue-name: user-queue | ||
| kueue.x-k8s.io/priority-class: sample-priority | ||
| spec: | ||
| ... | ||
| ``` | ||
|
|
||
| ## The relationship between pod's priority and workload's priority | ||
|
|
||
| When creating a `Workload` for a given job, Kueue considers the following scenarios: | ||
| 1. A job specifies both `WorkloadPriorityClass` and `PriorityClass` | ||
| - `WorkloadPriorityClass` is used for the workload's priority. | ||
| - `PriorityClass` is used for the pod's priority. | ||
| 2. A job specifies only `WorkloadPriorityClass` | ||
| - `WorkloadPriorityClass` is used for the workload's priority. | ||
| - `WorkloadPriorityClass` is not used for pod's priority. | ||
| 3. A job specifies only `PriorityClass` | ||
| - `PriorityClass` is used for the workload's priority and pod's priority. | ||
|
|
||
| In certain job frameworks, there are CRDs that: | ||
| - Define multiple pod specs, where each can have their own pod priority, or | ||
| - Define the overall pod priority in a dedicated field. | ||
| By default kueue will take the PriorityClassName of the first PodSet having one set, | ||
| however the integration of the CRD with Kueue can implement | ||
| [`JobWithPriorityClass interface`](https://github.com/kubernetes-sigs/kueue/blob/e162f8508b503d20feb9b31fd0b27d91e58f2c2f/pkg/controller/jobframework/interface.go#L81-L84) | ||
| to change this behavior. You can read the code for each job integration | ||
| to learn how the priority class is obtained. | ||
|
|
||
| ## Where workload's priority is used | ||
|
|
||
| The priority of workloads is used for: | ||
| - Sorting the workloads in the ClusterQueues. | ||
| - Determining whether a workload can preempt others. | ||
|
|
||
| ## Workload's priority values are always mutable | ||
|
|
||
| The `Workload`'s `Priority` field is always mutable. | ||
| If a `Workload` has been pending for a while, you can consider updating its priority to execute it earlier, | ||
| based on your own policies. | ||
| Workload's `PriorityClassSource` and `PriorityClassName` fields are immutable. | ||
|
|
||
| ## What's next? | ||
|
|
||
| - Learn how to [run jobs](/docs/tasks/run_jobs) | ||
| - Learn how to [run jobs with workload priority](/docs/tasks/run_job_with_workload_priority) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
85 changes: 85 additions & 0 deletions
85
site/content/en/docs/tasks/run_job_with_workload_priority.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,85 @@ | ||
| --- | ||
| title: "Run job with WorkloadPriority" | ||
| date: 2023-10-02 | ||
| weight: 8 | ||
| description: > | ||
| Run job with WorkloadPriority, which is independent from Pod's priority | ||
| --- | ||
|
|
||
| Usually, in Kueue, workload's priority is calculated using for pod's priority for queuing and preemption. | ||
| By using a [`WorkloadPriorityClass`](/docs/concepts/workload_priority_class), | ||
| you can independently manage the priority of workloads for queuing and preemption, separate from pod's priority. | ||
|
|
||
| This page contains instructions on how to run a job with workload priority. | ||
|
|
||
| ## Before you begin | ||
|
|
||
| Make sure the following conditions are met: | ||
|
|
||
| - A Kubernetes cluster is running. | ||
| - The kubectl command-line tool has communication with your cluster. | ||
| - [Kueue is installed](/docs/installation). | ||
|
|
||
| ## 0. Create WorkloadPriorityClass | ||
|
|
||
| The WorkloadPriorityClass should be created first. | ||
|
|
||
| ```yaml | ||
| apiVersion: kueue.x-k8s.io/v1beta1 | ||
| kind: WorkloadPriorityClass | ||
| metadata: | ||
| name: sample-priority | ||
| value: 10000 | ||
| description: "Sample priority" | ||
| ``` | ||
| ## 1. Create Job with `kueue.x-k8s.io/priority-class` label | ||
|
|
||
| You can specify the `WorkloadPriorityClass` by setting the label `kueue.x-k8s.io/priority-class`. | ||
| This is same for other CRDs like `RayJob`. | ||
|
|
||
| ```yaml | ||
| apiVersion: batch/v1 | ||
| kind: Job | ||
| metadata: | ||
| name: sample-job | ||
| labels: | ||
| kueue.x-k8s.io/queue-name: user-queue | ||
| kueue.x-k8s.io/priority-class: sample-priority | ||
| spec: | ||
| parallelism: 3 | ||
| completions: 3 | ||
| suspend: true | ||
| template: | ||
| spec: | ||
| containers: | ||
| - name: dummy-job | ||
| image: gcr.io/k8s-staging-perf-tests/sleep:latest | ||
| restartPolicy: Never | ||
| ``` | ||
|
|
||
| Kueue generates the following `Workload` for the Job above. | ||
| The priority of workloads is utilized in queuing, preemption, and other scheduling processes in Kueue. | ||
| This priority doesn't affect pod's priority. | ||
| Workload's `Priority` field is always mutable because it might be useful for the preemption. | ||
| Workload's `PriorityClassSource` and `PriorityClassName` fields are immutable. | ||
|
|
||
| ```yaml | ||
| apiVersion: kueue.x-k8s.io/v1beta1 | ||
| kind: Workload | ||
| metadata: | ||
| name: job-sample-job-7f173 | ||
| spec: | ||
| priorityClassSource: kueue.x-k8s.io/workloadpriorityclass | ||
| priorityClassName: sample-priority | ||
| priority: 10000 | ||
| queueName: user-queue | ||
| podSets: | ||
| - count: 3 | ||
| name: dummy-job | ||
| template: | ||
| spec: | ||
| containers: | ||
| - image: gcr.io/k8s-staging-perf-tests/sleep:latest | ||
| name: dummy-job | ||
| ``` |