add docs for workloadPriority #1170
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,114 @@ | ||
| --- | ||
| title: "Workload Priority Class" | ||
Gekko0114 marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| date: 2023-10-02 | ||
| weight: 6 | ||
| description: > | ||
| A priority class which value is utilized by Kueue controller and is independent from pod's priority. | ||
| --- | ||
|
|
||
| A `WorkloadPriorityClass` allows you to control the [`Workload`'s](/docs/concepts/workload) priority without affecting the pod's priority. | ||
| This feature is useful for these cases: | ||
| - want to prioritize workloads that remain inactive for a specific duration | ||
| - want to set a lower priority for development workloads and higher priority for production workloads | ||
|
|
||
| A sample WorkloadPriorityClass looks like the following: | ||
|
|
||
| ```yaml | ||
| apiVersion: kueue.x-k8s.io/v1beta1 | ||
| kind: WorkloadPriorityClass | ||
| metadata: | ||
| name: sample-priority | ||
| value: 10000 | ||
| description: "Sample priority" | ||
| ``` | ||
Gekko0114 marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
|
||
| `WorkloadPriorityClass` objects are cluster scoped, so they can be used by a job in any namespace. | ||
|
|
||
| ## How to use WorkloadPriorityClass on Jobs | ||
|
|
||
| You can specify the `WorkloadPriorityClass` by setting the label `kueue.x-k8s.io/priority-class`. | ||
|
|
||
| ```yaml | ||
| apiVersion: batch/v1 | ||
| kind: Job | ||
| metadata: | ||
| name: sample-job | ||
| labels: | ||
| kueue.x-k8s.io/queue-name: user-queue | ||
| kueue.x-k8s.io/priority-class: sample-priority | ||
| spec: | ||
| ... | ||
| ``` | ||
|
|
||
| Kueue generates the following `Workload` for the Job above. | ||
| The `PriorityClassName` field can accept either `PriorityClass` or | ||
| `WorkloadPriorityClass` name as a value. To distinguish, when using `WorkloadPriorityClass`, | ||
| a `priorityClassSource` field has the `kueue.x-k8s.io/workloadpriorityclass` value. | ||
| When using `PriorityClass`, a `priorityClassSource` field has the `scheduling.k8s.io/priorityclass` value. | ||
|
|
||
| ```yaml | ||
| apiVersion: kueue.x-k8s.io/v1beta1 | ||
| kind: Workload | ||
| metadata: | ||
| name: job-sample-job-7f173 | ||
| spec: | ||
| priorityClassSource: kueue.x-k8s.io/workloadpriorityclass | ||
| priorityClassName: sample-priority | ||
| priority: 10000 | ||
| queueName: user-queue | ||
| ... | ||
| ``` | ||
|
|
||
| For other job frameworks, you can set `WorkloadPriorityClass` using the same label. | ||
| The Following is an example of `MPIJob`. | ||
|
|
||
| ```yaml | ||
| apiVersion: kubeflow.org/v2beta1 | ||
| kind: MPIJob | ||
| metadata: | ||
| name: pi | ||
| labels: | ||
| kueue.x-k8s.io/queue-name: user-queue | ||
| kueue.x-k8s.io/priority-class: sample-priority | ||
| spec: | ||
| ... | ||
| ``` | ||
|
|
||
| ## The relationship between pod's priority and workload's priority | ||
|
|
||
| When creating a `Workload` for a given job, Kueue considers the following scenarios: | ||
| 1. A job specifies both `WorkloadPriorityClass` and `PriorityClass` | ||
| - `WorkloadPriorityClass` is used for the workload's priority. | ||
| - `PriorityClass` is used for the pod's priority. | ||
| 2. A job specifies only `WorkloadPriorityClass` | ||
| - `WorkloadPriorityClass` is used for the workload's priority. | ||
| - `WorkloadPriorityClass` is not used for pod's priority. | ||
| 3. A job specifies only `PriorityClass` | ||
| - `PriorityClass` is used for the workload's priority and pod's priority. | ||
|
|
||
|
There was a problem hiding this comment. Also, it would be better to explain what happens if the job implement the There was a problem hiding this comment. @tenzen-y this might be too developer focused and a user cannot do too much about it. There was a problem hiding this comment. I was also concerned about what you say. However, actually, which priorities are used depends on the interface. There was a problem hiding this comment. Or, instead of mentioning the interface, raise MPIJob as a specific case. @alculquicondor Any better ideas? There was a problem hiding this comment. Thanks for your comments! Added the explanation related to JobWithPriorityClass There was a problem hiding this comment. I like how Hiroyuki phrased it :) |
||
| In certain job frameworks, there are CRDs that: | ||
| - Define multiple pod specs, where each can have their own pod priority, or | ||
| - Define the overall pod priority in a dedicated field. | ||
| By default kueue will take the PriorityClassName of the first PodSet having one set, | ||
| however the integration of the CRD with Kueue can implement | ||
| [`JobWithPriorityClass interface`](https://github.com/kubernetes-sigs/kueue/blob/e162f8508b503d20feb9b31fd0b27d91e58f2c2f/pkg/controller/jobframework/interface.go#L81-L84) | ||
| to change this behavior. You can read the code for each job integration | ||
|
There was a problem hiding this comment. I guess that's what we have today :( If you have the time, please update the documentation for MPIJob and other kubeflow CRDs to explain how we obtain the priority class for each. |
||
| to learn how the priority class is obtained. | ||
|
|
||
| ## Where workload's priority is used | ||
|
|
||
| The priority of workloads is used for: | ||
| - Sorting the workloads in the ClusterQueues. | ||
| - Determining whether a workload can preempt others. | ||
|
|
||
| ## Workload's priority values are always mutable | ||
|
|
||
| The `Workload`'s `Priority` field is always mutable. | ||
| If a `Workload` has been pending for a while, you can consider updating its priority to execute it earlier, | ||
| based on your own policies. | ||
| Workload's `PriorityClassSource` and `PriorityClassName` fields are immutable. | ||
|
|
||
| ## What's next? | ||
|
|
||
| - Learn how to [run jobs](/docs/tasks/run_jobs) | ||
| - Learn how to [run jobs with workload priority](/docs/tasks/run_job_with_workload_priority) | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,85 @@ | ||
| --- | ||
| title: "Run job with WorkloadPriority" | ||
| date: 2023-10-02 | ||
| weight: 8 | ||
| description: > | ||
| Run job with WorkloadPriority, which is independent from Pod's priority | ||
| --- | ||
|
|
||
| Usually, in Kueue, workload's priority is calculated using for pod's priority for queuing and preemption. | ||
| By using a [`WorkloadPriorityClass`](/docs/concepts/workload_priority_class), | ||
| you can independently manage the priority of workloads for queuing and preemption, separate from pod's priority. | ||
|
|
||
| This page contains instructions on how to run a job with workload priority. | ||
|
|
||
| ## Before you begin | ||
|
|
||
| Make sure the following conditions are met: | ||
|
|
||
| - A Kubernetes cluster is running. | ||
| - The kubectl command-line tool has communication with your cluster. | ||
| - [Kueue is installed](/docs/installation). | ||
|
|
||
| ## 0. Create WorkloadPriorityClass | ||
|
|
||
| The WorkloadPriorityClass should be created first. | ||
|
|
||
| ```yaml | ||
| apiVersion: kueue.x-k8s.io/v1beta1 | ||
| kind: WorkloadPriorityClass | ||
| metadata: | ||
| name: sample-priority | ||
| value: 10000 | ||
| description: "Sample priority" | ||
| ``` | ||
|
|
||
| ## 1. Create Job with `kueue.x-k8s.io/priority-class` label | ||
|
|
||
| You can specify the `WorkloadPriorityClass` by setting the label `kueue.x-k8s.io/priority-class`. | ||
| This is same for other CRDs like `RayJob`. | ||
|
|
||
| ```yaml | ||
| apiVersion: batch/v1 | ||
| kind: Job | ||
| metadata: | ||
| name: sample-job | ||
| labels: | ||
| kueue.x-k8s.io/queue-name: user-queue | ||
| kueue.x-k8s.io/priority-class: sample-priority | ||
| spec: | ||
| parallelism: 3 | ||
| completions: 3 | ||
| suspend: true | ||
| template: | ||
| spec: | ||
| containers: | ||
| - name: dummy-job | ||
| image: gcr.io/k8s-staging-perf-tests/sleep:latest | ||
| restartPolicy: Never | ||
| ``` | ||
|
|
||
| Kueue generates the following `Workload` for the Job above. | ||
| The priority of workloads is utilized in queuing, preemption, and other scheduling processes in Kueue. | ||
| This priority doesn't affect pod's priority. | ||
| Workload's `Priority` field is always mutable because it might be useful for the preemption. | ||
| Workload's `PriorityClassSource` and `PriorityClassName` fields are immutable. | ||
|
|
||
| ```yaml | ||
| apiVersion: kueue.x-k8s.io/v1beta1 | ||
| kind: Workload | ||
| metadata: | ||
| name: job-sample-job-7f173 | ||
| spec: | ||
| priorityClassSource: kueue.x-k8s.io/workloadpriorityclass | ||
| priorityClassName: sample-priority | ||
| priority: 10000 | ||
| queueName: user-queue | ||
| podSets: | ||
| - count: 3 | ||
| name: dummy-job | ||
| template: | ||
| spec: | ||
| containers: | ||
| - image: gcr.io/k8s-staging-perf-tests/sleep:latest | ||
| name: dummy-job | ||
| ``` |
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can you update the
Prioritysection in this file as well?There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
updated