-
Notifications
You must be signed in to change notification settings - Fork 299
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add docs for pod integration #1103
Changes from 3 commits
25d5189
e11c93e
adfe10d
d0b2efd
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,90 @@ | ||
--- | ||
title: "Run A Plain Pod" | ||
date: 2023-09-27 | ||
weight: 6 | ||
description: > | ||
Run a Kueue scheduled Pod. | ||
--- | ||
|
||
This page shows how to leverage Kueue's scheduling and resource management capabilities when running plain Pods. | ||
|
||
This guide is for [batch users](/docs/tasks#batch-user) that have a basic understanding of Kueue. For more information, see [Kueue's overview](/docs/overview). | ||
|
||
## Before you begin | ||
|
||
1. By default, the integration for `v1/pod` is not enabled. | ||
Learn how to [install Kueue with a custom manager configuration](/docs/installation/#install-a-custom-configured-released-version) | ||
and enable the `pod` integration. | ||
|
||
A configuration for Kueue with enabled pod integration would look like follows: | ||
```yaml | ||
apiVersion: config.kueue.x-k8s.io/v1beta1 | ||
kind: Configuration | ||
integrations: | ||
frameworks: | ||
- "pod" | ||
podOptions: | ||
# You can change namespaceSelector to define in which | ||
# namespaces kueue will manage the pods. | ||
namespaceSelector: | ||
matchExpressions: | ||
- key: kubernetes.io/metadata.name | ||
operator: NotIn | ||
values: [ kube-system, kueue-system ] | ||
# Kueue uses podSelector to manage pods with particular | ||
# labels. The default podSelector will match all the pods. | ||
podSelector: | ||
matchExpressions: | ||
- key: kueue-job | ||
operator: In | ||
values: [ "true", "True", "yes" ] | ||
``` | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Mention that Pods that belong to some CRDs that we manage are excluded from being queued. And that kueue adds a label to indicate which nodes are managed. |
||
|
||
2. Pods that belong to other API resources managed by Kueue are excluded from being queued by `pod` integration. | ||
For example, pods managed by `batch/v1/Job` won't be managed by `pod` integration if the `job` integration is enabled. | ||
achernevskii marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
||
3. Kueue will inject a `kueue.x-k8s.io/managed=true` label to indicate which pods are managed by it. | ||
|
||
4. Check [Administer cluster quotas](/docs/tasks/administer_cluster_quotas) for details on the initial Kueue setup. | ||
|
||
## Pod definition | ||
|
||
When running Pods on Kueue, take into consideration the following aspects: | ||
|
||
### a. Queue selection | ||
|
||
The target [local queue](/docs/concepts/local_queue) should be specified in the `metadata.labels` section of the Pod configuration. | ||
|
||
```yaml | ||
metadata: | ||
labels: | ||
kueue.x-k8s.io/queue-name: user-queue | ||
``` | ||
|
||
### b. Configure the resource needs | ||
|
||
The resource needs of the workload can be configured in the `spec.containers`. | ||
|
||
```yaml | ||
- resources: | ||
requests: | ||
cpu: 3 | ||
``` | ||
|
||
### c. Limitations | ||
|
||
- A Kueue managed Pod cannot be created in `kube-system` or `kueue-system` namespaces. | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Mention that preemption actually terminates and deletes the pod. When mentioning preemption, link to the documentation related to it. |
||
- In case of [preemption](/docs/concepts/cluster_queue/#preemption), Pod will | ||
achernevskii marked this conversation as resolved.
Show resolved
Hide resolved
|
||
be terminated and deleted. | ||
|
||
## Example Pod | ||
|
||
Here is a sample Pod that just sleep for a few seconds: | ||
achernevskii marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
||
{{< include "examples/pods-kueue/kueue-pod.yaml" "yaml" >}} | ||
|
||
achernevskii marked this conversation as resolved.
Show resolved
Hide resolved
|
||
You can create the Pod using the following command: | ||
```sh | ||
# Create the pod | ||
kubectl apply -f kueue-pod.yaml | ||
``` |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,18 @@ | ||
apiVersion: v1 | ||
kind: Pod | ||
metadata: | ||
generateName: kueue-sleep- | ||
labels: | ||
kueue.x-k8s.io/queue-name: user-queue | ||
spec: | ||
containers: | ||
- name: sleep | ||
image: busybox | ||
command: | ||
- sleep | ||
args: | ||
- 3s | ||
resources: | ||
requests: | ||
cpu: 3 | ||
restartPolicy: OnFailure |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Add the top level fields, like kind
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should I create a minimal working configuration, or just add some of the top level fields?
Should we do the same thing for the following pages?
run_mpijobs.md
run_jobsets.md
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
just the top level fields, to have a good context of where in the configuration
integrations
fit.Yes, we should probably do the same for others. You can do that in a follow up.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Updated example configuration in adfe10d