Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Renamed requestableResources to resources in clusterQueue #206

Merged
merged 1 commit into from
Apr 11, 2022
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 5 additions & 5 deletions apis/core/v1alpha1/clusterqueue_types.go
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ import (

// ClusterQueueSpec defines the desired state of ClusterQueue
type ClusterQueueSpec struct {
// requestableResources represent the total pod requests of workloads dispatched
// resources represent the total pod requests of workloads dispatched
// via this clusterQueue. This doesn’t guarantee the actual availability of
// resources, although an integration with a resource provisioner like Cluster
// Autoscaler is possible to achieve that. Example:
Expand All @@ -40,7 +40,7 @@ type ClusterQueueSpec struct {
//
// +listType=map
// +listMapKey=name
RequestableResources []Resource `json:"requestableResources,omitempty"`
Resources []Resource `json:"resources,omitempty"`

// cohort that this ClusterQueue belongs to. QCs that belong to the
// same cohort can borrow unused resources from each other.
Expand Down Expand Up @@ -69,7 +69,7 @@ type ClusterQueueSpec struct {
// name: tenantA
// spec:
// cohort: borrowing-cohort
// requestableResources:
// resources:
// - name: cpu
// - name: spot
// quota:
Expand All @@ -95,7 +95,7 @@ type ClusterQueueSpec struct {
// name: tenantB
// spec:
// cohort: borrowing-cohort
// requestableResources:
// resources:
// - name: cpu
// - name: on-demand
// quota:
Expand Down Expand Up @@ -166,7 +166,7 @@ type Resource struct {
// flavor and must set different values of a shared key. For example:
//
// spec:
// requestableResources:
// resources:
// - name: nvidia.com/gpus
// - name: k80
// quota:
Expand Down
4 changes: 2 additions & 2 deletions apis/core/v1alpha1/zz_generated.deepcopy.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

46 changes: 23 additions & 23 deletions config/crd/bases/kueue.x-k8s.io_clusterqueues.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -75,18 +75,18 @@ spec:
will only be eligible to consume on-demand cores (the next in the
list of cpu flavors). 5. Before considering on-demand, the workload
will get assigned spot if the quota can be borrowed from the cohort.
\n metadata: name: tenantA spec: cohort: borrowing-cohort requestableResources:
\n metadata: name: tenantA spec: cohort: borrowing-cohort resources:
- name: cpu - name: spot quota: min: 1000 - name: on-demand quota:
min: 100 - name: nvidia.com/gpus - name: k80 quota: min: 10 max:
20 labels: - cloud.provider.com/accelerator: nvidia-tesla-k80 -
name: p100 quota: min: 10 max: 20 labels: - cloud.provider.com/accelerator:
nvidia-tesla-p100 \n metadata: name: tenantB spec: cohort: borrowing-cohort
requestableResources: - name: cpu - name: on-demand quota: min:
100 - name: nvidia.com/gpus - name: k80 quota: min: 10 max: 20 labels:
- cloud.provider.com/accelerator: nvidia-tesla-k80 \n If empty,
this ClusterQueue cannot borrow from any other ClusterQueue and
vice versa. \n The name style is similar to label keys. These are
just names to link QCs together, and they are meaningless otherwise."
resources: - name: cpu - name: on-demand quota: min: 100 - name:
nvidia.com/gpus - name: k80 quota: min: 10 max: 20 labels: - cloud.provider.com/accelerator:
nvidia-tesla-k80 \n If empty, this ClusterQueue cannot borrow from
any other ClusterQueue and vice versa. \n The name style is similar
to label keys. These are just names to link QCs together, and they
are meaningless otherwise."
type: string
namespaceSelector:
description: namespaceSelector defines which namespaces are allowed
Expand Down Expand Up @@ -151,13 +151,13 @@ spec:
- StrictFIFO
- BestEffortFIFO
type: string
requestableResources:
description: "requestableResources represent the total pod requests
of workloads dispatched via this clusterQueue. This doesn’t guarantee
the actual availability of resources, although an integration with
a resource provisioner like Cluster Autoscaler is possible to achieve
that. Example: \n - name: cpu flavors: - quota: min: 100 - name:
memory flavors: - quota: min: 100Gi"
resources:
description: "resources represent the total pod requests of workloads
dispatched via this clusterQueue. This doesn’t guarantee the actual
availability of resources, although an integration with a resource
provisioner like Cluster Autoscaler is possible to achieve that.
Example: \n - name: cpu flavors: - quota: min: 100 - name: memory
flavors: - quota: min: 100Gi"
items:
properties:
flavors:
Expand All @@ -169,15 +169,15 @@ spec:
\n For example, if the resource is nvidia.com/gpu, and we
want to define different limits for different gpu models,
then each model is mapped to a flavor and must set different
values of a shared key. For example: \n spec: requestableResources:
- name: nvidia.com/gpus - name: k80 quota: min: 10 - name:
p100 quota: min: 10 \n The flavors are evaluated in order,
selecting the first to satisfy a workload’s requirements.
Also the quantities are additive, in the example above the
GPU quota in total is 20 (10 k80 + 10 p100). A workload is
limited to the selected type by converting the labels to a
node selector that gets injected into the workload. This list
can’t be empty, at least one flavor must exist."
values of a shared key. For example: \n spec: resources: -
name: nvidia.com/gpus - name: k80 quota: min: 10 - name: p100
quota: min: 10 \n The flavors are evaluated in order, selecting
the first to satisfy a workload’s requirements. Also the quantities
are additive, in the example above the GPU quota in total
is 20 (10 k80 + 10 p100). A workload is limited to the selected
type by converting the labels to a node selector that gets
injected into the workload. This list can’t be empty, at least
one flavor must exist."
items:
properties:
name:
Expand Down
6 changes: 3 additions & 3 deletions config/samples/single-clusterqueue-setup.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -9,15 +9,15 @@ metadata:
name: cluster-total
spec:
namespaceSelector: {}
requestableResources:
resources:
- name: "cpu"
flavors:
- resourceFlavor: default
- name: default
quota:
min: 9
- name: "memory"
flavors:
- resourceFlavor: default
- name: default
quota:
min: 36Gi
---
Expand Down
14 changes: 7 additions & 7 deletions docs/concepts/cluster_queue.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ metadata:
name: cluster-total
spec:
namespaceSelector: {}
requestableResources:
resources:
- name: "cpu"
flavors:
- name: default
Expand Down Expand Up @@ -103,10 +103,10 @@ taints:
```

You can use the `.metadata.name` to reference a flavor from a ClusterQueue in
the `.spec.requestableResources[*].flavors[*].name` field.
the `.spec.resources[*].flavors[*].name` field.

For each resource of each [pod set](workload.md#pod-sets) in a Workload, Kueue
assigns the first flavor in the `.spec.requestableResources[*].resources.flavors`
assigns the first flavor in the `.spec.resources[*].flavors`
list that has enough unused quota in the ClusterQueue or the ClusterQueue's
[cohort](#cohort).

Expand Down Expand Up @@ -176,7 +176,7 @@ ClusterQueue.
When borrowing, Kueue satisfies the following semantics:

- When assigning flavors, Kueue goes through the list of flavors in
`.spec.requestableResources[*].flavors`. For each flavor, Kueue attempts to
`.spec.resources[*].flavors`. For each flavor, Kueue attempts to
fit the workload using the min quota of the ClusterQueue or the unused
min quota of other ClusterQueues in the cohort, up to the max quota of the
ClusterQueue. If the workload doesn't fit, Kueue proceeds evaluating the next
Expand All @@ -196,7 +196,7 @@ metadata:
spec:
namespaceSelector: {}
cohort: team-ab
requestableResources:
resources:
- name: "cpu"
flavors:
- name: default
Expand All @@ -217,7 +217,7 @@ metadata:
spec:
namespaceSelector: {}
cohort: team-ab
requestableResources:
resources:
- name: "cpu"
flavors:
- name: default
Expand Down Expand Up @@ -247,7 +247,7 @@ No admitted workloads will be stopped to make space for new workloads.
### Max quotas

To limit the amount of resources that a ClusterQueue can borrow from others,
you can set the `.spec.requestableResources[*].flavors[*].quota.max`
you can set the `.spec.resources[*].flavors[*].quota.max`
[quantity](https://kubernetes.io/docs/reference/kubernetes-api/common-definitions/quantity/) field.
`max` must be greater than or equal to `min`.

Expand Down
20 changes: 10 additions & 10 deletions docs/tasks/administer_cluster_quotas.md
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@ metadata:
name: cluster-total
spec:
namespaceSelector: {} # match all.
requestableResources:
resources:
- name: "cpu"
flavors:
- name: default
Expand Down Expand Up @@ -90,7 +90,7 @@ To create the ResourceFlavor, run the following command:
kubectl apply -f default-flavor.yaml
```

The `.metadata.name` matches the `.spec.requestableResources[*].flavors[0].resourceFlavor`
The `.metadata.name` matches the `.spec.resources[*].flavors[0].resourceFlavor`
field in the ClusterQueue.

### 3. Create [Queues](/docs/concepts/queue.md)
Expand Down Expand Up @@ -129,7 +129,7 @@ architectures, namely `x86` and `arm`, specified in the node label `cpu-arch`.

**Limitations**

- Using the same flavors in multiple `.requestableResources` of a ClusterQueue
- Using the same flavors in multiple `.resources` of a ClusterQueue
is [not supported](https://github.com/kubernetes-sigs/kueue/issues/167).

### 1. Create ResourceFlavors
Expand Down Expand Up @@ -181,7 +181,7 @@ metadata:
name: cluster-total
spec:
namespaceSelector: {}
requestableResources:
resources:
- name: "cpu"
flavors:
- name: x86
Expand All @@ -197,7 +197,7 @@ spec:
min: 84Gi
```

The flavor names in the fields `.spec.requestableResources[*].flavors[*].resourceFlavor`
The flavor names in the fields `.spec.resources[*].flavors[*].resourceFlavor`
should match the names of the ResourceFlavors created earlier.

Note that `memory` is referencing the `default` flavor created in the [single flavor setup](#single-clusterqueue-and-single-resourceflavor-setup)
Expand Down Expand Up @@ -227,7 +227,7 @@ metadata:
spec:
namespaceSelector: {}
cohort: team-ab
requestableResources:
resources:
- name: "cpu"
flavors:
- name: default
Expand All @@ -251,7 +251,7 @@ metadata:
spec:
namespaceSelector: {}
cohort: team-ab
requestableResources:
resources:
- name: "cpu"
flavors:
- name: default
Expand Down Expand Up @@ -295,7 +295,7 @@ metadata:
spec:
namespaceSelector: {}
cohort: team-ab
requestableResources:
resources:
- name: "cpu"
flavors:
- name: arm
Expand All @@ -321,7 +321,7 @@ metadata:
spec:
namespaceSelector: {}
cohort: team-ab
requestableResources:
resources:
- name: "cpu"
flavors:
- name: arm
Expand All @@ -347,7 +347,7 @@ metadata:
spec:
namespaceSelector: {}
cohort: team-ab
requestableResources:
resources:
- name: "cpu"
flavors:
- name: x86
Expand Down
6 changes: 3 additions & 3 deletions pkg/cache/cache.go
Original file line number Diff line number Diff line change
Expand Up @@ -115,15 +115,15 @@ func (c *Cache) newClusterQueue(cq *kueue.ClusterQueue) (*ClusterQueue, error) {
}

func (c *ClusterQueue) update(in *kueue.ClusterQueue, resourceFlavors map[string]*kueue.ResourceFlavor) error {
c.RequestableResources = resourceLimitsByName(in.Spec.RequestableResources)
c.RequestableResources = resourceLimitsByName(in.Spec.Resources)
nsSelector, err := metav1.LabelSelectorAsSelector(in.Spec.NamespaceSelector)
if err != nil {
return err
}
c.NamespaceSelector = nsSelector

usedResources := make(Resources, len(in.Spec.RequestableResources))
for _, r := range in.Spec.RequestableResources {
usedResources := make(Resources, len(in.Spec.Resources))
for _, r := range in.Spec.Resources {
if len(r.Flavors) == 0 {
continue
}
Expand Down
12 changes: 6 additions & 6 deletions pkg/cache/cache_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,7 @@ func TestCacheClusterQueueOperations(t *testing.T) {
{
ObjectMeta: metav1.ObjectMeta{Name: "a"},
Spec: kueue.ClusterQueueSpec{
RequestableResources: []kueue.Resource{
Resources: []kueue.Resource{
{
Name: corev1.ResourceCPU,
Flavors: []kueue.Flavor{{
Expand All @@ -60,7 +60,7 @@ func TestCacheClusterQueueOperations(t *testing.T) {
{
ObjectMeta: metav1.ObjectMeta{Name: "b"},
Spec: kueue.ClusterQueueSpec{
RequestableResources: []kueue.Resource{
Resources: []kueue.Resource{
{
Name: corev1.ResourceCPU,
Flavors: []kueue.Flavor{{
Expand Down Expand Up @@ -203,7 +203,7 @@ func TestCacheClusterQueueOperations(t *testing.T) {
{
ObjectMeta: metav1.ObjectMeta{Name: "a"},
Spec: kueue.ClusterQueueSpec{
RequestableResources: []kueue.Resource{
Resources: []kueue.Resource{
{
Name: corev1.ResourceCPU,
Flavors: []kueue.Flavor{
Expand Down Expand Up @@ -336,7 +336,7 @@ func TestCacheWorkloadOperations(t *testing.T) {
{
ObjectMeta: metav1.ObjectMeta{Name: "one"},
Spec: kueue.ClusterQueueSpec{
RequestableResources: []kueue.Resource{
Resources: []kueue.Resource{
{
Name: "cpu",
Flavors: []kueue.Flavor{
Expand All @@ -350,7 +350,7 @@ func TestCacheWorkloadOperations(t *testing.T) {
{
ObjectMeta: metav1.ObjectMeta{Name: "two"},
Spec: kueue.ClusterQueueSpec{
RequestableResources: []kueue.Resource{
Resources: []kueue.Resource{
{
Name: "cpu",
Flavors: []kueue.Flavor{
Expand Down Expand Up @@ -796,7 +796,7 @@ func TestClusterQueueUsage(t *testing.T) {
cq := kueue.ClusterQueue{
ObjectMeta: metav1.ObjectMeta{Name: "foo"},
Spec: kueue.ClusterQueueSpec{
RequestableResources: []kueue.Resource{
Resources: []kueue.Resource{
{
Name: corev1.ResourceCPU,
Flavors: []kueue.Flavor{
Expand Down
6 changes: 3 additions & 3 deletions pkg/cache/snapshot_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -48,7 +48,7 @@ func TestSnapshot(t *testing.T) {
},
Spec: kueue.ClusterQueueSpec{
Cohort: "foo",
RequestableResources: []kueue.Resource{
Resources: []kueue.Resource{
{
Name: corev1.ResourceCPU,
Flavors: []kueue.Flavor{
Expand All @@ -75,7 +75,7 @@ func TestSnapshot(t *testing.T) {
},
Spec: kueue.ClusterQueueSpec{
Cohort: "foo",
RequestableResources: []kueue.Resource{
Resources: []kueue.Resource{
{
Name: corev1.ResourceCPU,
Flavors: []kueue.Flavor{
Expand Down Expand Up @@ -106,7 +106,7 @@ func TestSnapshot(t *testing.T) {
Name: "bar",
},
Spec: kueue.ClusterQueueSpec{
RequestableResources: []kueue.Resource{
Resources: []kueue.Resource{
{
Name: corev1.ResourceCPU,
Flavors: []kueue.Flavor{
Expand Down
Loading