diff --git a/site/content/en/docs/concepts/_index.md b/site/content/en/docs/concepts/_index.md index 68e5bf79aaf..d3107808f29 100644 --- a/site/content/en/docs/concepts/_index.md +++ b/site/content/en/docs/concepts/_index.md @@ -34,18 +34,31 @@ single tenant. An application that will run to completion. It is the unit of _admission_ in Kueue. Sometimes referred to as _job_. +### [Admission Check](/docs/concepts/admission_check) + +A mechanism allowing custom internal or external components to control the timing of +workloads admission. + +### [Preemption Admission Check Controller](/docs/concepts/preemption_controller) + +A builtin admission check controller managing the eviction of preemption candidates +and reevaluate the workloads quota reservation. + ![Components](/images/queueing-components.svg) ## Glossary -### Admission - -The process of admitting a Workload to start (Pods to be created). A Workload -is admitted by a ClusterQueue according to the available resources and gets -resource flavors assigned for each requested resource. +### Quota Reservation Sometimes referred to as _workload scheduling_ or _job scheduling_ (not to be confused with [pod scheduling](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/)). +Is the process during which the kueue scheduler locks the resources needed by a workload within the targeted [ClusterQueues ResourceGroups](/docs/concepts/cluster_queue/#resource-groups) + +### Admission + +The process of admitting a Workload to start (Pods to be created). A Workload +is admitted when it has a Quota Reservation and all it's potential [AdmissionCheckStates](/docs/concepts/admission_check) +are `Ready`. ### [Cohort](/docs/concepts/cluster_queue#cohort) diff --git a/site/content/en/docs/concepts/admission_check.md b/site/content/en/docs/concepts/admission_check.md new file mode 100644 index 00000000000..673bc4e6e70 --- /dev/null +++ b/site/content/en/docs/concepts/admission_check.md @@ -0,0 +1,126 @@ +--- +title: "Admission Check" +date: 2023-10-05 +weight: 6 +description: > + A mechanism allowing custom internal or external components to control the timing of + workloads admission. +--- + +## Components + +### Admission Check + +Admission check is a none namespaced API object used to define details about an Admission Check like: + +#### controllerName + +Is the name with which controller identifies with, not necessarily a K8S Pod or Deployment name. Cannot be empty. + +#### retryDelayMinutes + +Specifies how long to keep the workload suspended after a failed check (after it transitioned to False). After that the check state goes to "Unknown". The default is 15 min. + +#### parameters + +Identifies the resource providing additional check parameters. + +#### preemptionPolicy + +Determines when to issue preemptions for the Workload, if necessary, in relationship to the status of the admission check. +The possible values are: +- `Anytime`: No need to wait for this check to pass before issuing preemptions. + Preemptions might be blocked on the preemptionPolicy of other AdmissionChecks. +- `AfterCheckPassedOrOnDemand`: Wait for this check to pass before issuing preemptions, + unless this or other checks requests preemptions through the Workload's admissionChecks. +Defaults to `Anytime`. + +```yaml +apiVersion: kueue.x-k8s.io/v1beta1 +kind: AdmissionCheck +metadata: + name: prov-test +spec: + controllerName: ProvisioningRequestController + retryDelayMinutes: 15 + parameters: + apiGroup: kueue.x-k8s.io + kind: ProvisioningRequestConfig + name: prov-test-config + preemptionPolicy: Anytime +``` + +### ClusterQueue admissionChecks + +Once defined an AdmissionCheck can be referenced in the ClsterQueues Spec resulting in all Workloads associated with the queue needing to be evaluated by its controller before being admitted. + +### AdmissionCheckState + +AdmissionCheckState is the way the way the state of an AdmissionCkeck for a specific Workload is tracked. + +It is following the schema: +```yaml +properties: + lastTransitionTime: + description: lastTransitionTime is the last time the condition + transitioned from one status to another. This should be when + the underlying condition changed. If that is not known, then + using the time when the API field changed is acceptable. + format: date-time + type: string + message: + description: message is a human readable message indicating + details about the transition. This may be an empty string. + maxLength: 32768 + type: string + name: + description: name identifies the admission check. + maxLength: 316 + type: string + podSetUpdates: + items: + description: PodSetUpdate contains a list of pod set modifications + suggested by AdmissionChecks. The modifications should be + additive only - modifications of already existing keys or + having the same key provided by multiple AdmissionChecks + is not allowed and will result in failure during workload + admission. + <...> + type: object + type: array + x-kubernetes-list-type: atomic + state: + description: status of the condition, one of True, False, Unknown. + enum: + - Pending + - Ready + - Retry + - Rejected + - PreemptionRequired + type: string +required: +- lastTransitionTime +- message +- name +- state +type: object +``` + +A list of states being maintained in the Status of all the monitored Workloads. + +Kueue ensurers that the list of the Workloads AdmissionCheckStates is in sync with the list of its associated ClusterQueue, new checks being added with the `Pending` state. + +- Once a workload has QuotaReservation and all its AdmissionChecks are in "Ready" state it will become Admitted. +- If at least one of the Workloads AdmissionCheck is in the `Retry` state. + - If `Admitted` the workload is evicted. + - If the workload has `QuotaReservation` it will be release released. +- If at least one of the Workloads AdmissionCheck is in the `Rejected`: + - If `Admitted` the workload is evicted. + - If the workload has `QuotaReservation` it will be release released. + - The workload is marked as 'Finished' with a relevant failure message. +- `PreemptionRequired` is a state specific to PreemptionPolicies, from an Admission point of view being synonym to `Pending` + +### Admission Check Controller + +Is a component that monitors Workloads and maintains the content of its specific `AdmissionCheckStates`. +The logic based on which an `AdmissionCheck` changes states in not in the scope of Kueue. diff --git a/site/content/en/docs/concepts/cluster_queue.md b/site/content/en/docs/concepts/cluster_queue.md index 3537c1e81ee..4e6f022898a 100644 --- a/site/content/en/docs/concepts/cluster_queue.md +++ b/site/content/en/docs/concepts/cluster_queue.md @@ -304,8 +304,10 @@ borrow `12+9` CPUs. ## Preemption When there is not enough quota left in a ClusterQueue or its cohort, an incoming -Workload can trigger preemption of previously admitted Workloads, based on -policies for the ClusterQueue. +Workload can still get a Quota Reservation, based on policies for the ClusterQueue +and the workloads priority. Following this the builtin +[Preemption Admission Check Controller]() will manage the eviction of its Preemption +targets and admission check state. A configuration for a ClusterQueue that enables preemption looks like the following: diff --git a/site/content/en/docs/concepts/preemption_controller.md b/site/content/en/docs/concepts/preemption_controller.md new file mode 100644 index 00000000000..6a31b2fc385 --- /dev/null +++ b/site/content/en/docs/concepts/preemption_controller.md @@ -0,0 +1,23 @@ +--- +title: "Preemption Admission Check Controller" +date: 2023-10-05 +weight: 7 +description: > + A builtin admission check controller managing the eviction of preemption candidates + and reevaluate the workloads quota reservation. +--- + +As described in [ClusteQueue Preemption](/docs/concepts/cluster_queue/#preemption) "When there is not enough quota left in a ClusterQueue or its cohort, an incoming Workload can still get a `QuotaReservation`, based on policies for the ClusterQueue and the workloads priority." case in which the Workload will have an additional admission check set `kueue-preemption`, this admission is manage by the builtin Preemption Admission Check Controller. + + +### Preemption Timing + +If any the [Workloads](/docs/concepts/admission_check) [AdmissionChecks](/docs/concepts/admission_check) is using `AfterCheckPassedOrOnDemand` preemption policy the preemption process for that Workload is postponed until its state becomes `Ready` or `Rejected`, this is done to avoid the premature eviction of the preemption candidates in case the [Admission Check Controller](/docs/concepts/admission_check/#admission-check-controller) is aware that the Workload cannot be Admitted immediately. + +### Preemption Process + +During the Preemption Process the Preemption Admission Check Controller will periodically: + +- Check if the preempting Workload can fit without evicting other workloads, case in which the preemption admission check condition will be set to `Ready`. +- If eviction of other workload is still needed, an updated list candidates is created and eviction is issued for all of them. +- If the updated list of candidates is empty, meaning that the preemption can no longer succeed, the preemption admission check is set as `Retry`, the workload will lose it's quota reservation and be requeued. diff --git a/site/static/examples/sample-admission-check.yaml b/site/static/examples/sample-admission-check.yaml new file mode 100644 index 00000000000..f9588e0ed28 --- /dev/null +++ b/site/static/examples/sample-admission-check.yaml @@ -0,0 +1,22 @@ +apiVersion: kueue.x-k8s.io/v1beta1 +kind: AdmissionCheck +metadata: + name: prov-test +spec: + controllerName: ProvisioningRequestController + parameters: + apiGroup: kueue.x-k8s.io + kind: ProvisioningRequestConfig + name: prov-test-config + +--- + + +apiVersion: kueue.x-k8s.io/v1beta1 +kind: ProvisioningRequestConfig +metadata: + name: prov-test-config +spec: + provisioningClassName: test-class + parameters: + parma1: value1