[WIP] Two step admission deocumentation

site/content/en/docs/concepts/_index.md

@@ -34,18 +34,31 @@ single tenant.
 An application that will run to completion. It is the unit of _admission_ in
 Kueue. Sometimes referred to as _job_.
 
+### [Admission Check](/docs/concepts/admission_check)
+
+A mechanism allowing custom internal or external components to control the timing of
+workloads admission.
+
+### [Preemption Admission Check Controller](/docs/concepts/preemption_controller)
+
+A builtin admission check controller managing the eviction of preemption candidates 
+and reevaluate the workloads quota reservation. 
+
 ![Components](/images/queueing-components.svg)
 
 ## Glossary
 
-### Admission
-
-The process of admitting a Workload to start (Pods to be created). A Workload
-is admitted by a ClusterQueue according to the available resources and gets
-resource flavors assigned for each requested resource.
+### Quota Reservation
 
 Sometimes referred to as _workload scheduling_ or _job scheduling_
 (not to be confused with [pod scheduling](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/)).
+Is the process during which the kueue scheduler locks the resources needed by a workload within the targeted [ClusterQueues ResourceGroups](/docs/concepts/cluster_queue/#resource-groups)
+
+### Admission
+
+The process of admitting a Workload to start (Pods to be created). A Workload
+is admitted when it has a Quota Reservation and all it's potential [AdmissionCheckStates](/docs/concepts/admission_check)
+are `Ready`.
 
 ### [Cohort](/docs/concepts/cluster_queue#cohort)
 

site/content/en/docs/concepts/admission_check.md

@@ -0,0 +1,126 @@
+---
+title: "Admission Check"
+date: 2023-10-05
+weight: 6
+description: >
+  A mechanism allowing custom internal or external components to control the timing of
+  workloads admission.
+---
+
+## Components
+
+### Admission Check
+
+Admission check is a none namespaced API object used to define details about an Admission Check like:
+
+#### controllerName 
+
+Is the name with which controller identifies with, not necessarily a K8S Pod or Deployment name. Cannot be empty.
+
+#### retryDelayMinutes 
+
+Specifies how long to keep the workload suspended after a failed check (after it transitioned to False). After that the check state goes to "Unknown". The default is 15 min.
+
+#### parameters 
+
+Identifies the resource providing additional check parameters.
+
+#### preemptionPolicy 
+
+Determines when to issue preemptions for the Workload, if necessary, in relationship to the status of the admission check.
+The possible values are:
+- `Anytime`: No need to wait for this check to pass before issuing preemptions.
+  Preemptions might be blocked on the preemptionPolicy of other AdmissionChecks.
+- `AfterCheckPassedOrOnDemand`: Wait for this check to pass before issuing preemptions,
+  unless this or other checks requests preemptions through the Workload's admissionChecks.
+Defaults to `Anytime`.
+
+```yaml
+apiVersion: kueue.x-k8s.io/v1beta1
+kind: AdmissionCheck
+metadata:
+  name: prov-test
+spec:
+  controllerName: ProvisioningRequestController
+  retryDelayMinutes: 15
+  parameters:
+    apiGroup: kueue.x-k8s.io
+    kind: ProvisioningRequestConfig
+    name: prov-test-config
+  preemptionPolicy: Anytime
+```
+
+### ClusterQueue admissionChecks
+
+Once defined an AdmissionCheck can be referenced in the ClsterQueues Spec resulting in all Workloads associated with the queue needing to be evaluated by its controller before being admitted.
+
+### AdmissionCheckState
+
+AdmissionCheckState is the way the way the state of an AdmissionCkeck for a specific Workload is tracked.
+
+It is following the schema:
+```yaml
+properties:
+  lastTransitionTime:
+    description: lastTransitionTime is the last time the condition
+      transitioned from one status to another. This should be when
+      the underlying condition changed.  If that is not known, then
+      using the time when the API field changed is acceptable.
+    format: date-time
+    type: string
+  message:
+    description: message is a human readable message indicating
+      details about the transition. This may be an empty string.
+    maxLength: 32768
+    type: string
+  name:
+    description: name identifies the admission check.
+    maxLength: 316
+    type: string
+  podSetUpdates:
+    items:
+      description: PodSetUpdate contains a list of pod set modifications
+        suggested by AdmissionChecks. The modifications should be
+        additive only - modifications of already existing keys or
+        having the same key provided by multiple AdmissionChecks
+        is not allowed and will result in failure during workload
+        admission.
+  <...>
+      type: object
+    type: array
+    x-kubernetes-list-type: atomic
+  state:
+    description: status of the condition, one of True, False, Unknown.
+    enum:
+    - Pending
+    - Ready
+    - Retry
+    - Rejected
+    - PreemptionRequired
+    type: string
+required:
+- lastTransitionTime
+- message
+- name
+- state
+type: object
+```
+
+A list of states being maintained in the Status of all the monitored Workloads.
+
+Kueue ensurers that the list of the Workloads AdmissionCheckStates is in sync with the list of its associated ClusterQueue, new checks being added with the `Pending` state.
+
+- Once a workload has QuotaReservation and all its AdmissionChecks are in "Ready" state it will become Admitted.
+- If at least one of the Workloads AdmissionCheck is in the `Retry` state.
+  - If `Admitted` the workload is evicted.
+  - If the workload has `QuotaReservation` it will be release released.
+- If at least one of the Workloads AdmissionCheck is in the `Rejected`:
+  - If `Admitted` the workload is evicted.
+  - If the workload has `QuotaReservation` it will be release released.
+  - The workload is marked as 'Finished' with a relevant failure message.
+- `PreemptionRequired` is a state specific to PreemptionPolicies, from an Admission point of view being synonym to `Pending`
+
+### Admission Check Controller
+
+Is a component that monitors Workloads and maintains the content of its specific `AdmissionCheckStates`.
+The logic based on which an `AdmissionCheck` changes states in not in the scope of Kueue.

site/content/en/docs/concepts/cluster_queue.md

@@ -304,8 +304,10 @@ borrow `12+9` CPUs.
 ## Preemption
 
 When there is not enough quota left in a ClusterQueue or its cohort, an incoming
-Workload can trigger preemption of previously admitted Workloads, based on
-policies for the ClusterQueue.
+Workload can still get a Quota Reservation, based on policies for the ClusterQueue
+and the workloads priority. Following this the builtin 
+[Preemption Admission Check Controller]() will manage the eviction of its Preemption
+targets and admission check state.
 
 A configuration for a ClusterQueue that enables preemption looks like the
 following:

site/content/en/docs/concepts/preemption_controller.md

@@ -0,0 +1,23 @@
+---
+title: "Preemption Admission Check Controller"
+date: 2023-10-05
+weight: 7
+description: >
+  A builtin admission check controller managing the eviction of preemption candidates 
+  and reevaluate the workloads quota reservation. 
+---
+
+As described in [ClusteQueue Preemption](/docs/concepts/cluster_queue/#preemption) "When there is not enough quota left in a ClusterQueue or its cohort, an incoming Workload can still get a `QuotaReservation`, based on policies for the ClusterQueue and the workloads priority." case in which the Workload will have an additional admission check set `kueue-preemption`, this admission is manage by the builtin Preemption Admission Check Controller. 
+
+
+### Preemption Timing
+
+If any the [Workloads](/docs/concepts/admission_check) [AdmissionChecks](/docs/concepts/admission_check) is using `AfterCheckPassedOrOnDemand` preemption policy the preemption process for that Workload is postponed until its state becomes `Ready` or `Rejected`, this is done to avoid the premature eviction of the preemption candidates in case the [Admission Check Controller](/docs/concepts/admission_check/#admission-check-controller) is aware that the Workload cannot be Admitted immediately.
+
+### Preemption Process
+
+During the Preemption Process the Preemption Admission Check Controller will periodically:
+
+- Check if the preempting Workload can fit without evicting other workloads, case in which the preemption admission check condition will be set to `Ready`.
+- If eviction of other workload is still needed, an updated list candidates is created and eviction is issued for all of them.
+- If the updated list of candidates is empty, meaning that the preemption can no longer succeed, the preemption admission check is set as `Retry`, the workload will lose it's quota reservation and be requeued.

site/static/examples/sample-admission-check.yaml

@@ -0,0 +1,22 @@
+apiVersion: kueue.x-k8s.io/v1beta1
+kind: AdmissionCheck
+metadata:
+  name: prov-test
+spec:
+  controllerName: ProvisioningRequestController
+  parameters:
+    apiGroup: kueue.x-k8s.io
+    kind: ProvisioningRequestConfig
+    name: prov-test-config
+
+---
+
+
+apiVersion: kueue.x-k8s.io/v1beta1
+kind: ProvisioningRequestConfig
+metadata:
+  name: prov-test-config
+spec:
+  provisioningClassName: test-class
+  parameters:
+    parma1: value1