feature:add preemptionpolicy in preempt and reclaim

[JesseStutler]

ref: #3642 ,#3738

[JesseStutler]

/assign @hwdef @Monokaix @lowang-bh Please take a look~

[JesseStutler]

Verification

test-1

Create two jobs, job-a and job-b, where the PriorityClass specified by job-b has a higher priority than the PriorityClass specified by job-a, but the PriorityClass specified by job-b comes with PreemptionPolicy: Never.

apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: high-priority
value: 200
preemptionPolicy: Never

The Pod created by job-b does not specify a PriorityClassName, so the PriorityClass specified by job is used.：

The Pod created by job-b fails to preempt because PreemptionPolicy is set to true:：

test-2

Create two jobs, job-a and job-c, job-c has a specified PriorityClass without the PreemptionPolicy field, but the task has a different PriorityClass with the PreemptionPolicy field specified, and job-a remains unchanged.

job-c yaml:

apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: high-priority
value: 200
---
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: cover
value: 200
preemptionPolicy: Never
---
apiVersion: batch.volcano.sh/v1alpha1
kind: Job
metadata:
  name: job-c
spec:
  schedulerName: volcano
  minAvailable: 4
  priorityClassName: high-priority
  tasks:
    - replicas: 4
      name: "test"
      template:
        spec:
          priorityClassName: cover
          containers:
            - image: alpine
              command: ["/bin/sh", "-c", "sleep 6000"]
              imagePullPolicy: IfNotPresent
              name: running
              resources:
                requests:
                  cpu: 3900m
          restartPolicy: OnFailure

The PriorityClassName specified by job-c's pod is the task template's

Preemption failure log:

test-3

Submit two jobs, one job-a and one job-b, in two separate queues, first submit job-a to fill up the cluster resources, then submit job-b, job-a needs to reclaim half of the resources to job-b, but because the PriorityClass specified by job-b specifies PreemptionPolicy as Never, job-a cannot reclaim job-a's resources, so job-b's pod all are pending.

[hwdef]

/lgtm

[JesseStutler]

/cc @Monokaix

[hwdef]

/lgtm

[Monokaix]

better wrap a func here.

[Monokaix]

Is there any difference with kube-scheduler, such as is there event reported to inform user.

[JesseStutler]

no, it only record the log, the codes are:

if ok, msg := ev.PodEligibleToPreemptOthers(ctx, pod, nominatedNodeStatus); !ok {
		logger.V(5).Info("Pod is not eligible for preemption", "pod", klog.KObj(pod), "reason", msg)
		return nil, framework.NewStatus(framework.Unschedulable, msg)
}

func (pl *DefaultPreemption) PodEligibleToPreemptOthers(_ context.Context, pod *v1.Pod, nominatedNodeStatus *framework.Status) (bool, string) {
	if pod.Spec.PreemptionPolicy != nil && *pod.Spec.PreemptionPolicy == v1.PreemptNever {
		return false, "not eligible due to preemptionPolicy=Never."
	}

	nodeInfos := pl.fh.SnapshotSharedLister().NodeInfos()
	nomNodeName := pod.Status.NominatedNodeName
	if len(nomNodeName) > 0 {
		// If the pod's nominated node is considered as UnschedulableAndUnresolvable by the filters,
		// then the pod should be considered for preempting again.
		if nominatedNodeStatus.Code() == framework.UnschedulableAndUnresolvable {
			return true, ""
		}

		if nodeInfo, _ := nodeInfos.Get(nomNodeName); nodeInfo != nil {
			podPriority := corev1helpers.PodPriority(pod)
			for _, p := range nodeInfo.Pods {
				if corev1helpers.PodPriority(p.Pod) < podPriority && podTerminatingByPreemption(p.Pod) {
					// There is a terminating pod on the nominated node.
					return false, "not eligible due to a terminating pod on the nominated node."
				}
			}
		}
	}
	return true, ""
}

[Monokaix]

/ok-to-test

[Monokaix]

/lgtm
/approve

[volcano-sh-bot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: Monokaix

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [Monokaix]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment