Fix Issue 2262: add priority capability for reclaim action

[lowang-bh]

Fixes #2262

1. priority plugin support reclaimable switch and add UT about priority plugin
2. priority plugin docs

Note: reclaimableFn usually is used in reclaim action to reclaim a queue's deserved resource when cluster has not enough resource to allocate new coming tasks in this queue. So please be carefully to set enableReclaimable to true in priority plugin, in case that a queue's resource owned by high priority jobs can not be released. And enableReclaimable is disabled by default for compatibility

[lowang-bh]

/assign @Monokaix @william-wang

[Monokaix]

If other queue's jobs all have higher priority，current queue can not reclaim their resources, reclaim will not happen, is this reasonable？

[lowang-bh]

If other queue's jobs all have higher priority，current queue can not reclaim their resources, reclaim will not happen, is this reasonable？

We just support this feature and close it by default。 How to use it is depended on the cluster admin.

One solutions is to limit the higher priority jobs used resource not exceed queue's deserved in application layer.

[Monokaix]

If other queue's jobs all have higher priority，current queue can not reclaim their resources, reclaim will not happen, is this reasonable？

We just support this feature and close it by default。 How to use it is depended on the cluster admin.

One solutions is to limit the higher priority jobs used resource not exceed queue's deserved in application layer.

But this seems places high demands on administrators and limits the job priority of the queue, we'd better add a desige doc and give some user-guide.

[volcano-sh-bot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
To complete the pull request process, please assign william-wang
You can assign the PR to them by writing /assign @william-wang in a comment when ready.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[lowang-bh]

we'd better add a desige doc and give some user-guide.

@Monokaix docs is added.
cc @wangyang0616 @hwdef please help to review. thanks.

[lowang-bh]

/close

[Monokaix]

It conflicts with the existing allocation logic between queues. When we allocate tasks, we do not consider the priority of jobs between queues, but only consider the priority at the queue level.

[zhoushuke]

I think there may be a bug in Reclaimable in session_plugins.go, for this configmap

    tiers:
    - plugins:
      - name: priority
      - name: proportion

as reclaim enabled in priority plugin, let us image that theres some victims return from priority, but if no victims return from proportion plugin , Reclaimable function will return no victims, and reclaim not work any more, put proportion in front of priority can fix this, but it may be a bug? @hwdef

[lowang-bh]

I think there may be a bug in Reclaimable in session_plugins.go, for this configmap

    tiers:
    - plugins:
      - name: priority
      - name: proportion

as reclaim enabled in priority plugin, let us image that theres some victims return from priority, but if no victims return from proportion plugin , Reclaimable function will return no victims, and reclaim not work any more, put proportion in front of priority can fix this, but it may be a bug? @hwdef

That is a problem of your config. You'd better to put plugins about resource in a same tier, but a tier different from gang/priority, eg:

tiers:
- plugins:
  - name: priority
    enableReclaimable: false
  - name: gang
    enablePreemptable: false
  - name: conformance
- plugins:
  - name: overcommit
  - name: drf
    enablePreemptable: false
  - name: predicates
  - name: proportion
  - name: nodeorder
  - name: binpack

[zhoushuke]

I think there may be a bug in Reclaimable in session_plugins.go, for this configmap

    tiers:
    - plugins:
      - name: priority
      - name: proportion

as reclaim enabled in priority plugin, let us image that theres some victims return from priority, but if no victims return from proportion plugin , Reclaimable function will return no victims, and reclaim not work any more, put proportion in front of priority can fix this, but it may be a bug? @hwdef

That is a problem of your config. You'd better to put plugins about resource in a same tier, but a tier different from gang/priority, eg:

tiers:
- plugins:
  - name: priority
    enableReclaimable: false
  - name: gang
    enablePreemptable: false
  - name: conformance
- plugins:
  - name: overcommit
  - name: drf
    enablePreemptable: false
  - name: predicates
  - name: proportion
  - name: nodeorder
  - name: binpack

ok, make sense

[zhoushuke]

@lowang-bh , there is another promble as following:

actions: "enqueue,allocate,backfill,preempt,reclaim"
tiers:
    - plugins:
      - name: priority
      - name: gang
        enablePreemptable: false
        enableJobStarving: false
        enableReclaimable: false
        enabledQueueScoreOrder: false
      - name: conformance
    - plugins:
      - name: predicates
      - name: proportion

gang enablePreemptable: false( make it true also don't work too), so it's cant do any preempt/reclaim, when a high priority job comes, if there no enough resource to meet gang constraint, I hope get resources by reclaim, but as not meet gang constraint, this job will pending, and reclaim action is just skip pending job, so reclaim not happend, no reclaim, no release resource to get job running, like deadlock, delete enqueue may work, but in my scene，enqueue is a must, and
put reclaim in front of enqueue is not good ideal.
and overcommit won't use for me.
what better soluation to fix it?

[lowang-bh]

@zhoushuke There are two kind of evicts:

1. evicts in same queue: this is for high priority job to evict low priority jobs when your queue's deserved resource is used up
2. evicts between different queues: a queue's coming job to reclaim their queue's deserved resource from other overused queue. In this scene, the overused resources in other queues are borrowed from this queue, and resource should be give back when need.

If you have any problems about how to use volcano, please file a issue to describe it.

[zhoushuke]

I have some volcano practice and use it in a production to support about 30k~50k pods a day.
I think you don't get my point, anyway, I will try to realize it by my own.

[Monokaix]

enablePreemptable: false

So can you explain why set enablePreemptable: false of gang？

[Monokaix]

If other queue's jobs all have higher priority，current queue can not reclaim their resources, reclaim will not happen, is this reasonable？

We just support this feature and close it by default。 How to use it is depended on the cluster admin.

One solutions is to limit the higher priority jobs used resource not exceed queue's deserved in application layer.

what about reclaim lower priority first, if no enough resourecs reclaimed then reclaim higher priority task?

[lowang-bh]

this job will pending, and reclaim action is just skip pending job

If you didn't use enqueue action, the default status will be pending and allocate action will change it to enqueue when there is no enqueue action configured. So you should put reclaim action after allocate.

[JesseStutler]

We should consider higher priority jobs in other queues can never be reclaimed if enabledReclaimable is enabled in Priority Plugin. The original design intention of reclaim should be to be able to reclaim jobs in other queues, without considering the priority, right?

[lowang-bh]

We should consider higher priority jobs in other queues can never be reclaimed if enabledReclaimable is enabled in Priority Plugin. The original design intention of reclaim should be to be able to reclaim jobs in other queues, without considering the priority, right?

Yes，docs has highlight the point that should be carefully to open it.

[stale]

Is this still relevant? If so, what is blocking it? Is there anything you can do to help move it forward?

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

[hwdef]

still need