Add PriorityClass in Workload api #104

denkensk · 2022-03-09T12:56:16Z

What type of PR is this?

/kind feature

What this PR does / why we need it:

Which issue(s) this PR fixes:

The first part of #82

denkensk · 2022-03-09T12:58:49Z

/assign @ahg-g

api/v1alpha1/queuedworkload_types.go

denkensk · 2022-03-09T16:09:54Z

/test pull-kueue-test-integration-main

denkensk · 2022-03-10T00:26:03Z

/test pull-kueue-test-integration-main

api/v1alpha1/queuedworkload_types.go

alculquicondor · 2022-03-10T16:19:22Z

pkg/controller/workload/job/job_controller.go

+func ConstructWorkloadFor(ctx context.Context, client client.Client,
+	job *batchv1.Job, scheme *runtime.Scheme) (w *kueue.QueuedWorkload, err error) {
+	var p int32
+	pcName := job.Spec.Template.Spec.PriorityClassName


What will be do for workloads that have multiple pod specs?

Should we do this in the queued_workload controller instead?

I rarely see people setting different priorities in a single workload. But in this case, we need a fixed treatment and clear understanding for the user: like choosing the highest one as the priority of the workload?

What will be do for workloads that have multiple pod specs?

It is up to the custom workloads to decide from where to set it (e.g., based on the driver).

Should we do this in the queued_workload controller instead?

how? queuedworkload_controller is not aware of the custom workload CRD.

That makes me think, since order is done at the ClusterQueue level and Queue is simply a pointer to it; what if we have the priority on the Queue, and workloads simply inherit it from the Queue they are submitted to, and so users don't need to set it on the custom workload directly and we avoid those issues altogether.

Should we do this in the queued_workload controller instead?

how? queuedworkload_controller is not aware of the custom workload CRD.

I'm thinking that workloads should have a priority independent of the pod priority, although it might be confusing. Also that would mean that users have to set an annotation for it in the Job or something like that. Having it in the Queue is certainly a cleaner option.

I also agree that workloads have a separate priority independent of the pod priority. In fact here it already is kubeflow
https://github.com/kubeflow/common/blob/2b40c8f8991e302920ee5536c0ad49dec6724c66/pkg/apis/common/v1/types.go#L208

I see. Do you know what they do with it? Perhaps insert it in the pod specs?

denkensk · 2022-03-15T08:09:52Z

/test pull-kueue-test-integration-main

denkensk · 2022-03-15T12:48:43Z

/test pull-kueue-test-integration-main

api/v1alpha1/queuedworkload_types.go

pkg/constants/constants.go

pkg/controller/core/queuedworkload_controller.go

pkg/queue/queue.go

denkensk · 2022-03-15T16:17:58Z

/test pull-kueue-test-integration-main

alculquicondor

what is populating the PriorityClassName now?

alculquicondor · 2022-03-15T18:12:52Z

api/v1alpha1/queuedworkload_types.go

+	// keywords which indicate the highest priorities with the former being
+	// the highest priority. Any other name must be defined by creating a
+	// PriorityClass object with that name. If not specified, the queuedWorkload
+	// priority will be default or zero if there is no default.


where is this default defined?

in the constant

DefaultPriority = 0

Since it wouldn't change by external factors, you should just say that the default priority is zero.

btw, I'm referring to If not specified, the queuedWorkload priority will be default

Is this referring to the default priority class defined for the cluster?

Yes. It's the default priority class defined for the cluster. But if there is no default. It will be 0.
Keep the comments as
https://github.com/kubernetes/kubernetes/blob/ca2cd3b18ef145c34311ba7fd9d389fe8233fae8/pkg/apis/core/types.go#L2902

api/v1alpha1/queuedworkload_types.go

pkg/queue/queue.go

alculquicondor · 2022-03-15T18:23:24Z

pkg/queue/queue.go

 	heap.Push(&cq.heap, *info)
 	return true
 }

-func (cq *ClusterQueue) PushOrUpdate(w *kueue.QueuedWorkload) {
+func (cq *ClusterQueue) PushOrUpdate(w *kueue.QueuedWorkload) bool {
 	item := cq.heap.items[workload.Key(w)]
 	info := *workload.NewInfo(w)


why not calculate the value in NewInfo?

I was initially hoping to calculate in NewInfo. But I find WorkloadInfo is a basic data struct which is invoked in other places like cache. It's hard and unclean if I pass the log/ctx/client to everywhere I want to invoke New.

denkensk · 2022-03-16T08:39:41Z

/retest

ahg-g · 2022-03-16T11:09:57Z

I am sorry for the back and forth, this plumbing doesn't look "right". I am pretty sure there will be places where we will miss to set info.Priority and so it will be dangerous to continue like this unless the priority look up is done inside NewInfo, which I think isn't going to be pretty because it means NewInfo will need to receive client/ctx/logger, and it is used in multiple places including the cache.

I coming back to having the priority int on the QueuedWorkload and having it populated by the job-controller. Thinking more about how to prevent users from circumventing priority classes: users shouldn't be creating QueuedWorkload instances directly, and so they really can't set the QueuedWorkload priority (at least in our MVP).

Alex, I am pretty sure this is frustrating to you, sorry about that, I am happy to make the change myself if you don't have the time.

pkg/workload/workload.go

denkensk · 2022-03-16T11:39:07Z

I coming back to having the priority int on the QueuedWorkload and having it populated by the job-controller. Thinking more about how to prevent users from circumventing priority classes: users shouldn't be creating QueuedWorkload instances directly, and so they really can't set the QueuedWorkload priority (at least in our MVP).

Shall we only have Priority in QueuedWorkload or both PriorityClass and Priority?

Alex, I am pretty sure this is frustrating to you, sorry about that, I am happy to make the change myself if you don't have the time.

It doesn't matter. It's worth putting in some effort before making the right decision. I will try to finish refactor the code today and tomorrow.

ahg-g · 2022-03-16T11:54:31Z

Shall we only have Priority in QueuedWorkload or both PriorityClass and Priority?

Both, the job-controller populates it just like we do for pods. Initially I was concerned that there was no path for Admins to block usage of higher priority classes or preventing users from setting directly a high priority on the workload. But that is actually not true because admins could prevent users from creating QW directly and only allow them to create v1.Job.

I still think we want to have priority class on the Queue object but we can leave that as a followup.

It doesn't matter. It's worth putting in some effort before making the right decision. I will try to finish refactor the code today and tomorrow.

Thank you, and I am sorry again for the back and forth, but indeed this was an informative exercise, at least for me.

denkensk · 2022-03-16T13:14:22Z

Since this is very similar to my first version of the implementation, so I finish it quickly 😄 pls take a look again. Thanks. @ahg-g

ahg-g

thanks, good for squash

pkg/queue/queue_test.go

ahg-g · 2022-03-16T13:34:46Z

It would be nice if we can add a scheduler integration test in a separate ginkgo.It

alculquicondor · 2022-03-16T13:51:13Z

api/v1alpha1/queuedworkload_types.go

+	// keywords which indicate the highest priorities with the former being
+	// the highest priority. Any other name must be defined by creating a
+	// PriorityClass object with that name. If not specified, the queuedWorkload
+	// priority will be default or zero if there is no default.


btw, I'm referring to If not specified, the queuedWorkload priority will be default

Is this referring to the default priority class defined for the cluster?

alculquicondor · 2022-03-16T13:51:58Z

pkg/controller/workload/job/job_controller.go

@@ -35,6 +35,7 @@ import (

 	kueue "sigs.k8s.io/kueue/api/v1alpha1"
 	"sigs.k8s.io/kueue/pkg/constants"
+	utilpriority "sigs.k8s.io/kueue/pkg/util/priority"


nit: remove the alias. Use just priority

utilpriority is auto add by auto import. I suggest keeping as utilpriority.

pkg/queue/queue.go

pkg/queue/queue_test.go

alculquicondor · 2022-03-16T13:57:06Z

pkg/queue/queue_test.go

@@ -24,6 +24,10 @@ import (
 	kueue "sigs.k8s.io/kueue/api/v1alpha1"
 )

+var (
+	lowPriority, highPriority = int32(0), int32(1000)


make it 2 lines

alculquicondor · 2022-03-16T13:59:19Z

pkg/queue/queue_test.go

@@ -83,3 +87,98 @@ func TestFIFOClusterQueue(t *testing.T) {
 		t.Errorf("Queue is not empty, poped workload %q", got.Obj.Name)
 	}
 }
+
+func TestStrictFIFO(t *testing.T) {


For a follow up, we should find a way to merge the 2 tests #72

alculquicondor · 2022-03-16T14:00:28Z

pkg/util/priority/priority.go

+)
+
+// Priority returns priority of the given workload.
+func Priority(w workload.Info) int32 {


why not make this a method of workload.Info?

No special reason. Just keep it as k/k.
And I also think it's not suitable as a member function. But I can change the input as kueue.workload to make it can be used other place in the future.

pkg/controller/workload/job/job_controller.go

alculquicondor · 2022-03-16T14:04:42Z

pkg/util/routine/wrapper.go

@@ -1,3 +1,19 @@
+/*


Oops, I wonder if we can make a verify step for this.

Signed-off-by: Alex Wang <wangqingcan1990@gmail.com>

denkensk · 2022-03-17T05:03:31Z

It would be nice if we can add a scheduler integration test in a separate ginkgo.It

Added integration test for scheduler and job-controller in the second commit @ahg-g

ahg-g

Thanks! this is looking great!

test/integration/scheduler/scheduler_test.go

ahg-g · 2022-03-17T15:47:33Z

/require-retest

ArangoGutierrez · 2022-03-17T15:59:09Z

/retest-required

Signed-off-by: Alex Wang <wangqingcan1990@gmail.com>

denkensk · 2022-03-17T16:04:07Z

Updated base on the review comments. @ahg-g

ahg-g · 2022-03-17T16:18:59Z

/retest-required

ahg-g · 2022-03-17T16:30:53Z

/lgtm
/approve

k8s-ci-robot · 2022-03-17T16:31:12Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ahg-g, denkensk

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [ahg-g]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

k8s-ci-robot added kind/feature Categorizes issue or PR as related to a new feature. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Mar 9, 2022

k8s-ci-robot requested review from alculquicondor and ArangoGutierrez March 9, 2022 12:56

k8s-ci-robot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Mar 9, 2022

k8s-ci-robot assigned ahg-g Mar 9, 2022

ArangoGutierrez reviewed Mar 9, 2022

View reviewed changes

api/v1alpha1/queuedworkload_types.go Outdated Show resolved Hide resolved

denkensk force-pushed the add-workload-priority branch from 9572a80 to 6730672 Compare March 9, 2022 15:58

ahg-g reviewed Mar 10, 2022

View reviewed changes

api/v1alpha1/queuedworkload_types.go Outdated Show resolved Hide resolved

ahg-g reviewed Mar 10, 2022

View reviewed changes

api/v1alpha1/queuedworkload_types.go Outdated Show resolved Hide resolved

alculquicondor reviewed Mar 10, 2022

View reviewed changes

denkensk changed the title ~~Add Priority in Workload api~~ IAdd Priority in Workload api Mar 11, 2022

denkensk changed the title ~~IAdd Priority in Workload api~~ wip Add Priority in Workload api Mar 11, 2022

k8s-ci-robot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Mar 11, 2022

denkensk force-pushed the add-workload-priority branch from 6730672 to e4ca343 Compare March 15, 2022 08:05

denkensk changed the title ~~wip Add Priority in Workload api~~ wip Add PriorityClass in Workload api Mar 15, 2022

denkensk changed the title ~~wip Add PriorityClass in Workload api~~ Add PriorityClass in Workload api Mar 15, 2022

k8s-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Mar 15, 2022

denkensk requested review from ahg-g and alculquicondor March 15, 2022 08:18

denkensk force-pushed the add-workload-priority branch from e4ca343 to 1d39c30 Compare March 15, 2022 08:20

ahg-g reviewed Mar 15, 2022

View reviewed changes

denkensk requested a review from ahg-g March 15, 2022 16:18

alculquicondor reviewed Mar 15, 2022

View reviewed changes

denkensk force-pushed the add-workload-priority branch from 6b14068 to 89b35c3 Compare March 16, 2022 09:18

ahg-g reviewed Mar 16, 2022

View reviewed changes

pkg/workload/workload.go Outdated Show resolved Hide resolved

denkensk force-pushed the add-workload-priority branch 2 times, most recently from edacea0 to 9215ca6 Compare March 16, 2022 13:11

ahg-g reviewed Mar 16, 2022

View reviewed changes

pkg/queue/queue_test.go Outdated Show resolved Hide resolved

pkg/queue/queue_test.go Outdated Show resolved Hide resolved

alculquicondor reviewed Mar 16, 2022

View reviewed changes

add priority in workload api

561fe4f

Signed-off-by: Alex Wang <wangqingcan1990@gmail.com>

denkensk force-pushed the add-workload-priority branch from 9215ca6 to e2c2e8c Compare March 17, 2022 05:00

denkensk force-pushed the add-workload-priority branch from e2c2e8c to 43a56b3 Compare March 17, 2022 05:12

ahg-g reviewed Mar 17, 2022

View reviewed changes

add integration test for workload priority

39e9333

Signed-off-by: Alex Wang <wangqingcan1990@gmail.com>

denkensk force-pushed the add-workload-priority branch from 43a56b3 to 39e9333 Compare March 17, 2022 16:00

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Mar 17, 2022

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Mar 17, 2022

k8s-ci-robot merged commit 32a2d30 into kubernetes-sigs:main Mar 17, 2022

denkensk mentioned this pull request Mar 21, 2022

Efficient re-queueing of unschedulable workloads #8

Closed

kerthcet mentioned this pull request Apr 10, 2022

introduce priorityFIFO as another QueueingStrategy of ClusterQueue #200

Closed

Add PriorityClass in Workload api #104

Add PriorityClass in Workload api #104

Conversation

denkensk commented Mar 9, 2022 • edited Loading

What type of PR is this?

What this PR does / why we need it:

Which issue(s) this PR fixes:

denkensk commented Mar 9, 2022

denkensk commented Mar 9, 2022

denkensk commented Mar 10, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ahg-g Mar 11, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

denkensk commented Mar 15, 2022

denkensk commented Mar 15, 2022

denkensk commented Mar 15, 2022

alculquicondor left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

denkensk commented Mar 16, 2022

ahg-g commented Mar 16, 2022

denkensk commented Mar 16, 2022

ahg-g commented Mar 16, 2022

denkensk commented Mar 16, 2022

ahg-g left a comment

Choose a reason for hiding this comment

ahg-g commented Mar 16, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

denkensk commented Mar 17, 2022

ahg-g left a comment

Choose a reason for hiding this comment

ahg-g commented Mar 17, 2022

ArangoGutierrez commented Mar 17, 2022

denkensk commented Mar 17, 2022

ahg-g commented Mar 17, 2022

ahg-g commented Mar 17, 2022

k8s-ci-robot commented Mar 17, 2022

denkensk commented Mar 9, 2022 •

edited

Loading

ahg-g Mar 11, 2022 •

edited

Loading