Support Kserve

[tenzen-y]

What would you like to be added:
I would like to support the serverless ML Inference tool, Kserve.

Why is this needed:
In the hybrid workload (which means training jobs and inference servers and so on) cluster, users often want to manage all cluster capacities by the kueue's flavorQuotas. So, as the first step to support the inference server, supporting Kserve in kueue is nice to have.

Completion requirements:

This enhancement requires the following artifacts:

• Design doc
• API change
• Docs update

The artifacts should be linked in subsequent comments.
We will probably implement suspend semantics on the Kserve side.
Additionally, we need to move #77 forward together to support the inference server's autoscaling semantics.

[alculquicondor]

I think I talked about this with @astefanutti
Also cc @mwielgus @ahg-g

[ahg-g]

How do you envision that working? Can you list a couple of CUJs?

[kerthcet]

Online inference service is somehow latency sensitive, scalability is highly required, reclaim/preempt the kserve managed services looks not right. I guess Kserve is not that good at offline inference, which in my mind maybe helpful. cc @terrytangyuan

[lizzzcai]

I would like to see possible support for this as I am looking for a unified way of managing resources for both model training and serving and Kueue looks like it has this capability. In our case, both training and serving are running in the same cluster. And how it can integrate with the recent MultiKueue feature to schedule workload to clusters with available GPU (sometimes there is a shortage of GPU in certain regions). As KServe deployment has min and max replicas, it should be scheduled to cluster that can meet the max replicas.

[tenzen-y]

How do you envision that working? Can you list a couple of CUJs?

I imagined that the similar approach as RayCluster.

So, I would like to add Suspend field to InferenceService resource.

[tenzen-y]

Online inference service is somehow latency sensitive, scalability is highly required, reclaim/preempt the kserve managed services looks not right. I guess Kserve is not that good at offline inference, which in my mind maybe helpful. cc @terrytangyuan

@kerthcet I believe that lending limit would allow us to guarantee capacities for latency sensitive Workloads.

[tenzen-y]

I would like to see possible support for this as I am looking for a unified way of managing resources for both model training and serving and Kueue looks like it has this capability. In our case, both training and serving are running in the same cluster. And how it can integrate with the recent MultiKueue feature to schedule workload to clusters with available GPU (sometimes there is a shortage of GPU in certain regions). As KServe deployment has min and max replicas, it should be scheduled to cluster that can meet the max replicas.

Yes, that's right. Actually, I also deploy Job and Inference Server into a single cluster.

[tenzen-y]

Let me try to design this integrations.

/assign

[terrytangyuan]

Thanks! Great to see this. Looking forward to your proposal. @tenzen-y

[tenzen-y]

Thanks! Great to see this. Looking forward to your proposal. @tenzen-y

I will create a dedicated issue later in Kserve side as well.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[tenzen-y]

/remove-lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[tenzen-y]

/remove-lifecycle stale

[tenzen-y]

I do not have enough bandwidth.
So, feel free to take this if anyone is interested in this feature.

/unasign

[tenzen-y]

/unassign

[kerthcet]

FYI: we have similar requirements for serving projects but mostly for resource fungibility rather quota managements(although related as a whole).

[tenzen-y]

FYI: we have similar requirements for serving projects but mostly for resource fungibility rather quota managements(although related as a whole).

Yeah, I fully agree with you. Actually, we wanted to provide the resource fungibility in the mixed type workload clusters.
For example, I want to allocate the free resources to Jobs when a few serving requests come. After serving requests go spike, we want to preempt Jobs and allocate resources to Serving for auto scaling.

But, I found that we can fulfill this motivation by using Kueue Pod integration with a little bit of Knative tuning.
So, maybe before we start natively supporting, we may want to add documentations for the way.

I will try to add Documentation on how to perform Kserve ISVC with Kueue.

[terrytangyuan]

@tenzen-y It would be great if you could share some docs and examples on that!

[tenzen-y]

@tenzen-y It would be great if you could share some docs and examples on that!

Sure, I will share it in the Kueue documentation.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[tenzen-y]

/remove-lifecycle stale

[kannon92]

@tenzen-y It would be great if you could share some docs and examples on that!

We are looking to add a section in the docs to show using pod integration with external frameworks.

#3898

is an example of using Pod Integration with TektonPipelines.

@varshaprasad96 has done some exploring on how to use kubeflow notebooks with the statefulset integration.

[varshaprasad96]

@varshaprasad96 has done some exploring on how to use kubeflow notebooks with the statefulset integration.

I'm doing a few tests with the latest version of Kueue for NB integration. Will open an upstream PR to add docs soon.