Replace borrowing ceiling with weight

[alculquicondor]

bit.ly/kueue-apis defined a weight to dynamically set a borrowing ceiling for each Capacity, based on the total resources in the Cohort and the capacities that have pending workloads.

We need to implement such behavior and remove the ceiling.
The weights and unused resources should lead to a dynamic ceiling that is calculated in every scheduling cycle. The exact semantics of this calculation are not fully understood.
In a given scheduling cycle, which capacities are considered for splitting the unused resources? Only the ones with pending jobs? What about the ones that are already borrowing but have no more pending jobs? What is considered unused resources once some resources have already being borrowed?

There are probably a few interpretations to these questions that lead to slightly different results. We need to explore them and pick one that sounds more reasonable or is based on existing systems.

[alculquicondor]

/kind feature
/size L
/priority important-longterm

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[alculquicondor]

/lifecycle frozen

[alculquicondor]

Alternatively, the weights could be added to the namespace, so that fairness can be evaluated for the cluster by tenant, rather than per ClusterQueue.

This calls for a proper design doc.

[ahg-g]

Alternatively, the weights could be added to the namespace, so that fairness can be evaluated for the cluster by tenant, rather than per ClusterQueue.

Where would those weights be stored? I think setting a weight per namespace is probably going to be difficult to maintain, there will be way more namespaces than CQs

[alculquicondor]

From @denkensk in https://github.com/kubernetes-sigs/kueue/pull/410/files#r998066062:

I think we can probably add some algorithms in the future to ensure that there is fairness between the cluster queues？ Even we don't violate the min，but maybe can not preempt the workloads which are all coming from the same CQ？Maybe you know, we always do not want to hurt some single user too much, even though he borrowed someone else's resources.

[alculquicondor]

/close
in favor of #1714

[k8s-ci-robot]

@alculquicondor: Closing this issue.

In response to this:

/close
in favor of #1714

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.