Add scalability tests

[ahg-g]

This is critical to better understand kueue's limits and where its bottlenecks. We should check if there is a way to use clusterloader for this

[ahg-g]

/help

[k8s-ci-robot]

@ahg-g:
This request has been marked as needing help from a contributor.

Guidelines

Please ensure that the issue body includes answers to the following questions:

• Why are we solving this issue?
• To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points?
• Does this issue have zero to low barrier of entry?
• How can the assignee reach out to you for help?

For more details on the requirements of such an issue, please see here and ensure that they are met.

If this request no longer meets these requirements, the label can be removed
by commenting with the /remove-help command.

In response to this:

/help

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[ArangoGutierrez]

I think the first we can do for this issue is to lay down what we want to test/bench for Kueue.
I think I good first approach is to get a baseline of resource utilization vs scale.

• How many sample queues can kueue hold when running on 1 cpu and 200MB mem.
• On an empty cluster with "infinite" resources, how long does it take to Kueue to fully admit all the jobs into a ClusterQueue

those are init test ideas

[alculquicondor]

I think we should test "infinite" resources but also "limited" (say, the ClusterQueues have 20% of the total required requests for all jobs)

[ArangoGutierrez]

With config

tuned-adm profile virtual-host
minikube start \
        --driver=kvm2 --container-runtime=cri-o \
        --extra-config=kubelet.cgroup-driver=systemd \
        --kubernetes-version=latest \
        --kvm-numa-count=1 \
        --nodes=3 --cpus=3 --memory=4g \

10 jobs CPU/Mem usage:

[ArangoGutierrez]

Can't get more on Minikube :( , will have to move to something else. will report more later

[ArangoGutierrez]

empty queues, staging Kueue, after 15 mins of inactivity (bootstrap stabilization)

Base line

[ArangoGutierrez]

We need to create a list of target features to stress test to move forward with this issue

[ArangoGutierrez]

Do we want to do resource utilization testing?
e.g :

• How many CPU and MEM is used by Kueue while processing X number of Workloads
• CPU and MEM on a system with "infinite" resources vs a system with no resources, to see how a high traffic moment can affect resource consumption by Kueue.

[ArangoGutierrez]

• In a system with "infinite" resources, how long does it take to Queue to admit X number of workloads

[ahg-g]

We need to run tests that measure:

1. Workload admission latency
2. Workload admission throughput
3. cpu/memory (we don't set the limits, so it can compete with whatever available resources on the master)

The scale test should measure that while varying

1. The number of Namespaces/Queues; tens to hundreds.
2. The number of ClusterQueues; few to as many as Namespaces
3. The number workloads; hundreds, thousands, to few tens of thousands

As for CPU and memory requests, we can use settings similar to kube-scheduler.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[alculquicondor]

/lifecycle frozen

[alculquicondor]

A first load test was added in #462.

We can create separate issues for follow ups and to improve performance.

/close

[k8s-ci-robot]

@alculquicondor: Closing this issue.

In response to this:

A first load test was added in #462.

We can create separate issues for follow ups and to improve performance.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.