Identify quota changes needed for scalability jobs, create pool of scalability projects

[spiffxp]

The default quotas for an e2e project (eg: k8s-infra-e2e-gce-project) are insufficient to run ci-kubernetes-e2e-gci-gce-scalability

Currently this job runs in the google.com k8s-prow-builds cluster, using a project from that boskos' scalability-project pool

• Identify what quotas are set that make these projects differ from a default e2e project
• Identify what jobs this will allow us to migrate
• Identify pool size
• Provision pool in CNCF org
• Add pool to k8s-infra-prow-build's boskos

[spiffxp]

I created k8s-infra-e2e-scale-project and tried manually running ci-kubernetes-e2e-gci-gce-scalability in k8s-infra-prow-build (with some local modifications not to dump results to gs://kubernetes-jenkins)

It got stuck on CPU quota

I was able to retrieve project info for k8s-periodic-scale-1, so I compared with k8s-infra-e2e-scale-project globally and in us-east1. CPUs and in-use addresses were bumped up to 125 in us-east1.

I filed a request to raise quota for k8s-infra-e2e-scale-project with the following info:

OSS Kubernetes scalability testing, 100 node clusters

https://testgrid.k8s.io/sig-release-master-blocking#gce-cos-master-scalability-100

Goal is to migrate OSS Kubernetes tests to use GCP projects under CNCF-owned kubernetes.io org instead of google.com org

Trying to match quotas of google.com project "k8s-periodic-scale-1". Anticipate 10 projects like this.

The quota got bumped during the job run, and the job was able to complete successfully.

Open questions:

• do quotas differ in other regions?
• what other jobs benefit from this?

[spiffxp]

Jobs that use scalability-project

100 node release-blocking tests (definitely migrate these)

• ci-kubernetes-e2e-gci-gce-scalability - https://testgrid.k8s.io/sig-release-master-blocking#gce-cos-master-scalability-100
• ci-kubernetes-e2e-gci-gce-scalability-stable3 - https://testgrid.k8s.io/sig-release-1.15-blocking#gce-cos-1.15-scalability-100
• etc for each release branch

100 node kubemark tests (these look like CI variants of merge-blocking jobs so make room for them)

• ci-kubernetes-kubemark-100-gce - https://testgrid.k8s.io/sig-scalability-kubemark#kubemark-100
• ci-kubernetes-kubemark-500-gce-stable3 - https://testgrid.k8s.io/sig-scalability-kubemark#kubemark-1.15-500
• etc for each release branch
• ci-kubernetes-kubemark-high-density-100-gce - https://testgrid.k8s.io/sig-scalability-kubemark#kubemark-100-high-density

experiment tests (I would hold off on these for now)

• ci-kubernetes-e2e-gci-gce-scalability-networkpolicies - https://testgrid.k8s.io/sig-scalability-experiments#calico
• ci-kubernetes-e2e-scalability-experimental-load - https://testgrid.k8s.io/sig-scalability-experiments#experimental-load
• ci-kubernetes-e2e-scalability-experimental-load-containerd - https://testgrid.k8s.io/sig-scalability-experiments#experimental-load-containerd
• ci-kubernetes-e2e-gci-gce-scalability-containerd - https://testgrid.k8s.io/sig-scalability-experiments#gce-cos-master-scalability-100-containerd
• ci-kubernetes-e2e-gci-gce-scalability-node-killer - https://testgrid.k8s.io/sig-scalability-experiments#gce-cos-master-scalability-100-nodekiller

[spiffxp]

I did a visual inspection of this with all regions, so filter down to just the two that had meaningful differences

for p in $(<~/w/kubernetes/test-infra/config/prow/cluster/boskos-resources.yaml yq -r '.resources[] | select (.type=="scalability-project").names[]'); do 
  echo $p...; 
  diff --ignore-all-space \
    <(gcloud compute regions list --project=$p)
    <(gcloud compute regions list --project=k8s-infra-e2e-gce-project) |\
  grep -E "us-east1|us-central1|---"
done

• 16/16 had at least 125 CPU and 125 in-use addresses in us-east1, e.g.
• 2/16 had additional CPU quota in us-central1
• 2/16 had 250 CPU instead of 125
• 4/16 had 10240 instead of default 40960 disk
• 4/16 had 50000 instead of default 40960 disk

So I'm inclined to suggest we stick with 125 CPU / in-use addresses for us-east1

The existing pool is 16 projects, but there are 5 jobs I'm not sure we want to move over yet. So I'm going to setup 10 projects

[spiffxp]

Blocked until #852 is resolved

[spiffxp]

No longer blocked. I've created a pool of 5 projects to start with via changes in #898

[spiffxp]

With an eye toward: kubernetes/test-infra#18550

Based on visual inspection of https://monitoring.prow.k8s.io/d/wSrfvNxWz/boskos-resource-usage?orgId=1&from=now-90d&to=now

Currently k8s-prow-build's boksos has two pools:

• scalability project: 15 total, peak usage of ~5
• scalability presubmit projects: 45 total, peak usage of ~30

Currently k8s-infra-prow-build has:

• scalability project: 5 total, peak usage of ~3

It's not clear to me whether the presubmit projects have different quotas than the regular projects

[spiffxp]

Added a canary job via kubernetes/test-infra#19049

https://prow.k8s.io/view/gcs/kubernetes-jenkins/pr-logs/pull/92316/pull-kubernetes-e2e-gce-100-performance-canary/1299860480084414464/ - confirmed that the existing scalability-project pool will work for presubmits

Based on visual inspection of https://monitoring.prow.k8s.io/d/wSrfvNxWz/boskos-resource-usage?orgId=1&from=now-90d&to=now, after removing a kubemark presubmit out of the set of merge-blocking jobs for kubernetes (ref: kubernetes/test-infra#18788)

k8s-prow-build's boksos has two pools:

• scalability project: 15 total, peak usage of ~5
• scalability presubmit projects: 45 total, peak usage of ~17

k8s-infra-prow-build has:

• scalability project: 5 total, peak usage of ~3

Going to provision scalability pool up to 30 projects (5+17+3=25 + 5 overhead)

[spiffxp]

Opened #1192 to grow the pool

[spiffxp]

/close
Closing because it looks like we've got enough capacity for now. Specifically, there's enough green leftover in the "scalability project (k8s-infra)" pool to account for usage of the other two scalability project pools. This issue covers provisioning the projects, not migration of the jobs

[k8s-ci-robot]

@spiffxp: Closing this issue.

In response to this:

/close
Closing because it looks like we've got enough capacity for now. Specifically, there's enough green leftover in the "scalability project (k8s-infra)" pool to account for usage of the other two scalability project pools. This issue covers provisioning the projects, not migration of the jobs

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.