Migrate merge-blocking jobs to dedicated cluster: pull-kubernetes-integration

[spiffxp]

Part of #18550

This is a trial balloon of a specific type of job (doesn't need an e2e project)

I'd like to get a sense of what sort of scaling jobs of this type are going to cause k8s-infra-prow-build. I'd also like to see whether this interferes with existing release-blocking job load. If so, we should consider a more boilerplate-filled approach of setting up a separate nodepool, and using taints/tolerations/nodeselectors for merge-blocking vs. release-blocking jobs.

I'm choosing pull-kubernetes-integration because it's the one I've seen flake the most lately. Getting it over to k8s-infra-prow-build will allow more community visibility, and ideally more resources thanks to autoscaling.

/area jobs
/wg k8s-infra
/sig testing
/sig release

[spiffxp]

Opened #18814

[spiffxp]

Change merged at 2020-08-12 7:40pm PT

Things to watch

• https://prow.k8s.io/?job=pull-kubernetes-integration
• https://testgrid.k8s.io/presubmits-kubernetes-blocking#pull-kubernetes-integration&graph-metrics=test-duration-minutes
• https://storage.googleapis.com/k8s-gubernator/triage/index.html?pr=1&job=pull-kubernetes-integration
• https://prow.k8s.io/job-history/gs/kubernetes-jenkins/pr-logs/directory/pull-kubernetes-integration
• prow-build dashboard 1w
• prowjobs-experiment 1w
• metrics explorer: CPU limit utilization for .*integration.* for 6h
• metrics explorer: Memory limit utilization for .*integration.* for 6h

[spiffxp]

Spot check: looks good

One interesting quirk: pull-kubernetes-integration runs (the more frequent runs that start to occur further on the right) appear to use more memory than the ci variants. Not sure if this is branch-specific at all

CPU usage looks comparable

Will continue to evaluate as traffic grows over the day

[spiffxp]

/assign

[spiffxp]

OK it's pretty clear this job is healthier since it was migrated over

Do we bother tuning resources now? Or wait until all jobs have moved over?

[spiffxp]

Just because it seemed pretty clear in this case, I'm lowring cpu from 7 to 6 to match the CI job, which is taking basically the same amount of time to run more tests. #18844

I'll hold this open to see the results, but am otherwise ready to call this done, and think other jobs of this type can be moved over

[spiffxp]

The presubmits-kubenetes-blocking testgrid doesn't go too far in history, so I can't show a lot of steady green. This was impacted by the same IO issues that plagued the CI variant of integration when we tried switching the build cluster from n1-highmem-8's to n1-highmem-16's (kubernetes/k8s.io#1173 (comment))

I think this is otherwise good to go, will poke at metrics explorer just to confirm good utilization

[spiffxp]

/close
Calling this done

[k8s-ci-robot]

@spiffxp: Closing this issue.

In response to this:

/close
Calling this done

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.