Migrate merge-blocking jobs to dedicated cluster: pull-kubernetes-e2e-kind

[spiffxp]

Part of #18550

This is a trial balloon of a specific type of job (kind)

I'd like to get a sense of what sort of scaling jobs of this type are going to cause k8s-infra-prow-build. I'd also like to see whether this interferes with existing release-blocking job load. If so, we should consider a more boilerplate-filled approach of setting up a separate nodepool, and using taints/tolerations/nodeselectors for merge-blocking vs. release-blocking jobs.

I'm choosing pull-kubernetes-e2e-kind because I've seen it hit scheduling errors due to lack of resources lately. Getting it over to k8s-infra-prow-build will allow more community visibility, and ideally more resources thanks to autoscaling.

/area jobs
/wg k8s-infra
/sig testing
/sig release

[spiffxp]

/kind cleanup

[spiffxp]

/assign

[spiffxp]

/assign @ameukam
who opened #18817

[spiffxp]

#18817 merged at 2020-08-13 10:24am PT
#18825 merged at 2020-08-13 12:17pm PT

Things to watch

• https://prow.k8s.io/?job=pull-kubernetes-e2e-kind
• https://testgrid.k8s.io/presubmits-kubernetes-blocking#pull-kubernetes-e2e-kind&graph-metrics=test-duration-minutes
• https://storage.googleapis.com/k8s-gubernator/triage/index.html?pr=1&job=pull-kubernetes-e2e-kind
• https://prow.k8s.io/job-history/gs/kubernetes-jenkins/pr-logs/directory/pull-kubernetes-e2e-kind
• prow-build dashboard 1w
• prowjobs-experiment 1w
• metrics explorer: CPU limit utilization for .*kind.* for 6h
• metrics explorer: Memory limit utilization for .*kind.* for 6h

[spiffxp]

https://prow.k8s.io/?job=pull-kubernetes-e2e-kind&state=error - so there's a bunch of jobs that errored between 10am and 12pm PT... which lines up roughly with the time between the two PR's merging

I'm guessing cpu: 7400m was too much to ask for on k8s-infra-prow-build nodes, and then switching to cpu: 4 made the jobs schedulable again.

[spiffxp]

I'm reasonably happy with this

[BenTheElder]

follow up: disable the network policy addon in the cluster

[ameukam]

@spiffxp Is there still something else to do? I think we can close.

@BenTheElder Out of curiosity, what's the reason behind this action :

disable the network policy addon in the cluster

[spiffxp]

what's the reason behind this action

We are not, as far as I know, actually doing anything with it? Maybe it would be better to get the resources back

[spiffxp]

/close
Calling this done, it continued to behave well once opened up traffic to v1.20 PRs

[k8s-ci-robot]

@spiffxp: Closing this issue.

In response to this:

/close
Calling this done, it continued to behave well once opened up traffic to v1.20 PRs

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.