ci-kubernetes-build-canary jobs are failing to be scheduled

[justaugustus]

What happened:

The last six runs of ci-kubernetes-build-canary have had scheduling issues.
(Some of these are triggered reruns.)

What you expected to happen:

Successful scheduling on a Prow node 🙃

How to reproduce it (as minimally and precisely as possible):

Reproduces on any current run of the job.

Please provide links to example occurrences, if any:

Job: https://prow.k8s.io/?job=ci-kubernetes-build-canary

Failed runs:

• https://prow.k8s.io/view/gs/kubernetes-jenkins/logs/ci-kubernetes-build-canary/1355251705272340480
• https://prow.k8s.io/view/gs/kubernetes-jenkins/logs/ci-kubernetes-build-canary/1355244686834405376
• https://prow.k8s.io/view/gs/kubernetes-jenkins/logs/ci-kubernetes-build-canary/1355229313380651008
• https://prow.k8s.io/view/gs/kubernetes-jenkins/logs/ci-kubernetes-build-canary/1355221848350003200
• https://prow.k8s.io/view/gs/kubernetes-jenkins/logs/ci-kubernetes-build-canary/1355215664226242560
• https://prow.k8s.io/view/gs/kubernetes-jenkins/logs/ci-kubernetes-build-canary/1355200313836965888

Anything else we need to know?:

The job was recently migrated from using bootstrap to krel ci-build in #20663.
ref: kubernetes/release#1711 (comment)

cc: @kubernetes/release-engineering @spiffxp

[justaugustus]

Slack thread in #testing-ops: https://kubernetes.slack.com/archives/C7J9RP96G/p1611953363009500

[chaodaiG]

Latest succeeded run is configured at 9f603e50-624b-11eb-bd55-a20b0ecfb997

First failed run is configured at 2395e2c8-6254-11eb-bd55-a20b0ecfb997

One noticeable difference in the prowjob is that in the failed run:

    resources:
      clonerefs:
        requests:
          cpu: 100m
      initupload:
        requests:
          cpu: 100m
      place_entrypoint:
        requests:
          cpu: 100m
      sidecar:
        requests:
          cpu: 100m

which is not present in the succeeded run. And test container both requested:

requests:
          cpu: 7300m
          memory: 34Gi

It's possible that each node in cluster: k8s-infra-prow-build can only allocate either 7.3 cpu or 34Gi memory, can someone have access check the node size?

[chaodaiG]

To make it more clear, the difference in the prowjobs resulted requested cpu from 7.3 to 7.7, and probably not related to memory

[spiffxp]

Thanks @chaodaiG that is most likely the problem.

A job trying to schedule all available CPU as a proxy for hogging a node all to itself is a risky strategy for this reason. The k8s-infra-prow-build nodes have (among other daemonsets) calico running on them for network policy enforcement, meaning they have slightly more overhead / less allocatable cpu than the google.com default nodes.

/area jobs
/sig release
/assign @justaugustus
The kind release-blocking jobs also use pod-utils and try to hog nodes, they use 7 cpu (e.g. https://github.com/kubernetes/test-infra/blob/master/config/jobs/kubernetes/sig-release/release-branch-jobs/1.20.yaml#L602-L650). Try that out.

[BenTheElder]

The kind release-blocking jobs also use pod-utils and try to hog nodes, they use 7 cpu (e.g. https://github.com/kubernetes/test-infra/blob/master/config/jobs/kubernetes/sig-release/release-branch-jobs/1.20.yaml#L602-L650). Try that out.

I/etcd just need those sweet sweet IOPs 🦆 , someday we'll have scheduling for this ...

[justaugustus]

Thanks everyone for taking a peek.
Opened #20674.

[spiffxp]

I/etcd just need those sweet sweet IOPs 🦆 , someday we'll have scheduling for this ...

If we can make progress on moving k8s-infra-prow-build up to 1.18 we can try out nodepools with local-ssd, waaaaaay more iops kubernetes/k8s.io#1187 (comment)