Migrate merge-blocking jobs to dedicated cluster: pull-kubernetes-e2e-gce

[spiffxp]

What should be cleaned up or changed:

This is part of #18550

To properly monitor the outcome of this, you should be a member of k8s-infra-prow-viewers@kubernetes.io. PR yourself into https://github.com/kubernetes/k8s.io/blob/master/groups/groups.yaml#L603-L628 if you're not a member.

Migrate pull-kubernetes-e2e-gce to k8s-infra-prow-build by adding a cluster: k8s-infra-prow-build field to the job:

• Make sure any release-branch variants are also updated
• Reference this issue, but do not "fix" it; instead leave this open to monitor the behavior of the job before/after
• Here's an example PR that did this for pull-kubernetes-integration: Migrate pull-kubernetes-integration to k8s-infra-prow-build #18814

NOTE: migrating this job is not as straightforward as some of the other #18550 issues, because:

• it's being demoted from merge-blocking on release-1.19 and the main branch (as of pull-kubernetes-e2e-gce always_run: false #18832)
• so in terms of appropriate amount of PR traffic, either manually trigger this job, or look for release-1.18 (and earlier) PRs

Once the PR has merged, note the date/time it merged. This will allow you to compare before/after behavior.

Things to watch for the job

• https://prow.k8s.io/?job=pull-kubernetes-e2e-gce
  • does the job start failing more often?
  • does the job start going into error state?
• https://testgrid.k8s.io/presubmits-kubernetes-blocking#pull-kubernetes-e2e-gce&graph-metrics=test-duration-minutes
  • does the job duration look worse than before? spikier than before?
• https://storage.googleapis.com/k8s-gubernator/triage/index.html?pr=1&job=pull-kubernetes-e2e-gce
  • do more failures show up than before?
• https://prow.k8s.io/job-history/gs/kubernetes-jenkins/pr-logs/directory/pull-kubernetes-e2e-gce
  • (can be used to answer some of the same questions as above)
• metrics explorer: CPU limit utilization for pull-kubernetes-e2e-gce for 6h
  • is the job wildly underutilizing its CPU limit? if so, perhaps tune down (if uncertain, post evidence in this issue and ask)
  • (it will probably be helpful to look at different time resolutions like 1h, 6h, 1d, 1w)
• metrics explorer: Memory limit utilization for pull-kubernetes-e2e-gce for 6h
  • is the job wildly underutilizing its memory limit? if so, perhaps tune down (if uncertain, post evidence in this issue and ask)
  • (it will probably be helpful to look at different time resolutions like 1h, 6h, 1d, 1w)

Things to watch for the build cluster

• prow-build dashboard 1w
  • is the build cluster scaling as needed? (e.g. maybe it can't scale because we've hit some kind of quota)
  • (it will probably be helpful to look at different time resolutions like 1h, 6h, 1d, 1w)
• prowjobs-experiment 1w
  • (shows resource consumption of all job runs, pretty noisy but putting this here for completeness)
• https://monitoring.prow.k8s.io/d/wSrfvNxWz/boskos-resource-usage?orgId=1
  • are there still plenty of free projects in the "GCE project (k8s-infra) usage status" graph?

Keep this open for at least 24h of weekday PR traffic. If everything continues to look good, then this can be closed.

/wg k8s-infra
/sig testing
/area jobs
/help

[spiffxp]

/remove-help
/assign

[spiffxp]

Tangentially related, it would be nice to know if we even need to use --stage=gs://kubernetes-release-pull (ref #18789). I already migrated over pull-kubernetes-e2e-gce-ubuntu-containerd which uses it, so I'll do the same here. But would then like to remove it if it's not needed, or migrate to kubernetes.io-owned gs://k8s-release-pull if it's needed

[spiffxp]

Opened #18916

The main branch and 1.19 variants aren't merge-blocking anymore, but earlier branches are. Moving them all over

[spiffxp]

#18916 merged 2020-08-19 16:40 PT

https://prow.k8s.io/?job=pull-kubernetes-e2e-gce - shows a reasonable amount of traffic since there is now a push to get PR's landed in time for the final cut of kubernetes v1.16. The only failures appear to be flakes

https://testgrid.k8s.io/presubmits-kubernetes-blocking#pull-kubernetes-e2e-gce&graph-metrics=test-duration-minutes - overall the job duration is less spiky and has maybe gone slightly down over time

https://storage.googleapis.com/k8s-gubernator/triage/index.html?pr=1&job=pull-kubernetes-e2e-gce%24 - no real change in errors

https://prow.k8s.io/job-history/gs/kubernetes-jenkins/pr-logs/directory/pull-kubernetes-e2e-gce - Seeing #19034, would like to understand whether this job caused that issue or something else

cpu utilization - big spikes in the beginning for build, then nothing

memory utilization - looks like that's about right

So if it turn out #19034 is unrelated to this change, we're good. But need to dig into that a little more first

[RobertKielty]

@spiffxp I this moved to In Progress. Will have a look at #19034 ...

[snowmanstark]

@RobertKielty @spiffxp I would like to work on this issue

[snowmanstark]

@spiffxp can you help me understand what does the following mean and how this affects the changes to be made for this issue

it's being demoted from merge-blocking on release-1.19 and the main branch (as of #18832)

[spiffxp]

@snowmanstark

So, the changes have already been made via #18916 (see #18852 (comment))

The reason this is still open is because #19034 is unexplained, and maybe happened around the same time #18916 merged? If we can either prove that #18916 didn't cause it (see #19034 (comment)), or if we can fix #19034, then this issue can be closed.

---

To answer your question

test-infra/config/tests/jobs/jobs_test.go

Lines 983 to 989 in 6e1d254

	// A job is merge-blocking if it:
	// - is not optional
	// - reports (aka does not skip reporting)
	// - always runs OR runs if some path changed
	func isMergeBlocking(job cfg.Presubmit) bool {
	return !job.Optional && !job.SkipReport && (job.AlwaysRun || job.RunIfChanged != "")
	}

#18832 set always_run to false for the main branch when v1.19 was under development, and the release-1.19 branch. There is no run_if_changed for it, thus it's not considered merge-blocking for those branches.

It is still merge blocking for older branches (release-1.18, release-1.17), as we generally don't backport policy or test changes back to already-released versions of kubernetes except under special circumstances.

The reason this complicates things is the job wouldn't see as much traffic as jobs that always run for all branches, so it's tougher to avoid variance due to a smaller sample-set size, and thus tougher to make a judgement call on "does everything still look OK."

However, I saw enough traffic in #18852 (comment) when cherry picks were being swept through in advance of upcoming patch releases. So aside from the question of #19034 I think this looks good

[snowmanstark]

Thanks @spiffxp for that explanation. It makes total sense to me now. I'll look into #19034 too to get this closed.

[snowmanstark]

@spiffxp I looked into #19034 and nothings seems to be off there.

[RobertKielty]

Hi @spiffxp can this issue be closed now?

I've updated #19034

We need to review #19034 for sure but I'm confused as to how both these issues are related?

[spiffxp]

Per #18852 (comment) the reason I held this open is because I'm still not certain that migration of this job did not cause #19034. But we've lived with it unresolved for about 90d now, so I guess we can live with it unexplained for longer.

/close

[k8s-ci-robot]

@spiffxp: Closing this issue.

In response to this:

Per #18852 (comment) the reason I held this open is because I'm still not certain that migration of this job did not cause #19034. But we've lived with it unresolved for about 90d now, so I guess we can live with it unexplained for longer.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.