[job failure] kubeadm-gce

[spiffxp]

/priority critical-urgent
/priority failing-test
/kind bug
/status approved-for-milestone
@kubernetes/sig-cluster-lifecycle-test-failures

This job has been failing since 2017-11-15. It's on the sig-release-master-blocking dashboard,
and prevents us from cutting [v1.9.0-beta.1] (kubernetes/sig-release#34). Is there work ongoing to bring this job back to green?

https://k8s-testgrid.appspot.com/sig-release-master-blocking#kubeadm-gce

• last good: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-kubeadm-gce/8585
• first bad: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-kubeadm-gce/8586
• latest bad: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-kubeadm-gce/8795

[dims]

@luxas @spiffxp here's the tentative plan - kubernetes/kubeadm#256 (comment)

[luxas]

@spiffxp yes, the PR that fixed the root cause (kubernetes-retired/kubernetes-anywhere#483) just merged. Sorry for the long approving window there. Now we're about to bump the git sha pinning for the job: https://github.com/kubernetes/test-infra/blob/master/images/kubeadm/runner#L29 to make the test green

[luxas]

Status update: kubernetes/test-infra#5610 went in yesterday
The actual test-infra image bump got merged now kubernetes/test-infra#5636, so the tests should start passing soon-ish

[xiangpengzhao]

The https://k8s-testgrid.appspot.com/sig-release-master-blocking#kubeadm-gce still fails after kubernetes/test-infra#5636 merged.

https://storage.googleapis.com/kubernetes-jenkins/logs/ci-kubernetes-e2e-kubeadm-gce/8906/artifacts/e2e-8906-node-vcw7/startupscript.log
shows:

Nov 23 05:13:24 ubuntu startupscript: kubeadm join --token "$KUBEADM_TOKEN" "$KUBEADM_MASTER_IP:443" --skip-preflight-checks --discovery-token-unsafe-skip-ca-verification true
Nov 23 05:13:24 ubuntu startupscript: + kubeadm join --token 043479.1a6ec0adafb593d0 10.128.0.2:443 --skip-preflight-checks --discovery-token-unsafe-skip-ca-verification true
Nov 23 05:13:25 ubuntu startupscript: Flag --skip-preflight-checks has been deprecated, it is now equivalent to --ignore-checks-errors=all
Nov 23 05:13:25 ubuntu startupscript: [kubeadm] WARNING: kubeadm is currently in beta
Nov 23 05:13:25 ubuntu startupscript: [preflight] Running pre-flight checks.
Nov 23 05:13:25 ubuntu startupscript: #011[WARNING FileExisting-crictl]: crictl not found in system path
Nov 23 05:13:25 ubuntu startupscript: [validation] WARNING: kubeadm doesn't fully support multiple API Servers yet
Nov 23 05:13:25 ubuntu startupscript: discovery: Invalid value: "true": address true: missing port in address
Nov 23 05:13:25 ubuntu startupscript: Finished running startup script /var/run/google.startup.script

Seems like param --discovery-token-unsafe-skip-ca-verification true is incorrect. Just pass --discovery-token-unsafe-skip-ca-verification would be OK. I will fix it.

I have a question here, for such bool type param, will only this format with just the param name means true, without the param defaults to false or it's default value be allowed?

[luxas]

Yes, only the param will mean true by default. Thanks for fixing!

[xiangpengzhao]

This needs two follow-up PRs like kubernetes/test-infra#5610 and kubernetes/test-infra#5636 after kubernetes-retired/kubernetes-anywhere#488 gets merged.

[xiangpengzhao]

waiting @pipejakob for kubernetes-retired/kubernetes-anywhere#488 merging.

[spiffxp]

Now waiting on kubernetes-retired/kubernetes-anywhere#491

[k8s-github-robot]

[MILESTONENOTIFIER] Milestone Issue Current

@dims @luxas @spiffxp @xiangpengzhao

Note: This issue is marked as priority/critical-urgent, and must be updated every 1 day during code freeze.

Example update:

ACK.  In progress
ETA: DD/MM/YYYY
Risks: Complicated fix required

Issue Labels

• sig/cluster-lifecycle: Issue will be escalated to these SIGs if needed.
• priority/critical-urgent: Never automatically move issue out of a release milestone; continually escalate to contributor and SIG through all available channels.
• kind/bug: Fixes a bug discovered during the current release.

Help

• Additional instructions
• Commands for setting labels

[luxas]

I merged the required PRs and updated test-infra together with @BenTheElder, hopefully we should start seeing green runs now :)

[xiangpengzhao]

https://k8s-testgrid.appspot.com/sig-release-master-blocking#kubeadm-gce is green now. The node log shows that the flag is passed correctly. Thanks @shashidharatd ! Awesome!

[luxas]

Closing this as the job is now green