Tracking: dns-app update timeout may not be long enough

[knisbet]

Description

What happened:
Under high scheduler load / long scheduling delays, it appears possible that the coredns upgrade can timeout before the new pods have been scheduled. We need to validate/double check the timers on the dns-app upgrades are long enough to tolerate longer than expected scheduling from kubernetes.

[knisbet]

Example log of a suspexted failure that is related... from the dns update job

2020-11-11T21:31:22Z INFO             found pod kube-system/coredns-f9262 on node 10.0.231.32 daemonset:kube-system/coredns rigging/utils.go:117
2020-11-11T21:31:22Z INFO             found pod kube-system/coredns-fdt8v on node 10.0.71.19 daemonset:kube-system/coredns rigging/utils.go:117
2020-11-11T21:31:22Z INFO             found pod kube-system/coredns-rx8zb on node 10.0.247.64 daemonset:kube-system/coredns rigging/utils.go:117
2020-11-11T21:31:22Z INFO             node 10.0.231.32: pod kube-system/coredns-f9262 is up and running daemonset:kube-system/coredns rigging/utils.go:196
2020-11-11T21:31:22Z ERRO             "
ERROR REPORT:
Original Error: *trace.BadParameterError namespace=default, name=dns-031, operations=292) is not completed yet
Stack Trace:
	/gopath/src/github.com/gravitational/rigging/changeset.go:193 github.com/gravitational/rigging.(*Changeset).Status.func1
	/gopath/src/github.com/gravitational/rigging/utils.go:137 github.com/gravitational/rigging.retry
	/gopath/src/github.com/gravitational/rigging/changeset.go:189 github.com/gravitational/rigging.(*Changeset).Status
	/gopath/src/github.com/gravitational/rigging/tool/rig/main.go:292 main.status
	/gopath/src/github.com/gravitational/rigging/tool/rig/main.go:124 main.run
	/gopath/src/github.com/gravitational/rigging/tool/rig/main.go:31 main.main
	/go/src/runtime/proc.go:207 runtime.main
	/go/src/runtime/asm_amd64.s:2362 runtime.goexit
User Message: namespace=default, name=dns-031, operations=292) is not completed yet
" rig/main.go:31
ERROR: namespace=default, name=dns-031, operations=292) is not completed yet
[root@ip-10-0-231-32 installer]# 2020-11-11T21:31:17Z INFO             "attempt 116, result:
ERROR REPORT:
Original Error: *trace.BadParameterError namespace=default, name=dns-031, operations=292) is not completed yet
Stack Trace:
	/gopath/src/github.com/gravitational/rigging/changeset.go:193 github.com/gravitational/rigging.(*Changeset).Status.func1
	/gopath/src/github.com/gravitational/rigging/utils.go:137 github.com/gravitational/rigging.retry
	/gopath/src/github.com/gravitational/rigging/changeset.go:189 github.com/gravitational/rigging.(*Changeset).Status
	/gopath/src/github.com/gravitational/rigging/tool/rig/main.go:292 main.status
	/gopath/src/github.com/gravitational/rigging/tool/rig/main.go:124 main.run
	/gopath/src/github.com/gravitational/rigging/tool/rig/main.go:31 main.main
	/go/src/runtime/proc.go:207 runtime.main
	/go/src/runtime/asm_amd64.s:2362 runtime.goexit
User Message: namespace=default, name=dns-031, operations=292) is not completed yet
, retry in 1s" rigging/utils.go:129

[knisbet]

Merged on each release branch, pending releases.