[release-1.22] pull-kubernetes-integration is failing

[ialidzhikov]

Which jobs are failing:

pull-kubernetes-integration

See the runs related to #105139 - https://prow.k8s.io/pr-history/?org=kubernetes&repo=kubernetes&pr=105139

Which test(s) are failing:

Not clear. Even for some of the runs I don't see a particular test failure.

Since when has it been failing:

Not clear.

Testgrid link:

See the runs related to #105139 - https://prow.k8s.io/pr-history/?org=kubernetes&repo=kubernetes&pr=105139

Reason for failure:

Not clear.

Relevant SIG

/sig testing

[ehashman]

/priority critical-urgent
/triage accepted

https://storage.googleapis.com/k8s-triage/index.html?ci=0&pr=1&job=pull-kubernetes-integration

#105452 (comment)

[spiffxp]

Tough to confirm when this started since there's no easy mechanism to filter history for presubmits by base branch

I'm trying to script something together to scrape GCS, so far I see the following have had failed pull-kubernetes-integration runs

• Automated cherry pick of #104794: pkg/kubelet/cm/memorymanager: Fix ErrorS key/value pair #105139
• Automated cherry pick of #103855: Update Containerd version - GCE Windows #105331
• Automated cherry pick of #105048: Remove a duplicate StorageClass creation call #105355
• Automated cherry pick of #105188: fix: consolidate logs for instance not found error #105366
• Automated cherry pick of #105396: Fix CSR test to accept certs shorter than the requested #105397
• Automated cherry pick of #101063: tests: Spawn poststart / prestop pods on the same node as the #105444
• Automated cherry pick of #105352: Revert "Build non-static binaries with PIE buildmode" #105452

[spiffxp]

Maybe related, we saw the presubmit timing out a bunch back in July, seems like that mostly stopped by 2020-07-09. But we never closed out investigating, and I suspect these are timeouts based on spyglass lying (it's UI shows pass but the job is failing), ref: #103512

[ehashman]

Cold this be networking related?

W1004 19:01:13.392630  118232 reflector.go:441] k8s.io/client-go/informers/factory.go:134: watch of *v1beta1.PriorityLevelConfiguration ended with: an error on the server ("unable to decode a
n event from the watch stream: http2: client connection lost") has prevented the request from succeeding
E1004 19:01:07.379813  118232 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Endpoints: Get "https://127.0.0.1:44191/api/v1/endpoints?allowWatchBookmarks=tru
e&resourceVersion=19603&timeout=6m30s&timeoutSeconds=390&watch=true": http2: client connection lost
E1004 19:01:22.793215  118232 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.ServiceAccount: Get "https://127.0.0.1:42523/api/v1/serviceaccounts?allowWatchBo
okmarks=true&resourceVersion=20170&timeout=5m19s&timeoutSeconds=319&watch=true": http2: client connection lost
E1004 19:01:16.434912  118232 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.ServiceAccount: Get "https://127.0.0.1:44191/api/v1/serviceaccounts?allowWatchBo
okmarks=true&resourceVersion=19610&timeout=7m57s&timeoutSeconds=477&watch=true": http2: client connection lost
W1004 19:01:14.622372  118232 reflector.go:441] k8s.io/kube-aggregator/pkg/client/informers/externalversions/factory.go:117: watch of *v1.APIService ended with: an error on the server ("unabl
e to decode an event from the watch stream: http2: client connection lost") has prevented the request from succeeding
E1004 19:01:37.310019  118232 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Pod: Get "https://127.0.0.1:35693/api/v1/pods?allowWatchBookmarks=true&resourceV
ersion=24561&timeout=5m33s&timeoutSeconds=333&watch=true": http2: client connection lost
E1004 19:01:25.106278  118232 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.ValidatingWebhookConfiguration: Get "https://127.0.0.1:43013/apis/admissionregis
tration.k8s.io/v1/validatingwebhookconfigurations?allowWatchBookmarks=true&resourceVersion=18618&timeout=5m3s&timeoutSeconds=303&watch=true": http2: client connection lost
E1004 19:01:33.548156  118232 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Service: Get "https://127.0.0.1:37093/api/v1/services?allowWatchBookmarks=true&r
esourceVersion=21826&timeout=6m39s&timeoutSeconds=399&watch=true": http2: client connection lost
E1004 19:01:33.127381  118232 reflector.go:138] k8s.io/kubernetes/pkg/controlplane/controller/clusterauthenticationtrust/cluster_authentication_trust_controller.go:444: Failed to watch *v1.Co
nfigMap: Get "https://127.0.0.1:45715/api/v1/namespaces/kube-system/configmaps?allowWatchBookmarks=true&resourceVersion=818&timeout=8m30s&timeoutSeconds=510&watch=true": http2: client connect
ion lost
E1004 19:01:43.283532  118232 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.ValidatingWebhookConfiguration: Get "https://127.0.0.1:37093/apis/admissionregis
tration.k8s.io/v1/validatingwebhookconfigurations?allowWatchBookmarks=true&resourceVersion=21114&timeout=7m16s&timeoutSeconds=436&watch=true": http2: client connection lost
E1004 19:01:41.620012  118232 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.MutatingWebhookConfiguration: Get "https://127.0.0.1:43031/apis/admissionregistr
ation.k8s.io/v1/mutatingwebhookconfigurations?allowWatchBookmarks=true&resourceVersion=19493&timeout=7m1s&timeoutSeconds=421&watch=true": http2: client connection lost
E1004 19:01:49.585794  118232 reflector.go:138] k8s.io/kube-aggregator/pkg/client/informers/externalversions/factory.go:117: Failed to watch *v1.APIService: Get "https://127.0.0.1:43013/apis/
apiregistration.k8s.io/v1/apiservices?allowWatchBookmarks=true&resourceVersion=18814&timeout=7m42s&timeoutSeconds=462&watch=true": http2: client connection lost
W1004 19:01:42.463442  118232 reflector.go:441] k8s.io/client-go/informers/factory.go:134: watch of *v1.Namespace ended with: an error on the server ("unable to decode an event from the watch
 stream: http2: client connection lost") has prevented the request from succeeding
E1004 19:01:45.587331  118232 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.ServiceAccount: Get "https://127.0.0.1:39851/api/v1/serviceaccounts?allowWatchBo
okmarks=true&resourceVersion=13750&timeout=5m17s&timeoutSeconds=317&watch=true": http2: client connection lost
W1004 19:01:59.024099  118232 reflector.go:441] k8s.io/client-go/informers/factory.go:134: watch of *v1beta1.FlowSchema ended with: an error on the server ("unable to decode an event from the
 watch stream: http2: client connection lost") has prevented the request from succeeding
W1004 19:02:01.132622  118232 clientconn.go:1326] [core] grpc: addrConn.createTransport failed to connect to {127.0.0.1:2379 127.0.0.1 <nil> 0 <nil>}. Err: connection error: desc = "transport
: Error while dialing dial tcp 127.0.0.1:2379: i/o timeout". Reconnecting...
W1004 19:01:54.388525  118232 reflector.go:441] k8s.io/client-go/informers/factory.go:134: watch of *v1.StorageClass ended with: an error on the server ("unable to decode an event from the wa
tch stream: http2: client connection lost") has prevented the request from succeeding
SIGQUIT: quit
PC=0x475ee1 m=0 sigcode=0

<giant stacktrace>

from https://storage.googleapis.com/kubernetes-jenkins/pr-logs/pull/105452/pull-kubernetes-integration/1445094410348924928/build-log.txt (note filesize is 114MB)

[ehashman]

It looks like === FAIL: test/integration/apiserver/admissionwebhook TestWebhookTimeoutWithoutWatchCache (unknown) is timing out and everything afterwards is breaking

build-log.txt

This is a reasonable sized snippet of the 110+ MB build log from https://storage.googleapis.com/kubernetes-jenkins/pr-logs/pull/105452/pull-kubernetes-integration/1445094410348924928/build-log.txt

[ehashman]

/sig api-machinery
cc @kubernetes/sig-api-machinery-bugs