Windows kubelet stats timeout updates #87730
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
k8s-ci-robot
added
release-note
|
/sig windows /test pull-kubernetes-e2e-aks-engine-azure-windows |
k8s-ci-robot
added
the
sig/windows
k8s-ci-robot
added
area/kubelet
area/test
sig/node
PatrickLang
reviewed
Jan 31, 2020
|
/retest pull-kubernetes-integration |
|
/milestone v1.18 |
|
I'm testing a custom build with these changes plus #86101 since it also depends on metrics :) |
|
/assign @yliaog |
|
/assign @benmoss |
yliaog
reviewed
Feb 1, 2020
|
/lgtm |
liggitt
reviewed
Feb 6, 2020
test/e2e/windows/kubelet_stats.go
Outdated
| } | ||
|
|
||
| if foundNode == false { | ||
| framework.Skipf("Could not find and ready and schedulable Windows nodes") |
There was a problem hiding this comment.
this is failing batch merge (see https://prow.k8s.io/view/gcs/kubernetes-jenkins/pr-logs/pull/batch/pull-kubernetes-bazel-build/1225465355036528642)
There was a problem hiding this comment.
Looks like skipf got moved from framework to framework/skipper with this commit
641321c
I'll rebase and push an update
|
/test pull-kubernetes-bazel-build |
…let stats for windows nodes
…atly reduce latency
marosset
force-pushed
the
windows-kubelet-stats-timeout-updates
branch
from
e54fd36
to
999fdfa
Compare
k8s-ci-robot
removed
the
lgtm
|
@marosset: The following test failed, say
Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here. |
|
/retest |
|
/lgtm |
k8s-ci-robot
added
the
lgtm
This was referenced Feb 24, 2020
This was referenced Mar 2, 2020
k8s-ci-robot
added a commit
that referenced
this pull request
Apr 7, 2020
…30-upstream-release-1.15 Automated cherry pick of #87730 upstream release 1.15
k8s-ci-robot
added a commit
that referenced
this pull request
Apr 7, 2020
…30-upstream-release-1.17 Automated cherry pick of #87730 upstream release 1.17
k8s-ci-robot
added a commit
that referenced
this pull request
Apr 7, 2020
…30-upstream-release-1.16 Automated cherry pick of #87730 upstream release 1.16
vboulineau
added a commit
to vboulineau/kubernetes
that referenced
this pull request
May 4, 2020
…s are present on the node Following changes in kubernetes#87730, Kubelet is directly hcsshim to gather stats. However, unlike `docker stats` API that was used before, hcsshim does not keep information about exited containers. When the Kubelet lists containers (`docker_container.go:ListContainers()`), it sets `All: true`, retrieving non-running containers. When docker stats is called with such container id, it'll return a valid JSON with all values set to 0. The non-running containers are filtered later on in the process. When the hcsshim is called with such container id, it'll return an error, effectively stopping the stats retrieval for all containers.
vboulineau
added a commit
to vboulineau/kubernetes
that referenced
this pull request
May 14, 2020
…s are present on the node Following changes in kubernetes#87730, Kubelet is directly hcsshim to gather stats. However, unlike `docker stats` API that was used before, hcsshim does not keep information about exited containers. When the Kubelet lists containers (`docker_container.go:ListContainers()`), it sets `All: true`, retrieving non-running containers. When docker stats is called with such container id, it'll return a valid JSON with all values set to 0. The non-running containers are filtered later on in the process. When the hcsshim is called with such container id, it'll return an error, effectively stopping the stats retrieval for all containers.
vboulineau
added a commit
to vboulineau/kubernetes
that referenced
this pull request
May 14, 2020
…s are present on the node Following changes in kubernetes#87730, Kubelet is directly hcsshim to gather stats. However, unlike `docker stats` API that was used before, hcsshim does not keep information about exited containers. When the Kubelet lists containers (`docker_container.go:ListContainers()`), it sets `All: true`, retrieving non-running containers. When docker stats is called with such container id, it'll return a valid JSON with all values set to 0. The non-running containers are filtered later on in the process. When the hcsshim is called with such container id, it'll return an error, effectively stopping the stats retrieval for all containers.
vboulineau
added a commit
to vboulineau/kubernetes
that referenced
this pull request
May 14, 2020
…s are present on the node Following changes in kubernetes#87730, Kubelet is directly hcsshim to gather stats. However, unlike `docker stats` API that was used before, hcsshim does not keep information about exited containers. When the Kubelet lists containers (`docker_container.go:ListContainers()`), it sets `All: true`, retrieving non-running containers. When docker stats is called with such container id, it'll return a valid JSON with all values set to 0. The non-running containers are filtered later on in the process. When the hcsshim is called with such container id, it'll return an error, effectively stopping the stats retrieval for all containers.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What type of PR is this?
/kind bug
What this PR does / why we need it:
This PR addresses an issue where kubelet metrics call take a very long time on Windows nodes if more than a handful of containers are running.
Which issue(s) this PR fixes:
Fixes Stats performance is slow on Windows #74991
Special notes for your reviewer:
Does this PR introduce a user-facing change?:
Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.: