kubelet: pod metrics not visible with Docker 18 (19.03 works) #94281

pmyjavec · 2020-08-27T13:59:33Z

What happened:

When deploying a cluster on v1.19.0 using kubeadm, we're no longer seeing kubelet reporting pod metrics from the node, this breaking metrics-server to some degree as we're missing pod statistics:

{                    
 "node": {               
  "nodeName": "k8s-node-0",
  "systemContainers": [
   {
    "name": "kubelet",                
    "startTime": "2020-08-27T11:17:49Z",
    "cpu": {                      
     "time": "2020-08-27T12:19:43Z",
     "usageNanoCores": 18925650,        
     "usageCoreNanoSeconds": 71340541215
    },       
    "memory": {                   
     "time": "2020-08-27T12:19:43Z",
     "usageBytes": 40591360,
     "workingSetBytes": 39964672,
     "rssBytes": 39510016,
     "pageFaults": 345372,
     "majorPageFaults": 10
    }
   },
   {       
    "name": "runtime",                 
    "startTime": "2020-08-27T04:23:40Z",
    "cpu": {
     "time": "2020-08-27T12:19:51Z",
     "usageNanoCores": 13500270,
     "usageCoreNanoSeconds": 339260029980
    },
    "memory": {
     "time": "2020-08-27T12:19:51Z",
     "usageBytes": 686555136,
     "workingSetBytes": 172347392,
     "rssBytes": 58580992,
     "pageFaults": 349662,
     "majorPageFaults": 309
    }
   },
   {
    "name": "runtime",
    "startTime": "2020-08-27T04:23:40Z",
    "cpu": {
     "time": "2020-08-27T12:19:51Z",
     "usageNanoCores": 13500270,
     "usageCoreNanoSeconds": 339260029980
    },
    "memory": {
     "time": "2020-08-27T12:19:51Z",
     "usageBytes": 686555136,
     "workingSetBytes": 172347392,
     "rssBytes": 58580992,
     "pageFaults": 349662,
     "majorPageFaults": 309
    }
   },
   {
    "name": "pods",
    "startTime": "2020-08-27T04:23:40Z",
    "cpu": {
     "time": "2020-08-27T12:19:48Z",
     "usageNanoCores": 53850241,
     "usageCoreNanoSeconds": 1162308046776
    },
    "memory": {
     "time": "2020-08-27T12:19:48Z",
     "availableBytes": 16232984576,
     "usageBytes": 168726528,
     "workingSetBytes": 107012096,
     "rssBytes": 36741120,
     "pageFaults": 0,
     "majorPageFaults": 0
    }
   }
  ],
  "startTime": "2020-08-27T04:22:56Z",
  "cpu": {
   "time": "2020-08-27T12:19:48Z",
   "usageNanoCores": 85723273,
   "usageCoreNanoSeconds": 2118444201905
  },
  "memory": {
   "time": "2020-08-27T12:19:48Z",
   "availableBytes": 15762960384,
   "usageBytes": 1376034816,
   "workingSetBytes": 577036288,
   "rssBytes": 213475328,
   "pageFaults": 38512,
   "majorPageFaults": 564
  }
 },
 "pods": []

Note the empty pods section

What you expected to happen:

We'd see a populated list of pod metrics as we did before that looks like:

 "pods": [                                                                                                                                                                                                
  {                                                                                                                                                                                                       
   "podRef": {                                                                                       
    "name": "metrics-server-64b57fd654-bcmlv",                                                       
    "namespace": "kube-system",                   
    "uid": "8f08ed66-2f6f-4ded-89d7-b49a717f2127"
   },                                                                                                                                                                                                     
   "startTime": "2020-08-27T11:53:27Z",
   "containers": [
    {
     "name": "metrics-server",
     "startTime": "2020-08-27T11:53:28Z",
     "cpu": {
      "time": "2020-08-27T12:23:00Z",
      "usageNanoCores": 178862,
      "usageCoreNanoSeconds": 1451974591
     },
     "memory": {
      "time": "2020-08-27T12:23:00Z",
      "usageBytes": 42545152,
      "workingSetBytes": 15671296,
      "rssBytes": 12148736,
      "pageFaults": 5633,
      "majorPageFaults": 223
     }
    }
   ],
   "cpu": {
    "time": "2020-08-27T12:23:04Z",
    "usageNanoCores": 348505,
    "usageCoreNanoSeconds": 1496841316
   },
   "memory": {
    "time": "2020-08-27T12:23:04Z",
    "usageBytes": 44687360,
    "workingSetBytes": 17813504,
    "rssBytes": 12185600,
    "pageFaults": 0,
    "majorPageFaults": 0
   }
  }]

How to reproduce it (as minimally and precisely as possible):

Use kubeadm init on 1.19.0 to create a new cluster and query the stats endpoint using:

curl -X GET  https://10.250.0.5:10250/stats/summary?only_cpu_and_memory=true --header "Authorization: Bearer $TOKEN" --insecure

We can ran the same configuration on v1.18.2 and didn't hit the issue.

Anything else we need to know?:

Environment:

Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.0", GitCommit:"e19964183377d0ec2052d1f1fa930c4d7575bd50", GitTreeState:"clean", BuildDate:"2020-08-26T14:30:33Z", GoVersion:"go1.15", Compiler:"gc", Platform:"linux/amd64"}

Cloud provider or hardware configuration:

  AWS EC2 Instance: c5.2xlarge

OS (e.g: cat /etc/os-release):

NAME="Ubuntu"
VERSION="16.04.7 LTS (Xenial Xerus)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 16.04.7 LTS"
VERSION_ID="16.04"
HOME_URL="http://www.ubuntu.com/"
SUPPORT_URL="http://help.ubuntu.com/"
BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"
VERSION_CODENAME=xenial
UBUNTU_CODENAME=xenial

Kernel (e.g. uname -a):

Linux k8s-master-0 4.4.0-1109-aws #120-Ubuntu SMP Fri Jun 5 01:26:57 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

Install tools: deb
Network plugin and version (if this is a network-related bug):
Others:
N/A

The text was updated successfully, but these errors were encountered:

pmyjavec · 2020-08-27T14:01:07Z

/sig node
/sig instrumentation

pmyjavec · 2020-08-27T14:18:37Z

We've narrowed this down to it being a problem with Docker 18, it looks like once we upgraded to Docker 19.03, please feel free to close this off if it's not longer relevant.

I'm wondering if this would be better documented somewhere? Maybe it is and we missed it?

Thanks.

neolit123 · 2020-08-27T14:23:08Z

hi,

We can ran the same configuration on v1.18.2 and didn't hit the issue.

^ you've mentioned in the OP that this does not happen in 1.18.2.

We've narrowed this down to it being a problem with Docker 18, it looks like once we upgraded to Docker 19.03, please feel free to close this off if it's not longer relevant.

to clarify, with 1.19.0 and Docker 18 the problem happens, but not with 1.19 and Docker 19.03?

pmyjavec · 2020-08-27T14:29:01Z

@neolit123,

to clarify, with 1.19.0 and Docker 18 the problem happens, but not with 1.19 and Docker 19.03?

Yes sorry, if that wasn't clear.

neolit123 · 2020-08-27T14:30:18Z

ok, this is very odd. hopefully someone from SIG node sees this.
if you want to raise attention you could post in the #sig-node channel on k8s slack.

neolit123 · 2020-08-27T14:31:44Z

/retitle kubelet: pod metrics not visible with Docker 18 (19.03 works)

serathius · 2020-08-27T14:57:41Z

/cc @dashpole

ialidzhikov · 2020-09-01T10:29:18Z

I can also confirm that issue reproduces with metrics-server@v0.3.7, kubernetes@v1.19.0 and docker://18.6.3.

$ k -n kube-system top po
W0901 13:13:52.018978   35484 top_pod.go:274] Metrics not available for pod kube-system/calico-kube-controllers-68657c7d94-727n2, age: 2m25.018968s
error: Metrics not available for pod kube-system/calico-kube-controllers-68657c7d94-727n2, age: 2m25.018968s


$ k get no -o wide
NAME                                        STATUS   ROLES    AGE     VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE                                    KERNEL-VERSION    CONTAINER-RUNTIME
ip-10-222-0-22.eu-west-1.compute.internal   Ready    <none>   80s     v1.19.0   10.222.0.22   <none>        Container Linux by CoreOS 2512.3.0 (Oklo)   4.19.123-coreos   docker://18.6.3
ip-10-222-0-37.eu-west-1.compute.internal   Ready    <none>   4m32s   v1.19.0   10.222.0.37   <none>        Container Linux by CoreOS 2512.3.0 (Oklo)   4.19.123-coreos   docker://18.6.3

$ k -n kube-system logs metrics-server-68498c577c-bvtpc
I0901 10:12:21.118498       1 manager.go:95] Scraping metrics from 0 sources
I0901 10:12:21.118580       1 manager.go:148] ScrapeMetrics: time: 908ns, nodes: 0, pods: 0
I0901 10:12:21.221635       1 secure_serving.go:116] Serving securely on [::]:8443
E0901 10:13:19.026582       1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/coredns-97dfb4f59-zht6w: no metrics known for pod
E0901 10:13:19.026615       1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/coredns-97dfb4f59-x4wfv: no metrics known for pod
I0901 10:13:21.118663       1 manager.go:95] Scraping metrics from 2 sources
I0901 10:13:21.126702       1 manager.go:120] Querying source: kubelet_summary:ip-10-222-0-22.eu-west-1.compute.internal
I0901 10:13:21.130055       1 manager.go:120] Querying source: kubelet_summary:ip-10-222-0-37.eu-west-1.compute.internal
I0901 10:13:21.189014       1 manager.go:148] ScrapeMetrics: time: 70.170272ms, nodes: 2, pods: 0
E0901 10:13:49.538888       1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/coredns-97dfb4f59-x4wfv: no metrics known for pod
E0901 10:13:49.538906       1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/coredns-97dfb4f59-zht6w: no metrics known for pod
E0901 10:13:51.747200       1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/node-problem-detector-7hqhq: no metrics known for pod
E0901 10:13:51.747219       1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/csi-driver-node-ps2xk: no metrics known for pod
E0901 10:13:51.747224       1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/node-problem-detector-c5bdf: no metrics known for pod
E0901 10:13:51.747229       1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/coredns-97dfb4f59-zht6w: no metrics known for pod
E0901 10:13:51.747234       1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/calico-typha-vertical-autoscaler-55b85db9c9-9p9px: no metrics known for pod
E0901 10:13:51.747239       1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/coredns-97dfb4f59-x4wfv: no metrics known for pod
E0901 10:13:51.747244       1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/calico-typha-deploy-7bc68bcd86-9ln87: no metrics known for pod
E0901 10:13:51.747249       1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/calico-kube-controllers-68657c7d94-727n2: no metrics known for pod
E0901 10:13:51.747254       1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/kube-proxy-2sgdh: no metrics known for pod
E0901 10:13:51.747259       1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/calico-node-lkrdd: no metrics known for pod
E0901 10:13:51.747264       1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/csi-driver-node-bctws: no metrics known for pod
E0901 10:13:51.747268       1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/calico-node-g8bvp: no metrics known for pod
E0901 10:13:51.747272       1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/kube-proxy-tghqn: no metrics known for pod
E0901 10:13:51.747278       1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/calico-typha-horizontal-autoscaler-86cfb97885-8462k: no metrics known for pod
E0901 10:13:51.747283       1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/vpn-shoot-7666fbf977-hnljr: no metrics known for pod
E0901 10:13:51.747287       1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/metrics-server-68498c577c-bvtpc: no metrics known for pod
E0901 10:13:51.747291       1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/calico-node-vertical-autoscaler-5788bdd9cb-vwjjs: no metrics known for pod
E0901 10:13:53.360448       1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/csi-driver-node-bctws: no metrics known for pod
E0901 10:13:53.360463       1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/calico-typha-vertical-autoscaler-55b85db9c9-9p9px: no metrics known for pod
E0901 10:13:53.360467       1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/coredns-97dfb4f59-x4wfv: no metrics known for pod
E0901 10:13:53.360470       1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/calico-typha-deploy-7bc68bcd86-9ln87: no metrics known for pod
E0901 10:13:53.360474       1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/calico-kube-controllers-68657c7d94-727n2: no metrics known for pod
E0901 10:13:53.360477       1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/kube-proxy-2sgdh: no metrics known for pod
E0901 10:13:53.360480       1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/calico-node-lkrdd: no metrics known for pod
E0901 10:13:53.360483       1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/kube-proxy-tghqn: no metrics known for pod
E0901 10:13:53.360498       1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/calico-node-g8bvp: no metrics known for pod
E0901 10:13:53.360501       1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/calico-node-vertical-autoscaler-5788bdd9cb-vwjjs: no metrics known for pod
E0901 10:13:53.360514       1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/calico-typha-horizontal-autoscaler-86cfb97885-8462k: no metrics known for pod
E0901 10:13:53.360517       1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/vpn-shoot-7666fbf977-hnljr: no metrics known for pod
E0901 10:13:53.360521       1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/metrics-server-68498c577c-bvtpc: no metrics known for pod
E0901 10:13:53.360524       1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/coredns-97dfb4f59-zht6w: no metrics known for pod
E0901 10:13:53.360527       1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/node-problem-detector-7hqhq: no metrics known for pod
E0901 10:13:53.360530       1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/csi-driver-node-ps2xk: no metrics known for pod
E0901 10:13:53.360533       1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/node-problem-detector-c5bdf: no metrics known for pod
E0901 10:14:20.051957       1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/coredns-97dfb4f59-x4wfv: no metrics known for pod
E0901 10:14:20.051978       1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/coredns-97dfb4f59-zht6w: no metrics known for pod
I0901 10:14:21.118610       1 manager.go:95] Scraping metrics from 2 sources
I0901 10:14:21.129795       1 manager.go:120] Querying source: kubelet_summary:ip-10-222-0-37.eu-west-1.compute.internal
I0901 10:14:21.133783       1 manager.go:120] Querying source: kubelet_summary:ip-10-222-0-22.eu-west-1.compute.internal
I0901 10:14:21.146267       1 manager.go:148] ScrapeMetrics: time: 27.627598ms, nodes: 2, pods: 0

/priority important-soon

dashpole · 2020-09-11T17:17:56Z

Can anyone grab metrics from the kubelet on one of those nodes?

saul-data · 2020-11-02T21:10:39Z

I am experiencing the same issue with new Kubernetes 1.19 and docker 18.9.9 update on Digitalocean

(base) ➜  ~ kubectl -n kube-system top po
W1102 21:03:51.720671   71950 top_pod.go:266] Metrics not available for pod kube-system/cilium-2g625, age: 31h31m35.72066s
error: Metrics not available for pod kube-system/cilium-2g625, age: 31h31m35.72066s

(base) ➜  ~ kubectl get no -o wide
NAME              STATUS   ROLES    AGE   VERSION   INTERNAL-IP   EXTERNAL-IP      OS-IMAGE                       KERNEL-VERSION    CONTAINER-RUNTIME
xxx   Ready    <none>   31h   v1.19.3   10.106.xxx    178.128.xxx   Debian GNU/Linux 10 (buster)   4.19.0-11-amd64   docker://18.9.9
xxx  Ready    <none>   32h   v1.19.3   10.106.xxx    46.101.xxx    Debian GNU/Linux 10 (buster)   4.19.0-11-amd64   docker://18.9.9

ghouscht · 2020-11-03T08:35:03Z

Same issue here on our cluster running k8s 1.19.3 with docker 18.9.8. node metrics still work, but there are no pod metrics:

#> k get --raw /apis/metrics.k8s.io/v1beta1/pods
{"kind":"PodMetricsList","apiVersion":"metrics.k8s.io/v1beta1","metadata":{"selfLink":"/apis/metrics.k8s.io/v1beta1/pods"},"items":[]}

On a different cluster also running k8s 1.19.3 with docker 19.3.12 the pod and node metrics are ok.

Edit:
Had a quick look at the kubelet stats api and as far as I can tell the code there seems to be ok. So this is probably related to the docker client being updated by commit: 8e8e153. As the commit states, this points to the tag 19.03.8 in docker/docker.

This led me to check the kubelet logs and look what I found there:

Nov 03 10:08:21 e1-k8s-alsu120 kubelet[31240]: I1103 10:08:21.689487 31240 factory.go:161] Registration of the docker container factory failed: failed to validate Docker info: failed to detect Docker info: Error response from daemon: client version 1.40 is too new. Maximum supported API version is 1.39

So the kubelet expects a docker server with api version >=1.40 which was introduced with docker 19 (docker 18 is on api version 1.39). I always thaught that docker 18 is still supported by kubernetes but probably it's worth upgrading docker now... Anyway I'm not 100% confident that this is caused by the referenced commit, but at least it seems possible to me.

The Kubernetes release notes list which versions of Docker are compatible with that version of Kubernetes.
https://kubernetes.io/docs/setup/production-environment/container-runtimes/#docker

Can't find such an info in https://kubernetes.io/docs/setup/release/notes/ can someone clarify which versions of docker are supported by k8s 1.19.X?

Edit 2:
Bisected through a few commits and can confirm that 8e8e153 is the cause for this.

dims · 2020-11-06T13:50:01Z

@JornShen please see #89687 (review)

please try WithAPIVersionNegotiation

JornShen · 2020-11-06T16:12:02Z

@dims thanks for guide. I know one solution to deal with the problem inspiring from your advise.
I will take PR to try to fix it.

/assign

JornShen · 2020-11-10T01:23:19Z

The fix PR google/cadvisor#2714 has been merged.

So we wait for cadvisor' owner to update k/k dependency of cadvisor to v0.38 in 1.20 k8s? And cherry pick the dependency to v1.19(k8s) ? @dims

I see there have been a issuse #96287 to trace this thing.

dims · 2020-11-10T01:46:44Z

@JornShen yes someone will pick up #96287 soon. @bobbypage are we ready to cut a release of cadvisor?

bobbypage · 2020-11-10T01:47:41Z

ack, I will cut new release of cadvisor shortly.

JornShen · 2020-11-10T02:03:11Z

@dims @bobbypage all right, thanks!

JornShen · 2020-11-27T01:47:41Z

@pmyjavec @saul-gush @ialidzhikov @serathius @ghouscht

update to the newest v1.19-release, problem had been solve by #96849, or update the v1.20 alph version after 2020/11/14 by #96425 merged if you use docker verison under v19.X

k8s-ci-robot · 2020-11-30T02:42:11Z

@JornShen: Closing this issue.

In response to this:

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

pmyjavec added the kind/bug Categorizes issue or PR as related to a bug. label Aug 27, 2020

k8s-ci-robot added the needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. label Aug 27, 2020

k8s-ci-robot added sig/node Categorizes an issue or PR as relevant to SIG Node. sig/instrumentation Categorizes an issue or PR as relevant to SIG Instrumentation. and removed needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. labels Aug 27, 2020

k8s-ci-robot changed the title ~~No longer receiveing pod stats from stats/summary since 1.19.0~~ kubelet: pod metrics not visible with Docker 18 (19.03 works) Aug 27, 2020

rfranzke mentioned this issue Sep 1, 2020

Support for Kubernetes v1.19 gardener/gardener#2799

Merged

k8s-ci-robot added the priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. label Sep 1, 2020

xmudrii mentioned this issue Sep 1, 2020

Update Docker to 19.03.12 for Ubuntu, CentOS, RHEL kubermatic/machine-controller#815

Merged

serathius mentioned this issue Sep 1, 2020

Missing pod metrics on Docker 1.18 (kubeadm K8s version 1.19) kubernetes-sigs/metrics-server#581

Closed

xrstf mentioned this issue Sep 1, 2020

Add Kubernetes 1.19, Remove 1.15 kubermatic/kubermatic#5794

Merged

xmudrii mentioned this issue Sep 2, 2020

[release/v1.0] Update machine-controller to v1.17.1 kubermatic/kubeone#1087

Merged

jchengsfx mentioned this issue Sep 14, 2020

Add K8s v1.19 test job to circleci signalfx/signalfx-agent#1444

Merged

serathius mentioned this issue Sep 23, 2020

Kubernetes 1.19.x pod metrics empty kubernetes-sigs/metrics-server#586

Closed

bitva77 mentioned this issue Oct 23, 2020

Dashboards Showing No Data prometheus-operator/kube-prometheus#734

Closed

This was referenced Nov 4, 2020

Digital Ocean implementation of K8's does not allow metrics-server to function digitalocean/digitalocean-cloud-controller-manager#150

Closed

Support default/secure metrics-server installation digitalocean/DOKS#2

Open

k8s-ci-robot assigned JornShen Nov 6, 2020

JornShen mentioned this issue Nov 8, 2020

update docker client method google/cadvisor#2714

Merged

serathius mentioned this issue Nov 8, 2020

Cannot fetch Pods Metrics on Kubernetes 1.19 kubernetes-sigs/metrics-server#629

Closed

JornShen mentioned this issue Nov 12, 2020

Vendor cadvisor v0.38.4 #96425

Merged

This was referenced Nov 19, 2020

Updating to cadvisor v0.37.1 #96695

Merged

bump cadvisor to v0.37.2 #96849

Merged

k8s-ci-robot closed this as completed Nov 30, 2020

linchong mentioned this issue Dec 28, 2020

metrics-server doesn't return pod metrics rancher/rancher#30640

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

kubelet: pod metrics not visible with Docker 18 (19.03 works) #94281

kubelet: pod metrics not visible with Docker 18 (19.03 works) #94281

pmyjavec commented Aug 27, 2020 •

edited

Loading

pmyjavec commented Aug 27, 2020

pmyjavec commented Aug 27, 2020 •

edited

Loading

neolit123 commented Aug 27, 2020

pmyjavec commented Aug 27, 2020

neolit123 commented Aug 27, 2020

neolit123 commented Aug 27, 2020

serathius commented Aug 27, 2020

ialidzhikov commented Sep 1, 2020

dashpole commented Sep 11, 2020

saul-data commented Nov 2, 2020

ghouscht commented Nov 3, 2020 •

edited

Loading

dims commented Nov 6, 2020 •

edited

Loading

JornShen commented Nov 6, 2020

JornShen commented Nov 10, 2020

dims commented Nov 10, 2020

bobbypage commented Nov 10, 2020

JornShen commented Nov 10, 2020

JornShen commented Nov 27, 2020

k8s-ci-robot commented Nov 30, 2020

kubelet: pod metrics not visible with Docker 18 (19.03 works) #94281

kubelet: pod metrics not visible with Docker 18 (19.03 works) #94281

Comments

pmyjavec commented Aug 27, 2020 • edited Loading

pmyjavec commented Aug 27, 2020

pmyjavec commented Aug 27, 2020 • edited Loading

neolit123 commented Aug 27, 2020

pmyjavec commented Aug 27, 2020

neolit123 commented Aug 27, 2020

neolit123 commented Aug 27, 2020

serathius commented Aug 27, 2020

ialidzhikov commented Sep 1, 2020

dashpole commented Sep 11, 2020

saul-data commented Nov 2, 2020

ghouscht commented Nov 3, 2020 • edited Loading

dims commented Nov 6, 2020 • edited Loading

JornShen commented Nov 6, 2020

JornShen commented Nov 10, 2020

dims commented Nov 10, 2020

bobbypage commented Nov 10, 2020

JornShen commented Nov 10, 2020

JornShen commented Nov 27, 2020

k8s-ci-robot commented Nov 30, 2020

pmyjavec commented Aug 27, 2020 •

edited

Loading

pmyjavec commented Aug 27, 2020 •

edited

Loading

ghouscht commented Nov 3, 2020 •

edited

Loading

dims commented Nov 6, 2020 •

edited

Loading