Kubernetes 1.19.x pod metrics empty

[vaassi]

What happened:
after I upgrade kubernetes from 1.18 to 1.19.2 version
kubectl top node works, but kubectl top pods not work anymore

kubectl top pods -n kube-system

W0920 22:24:22.477153 1935618 top_pod.go:265] Metrics not available for pod kube-system/coredns-6f5c7bbdfb-pz95z, age: 1h50m46.47714468s
error: Metrics not available for pod kube-system/coredns-6f5c7bbdfb-pz95z, age: 1h50m46.47714468s

kubectl get --raw "/apis/metrics.k8s.io/v1beta1/pods/" | jq

{
  "kind": "PodMetricsList",
  "apiVersion": "metrics.k8s.io/v1beta1",
  "metadata": {
    "selfLink": "/apis/metrics.k8s.io/v1beta1/pods/"
  },
  "items": []
}

kubectl -n kube-system logs metrics-server-56df6d849-c92fx

E0920 19:27:30.715373       1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/kube-proxy-7mbbt: no metrics known for pod
E0920 19:27:30.715383       1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/weave-net-7fv7w: no metrics known for pod
E0920 19:27:30.715396       1 reststorage.go:160] unable to fetch pod metrics for pod logging/metricbeat-metricbeat-jm6h6: no metrics known for pod
...

What you expected to happen:
metrics for pods must not be empty

Anything else we need to know?:
I use metrics-server 0.3.7

kubectl -n kube-system get deploy metrics-server -o yaml

    spec:
      containers:
      - args:
        - --cert-dir=/tmp
        - --secure-port=4443
        - --kubelet-insecure-tls
        image: k8s.gcr.io/metrics-server/metrics-server:v0.3.7
        imagePullPolicy: IfNotPresent
        name: metrics-server
        ports:
        - containerPort: 4443
          name: main-port
          protocol: TCP

Environment:

• Kubernetes distribution (GKE, EKS, Kubeadm, the hard way, etc.): kubeadm 1.19.2
• Container Network Setup (flannel, calico, etc.): weave-net
• Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.2", GitCommit:"f5743093fd1c663cb0cbc89748f730662345d44d", GitTreeState:"clean", BuildDate:"2020-09-16T13:41:02Z", GoVersion:"go1.15", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.2", GitCommit:"f5743093fd1c663cb0cbc89748f730662345d44d", GitTreeState:"clean", BuildDate:"2020-09-16T13:32:58Z", GoVersion:"go1.15", Compiler:"gc", Platform:"linux/amd64"}

/king bug

[serathius]

Guess that's the same as kubernetes/kubernetes#94281

[serathius]

Please check if your running Docker 1.18 and confirm that this issue is a duplicate checking if kubelet "/stats/summary" returns metrics

[vaassi]

I check my docker version and it's already 19, but problem still exist

docker --version
Docker version 19.03.12, build 48a66213fe

[vaassi]

Finally, I found solution for this problem:

I found strange behaviour, that docker shows me version 19, but version 18 is still installed:

# docker --version
Docker version 19.03.12, build 48a66213fe

# dnf list docker-ce
Last metadata expiration check: 2:39:27 ago on Thu 29 Oct 2020 09:13:20 AM MSK.
Installed Packages
docker-ce.x86_64 3:18.09.1-3.el7 @docker-ce-stable
Available Packages
docker-ce.x86_64 3:19.03.13-3.el7 docker-ce-stable

So I readd docker-ce repository with command:
# dnf config-manager --add-repo=https://download.docker.com/linux/centos/docker-ce.repo
then dnf update, and finally docker version 19 is installed properly and metrics start to work.

This issue can be closed, thank you.

[BiTree]

You could try to restart each kubelet on your node, I met the same problem, and solved it by just restarted the all kubelets.