Update metrics-server from 0.2.1 to 0.3.3 don't work properly.

[pierluigilenoci]

Describe the bug
I tried to update the metrics-server (installed with addons manager) from v0.2.1 to v0.3.3. After the update the metrics-server stop to work properly.

I changed in the yaml file of metrics server only the version of the docker image, removed the option --source=kubernetes.summary_api:'' and added these two other options --kubelet-insecure-tls --kubelet-preferred-address-types=InternalIP

These are the logs:

I0731 14:37:31.683039 1 serving.go:312] Generated self-signed cert (apiserver.local.config/certificates/apiserver.crt, apiserver.local.config/certificates/apiserver.key) I0731 14:37:32.938791 1 secure_serving.go:116] Serving securely on [::]:443 E0731 14:37:38.441048 1 reststorage.go:147] unable to fetch pod metrics for pod kube-system/nginx-ingress-release-controller-cc7968ffb-jjp2c: no metrics known for pod
[... a lot of logs in this way and sometimes this]
E0731 14:38:32.957151 1 manager.go:111] unable to fully collect metrics: [unable to fully scrape metrics from source kubelet_summary:k8s-minion-75668405-vmss000076: unable to fetch metrics from Kubelet k8s-minion-75668405-vmss000076 (10.245.0.114): request failed - "401 Unauthorized", response: "Unauthorized", unable to fully scrape metrics from source kubelet_summary:k8s-master-75668405-0: unable to fetch metrics from Kubelet k8s-master-75668405-0 (10.245.255.200): request failed - "401 Unauthorized", response: "Unauthorized", unable to fully scrape metrics from source kubelet_summary:k8s-minion-75668405-vmss000078: unable to fetch metrics from Kubelet k8s-minion-75668405-vmss000078 (10.245.1.80): request failed - "401 Unauthorized", response: "Unauthorized"]

Steps To Reproduce
Install metrics-server v0.2.1 on a cluster, edit the addons-manager config file for metrics-server changing docker version and running options as mentioned above and wait a couple of minutes that addons-manager redeploy the package.

Expected behavior
Nothing special, I would like metrics-server continue to work

AKS Engine version
v0.33.2

Kubernetes version
1.13.5

Additional context
Other ref:
kubernetes-sigs/metrics-server#293
#1172

[welcome]

👋 Thanks for opening your first issue here! If you're reporting a 🐞 bug, please make sure you include steps to reproduce it.

[pierluigilenoci]

Ref #1109

[zhiweiv]

I upgraded to 0.3.x successfully before, I remember there are two more steps:
1 Add - nodes/stats to ClusterRole system:metrics-server, you can find it in https://github.com/kubernetes-incubator/metrics-server/blob/master/deploy/1.8%2B/resource-reader.yaml.
2 Pass --anonymous-auth=true to kubelet, seems latest AKS-Engine has this enabled by default, but I haven't verify it yet.

[pierluigilenoci]

I upgraded to 0.3.x successfully before, I remember there are two more steps:

Which tool did you used to deploy metrics-server?

1 Add - nodes/stats to ClusterRole system:metrics-server, you can find it in https://github.com/kubernetes-incubator/metrics-server/blob/master/deploy/1.8%2B/resource-reader.yaml.

Fixed everywhere
#1714
kubernetes-sigs/metrics-server#296
helm/charts#16060

2 Pass --anonymous-auth=true to kubelet, seems latest AKS-Engine has this enabled by default, but I haven't verify it yet.

I don’t like the second point of your suggestion for security reason (ref: https://gist.github.com/erictune/9dc7ae4b22505b9a8c20ad9cd03a45cc )

[zhiweiv]

I used same way as yours, modify the yaml on master server directly.

For #2 I think it is --authentication-token-webhook=true and --authorization-mode=Webhook

See kubernetes-sigs/metrics-server#212

[andyzhangx]

I used same way as yours, modify the yaml on master server directly.

For #2 I think it is --authentication-token-webhook=true and --authorization-mode=Webhook

See kubernetes-incubator/metrics-server#212

That's correct, I think you could refer to this PR: #1109, on aks-engine cluster, it works