master kubelet stuck in system:unauthorized

[cgwalters]

I've seen this happen a few times:

[root@osiris-master-0 ~]# systemctl status kubelet -l
● kubelet.service - Kubernetes Kubelet
   Loaded: loaded (/etc/systemd/system/kubelet.service; enabled; vendor preset: enabled)
   Active: active (running) since Tue 2018-11-20 21:52:04 UTC; 30s ago
  Process: 4173 ExecStartPre=/bin/mkdir --parents /etc/kubernetes/manifests (code=exited, status=0/SUCCESS)
 Main PID: 4175 (hyperkube)
   Memory: 41.2M
   CGroup: /system.slice/kubelet.service
           └─4175 /usr/bin/hyperkube kubelet --config=/etc/kubernetes/kubelet.conf --bootstrap-kubeconfig=/etc/kubernetes/kubeconfig --rotate-certificates --kubeconfig=/var/lib/kubelet/kubeconfig --container-runtime=remote --container-runtime-endpoint=/var/run/crio/crio.sock --allow-privileged --node-labels=node-role.kubernetes.io/master --minimum-container-ttl-duration=6m0s --client-ca-file=/etc/kubernetes/ca.crt --cloud-provider= --anonymous-auth=false --register-with-taints=node-role.kubernetes.io/master=:NoSchedule

Nov 20 21:52:32 osiris-master-0 hyperkube[4175]: E1120 21:52:32.774998    4175 reflector.go:136] k8s.io/kubernetes/pkg/kubelet/kubelet.go:464: Failed to list *v1.Node: nodes "osiris-master-0" is forbidden: User "system:anonymous" cannot list nodes at the cluster scope: no RBAC policy matched
Nov 20 21:52:32 osiris-master-0 hyperkube[4175]: E1120 21:52:32.782850    4175 reflector.go:136] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: pods is forbidden: User "system:anonymous" cannot list pods at the cluster scope: no RBAC policy matched
Nov 20 21:52:33 osiris-master-0 hyperkube[4175]: E1120 21:52:33.762588    4175 reflector.go:136] k8s.io/kubernetes/pkg/kubelet/kubelet.go:455: Failed to list *v1.Service: services is forbidden: User "system:anonymous" cannot list services at the cluster scope: no RBAC policy matched
Nov 20 21:52:33 osiris-master-0 hyperkube[4175]: E1120 21:52:33.779189    4175 reflector.go:136] k8s.io/kubernetes/pkg/kubelet/kubelet.go:464: Failed to list *v1.Node: nodes "osiris-master-0" is forbidden: User "system:anonymous" cannot list nodes at the cluster scope: no RBAC policy matched
Nov 20 21:52:33 osiris-master-0 hyperkube[4175]: E1120 21:52:33.785375    4175 reflector.go:136] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: pods is forbidden: User "system:anonymous" cannot list pods at the cluster scope: no RBAC policy matched
Nov 20 21:52:34 osiris-master-0 hyperkube[4175]: E1120 21:52:34.767188    4175 reflector.go:136] k8s.io/kubernetes/pkg/kubelet/kubelet.go:455: Failed to list *v1.Service: services is forbidden: User "system:anonymous" cannot list services at the cluster scope: no RBAC policy matched
Nov 20 21:52:34 osiris-master-0 hyperkube[4175]: E1120 21:52:34.782492    4175 reflector.go:136] k8s.io/kubernetes/pkg/kubelet/kubelet.go:464: Failed to list *v1.Node: nodes "osiris-master-0" is forbidden: User "system:anonymous" cannot list nodes at the cluster scope: no RBAC policy matched
Nov 20 21:52:34 osiris-master-0 hyperkube[4175]: E1120 21:52:34.789195    4175 reflector.go:136] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: pods is forbidden: User "system:anonymous" cannot list pods at the cluster scope: no RBAC policy matched
Nov 20 21:52:34 osiris-master-0 hyperkube[4175]: E1120 21:52:34.938127    4175 eviction_manager.go:243] eviction manager: failed to get get summary stats: failed to get node info: node "osiris-master-0" not found
Nov 20 21:52:34 osiris-master-0 hyperkube[4175]: E1120 21:52:34.938592    4175 kubelet.go:2101] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni config uninitialized

[sjenning]

Seems like the bootstrap-kubeconfig is using a client cert that is not signed by the cluster CA

[wking]

If you're cleaning your state between runs, maybe the kubelet cert expired (#650)? Was the node offline for long enough for its cert to expire?

Also, can we get some of the details requested by the issue template and troubleshooting docs? It seems unlikely that this is particularly dependent on kubelet versions, but it is probably tied to specific installer versions. And "happens a few times" is going to be hard enough to reproduce without spending time working on potentially different versions of things we can pin down ;).

[cgwalters]

This may have been another variant of #785 - closing for now.