You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Cloud provider or hardware configuration:
Bare metal (VMs)
OS (printf "$(uname -srm)\n$(cat /etc/os-release)\n"):
Linux 3.10.0-1160.42.2.el7.x86_64 x86_64
NAME="CentOS Linux"
VERSION="7 (Core)"
Version of Ansible (ansible --version):
ansible 2.10.11
Version of Python (python --version):
Python 3.6.8
Kubespray version (commit) (git rev-parse --short HEAD): b83e8b0 v2.17.0
Network plugin used:
Weave
In #7181 you upgraded to Weave 2.8.0 and both shuffled around and removed a number of volume mounts from the Pod in the DaemonSet.
Subsequently you upgraded Weave to 2.8.1 for your v2.16.0 release.
However I think weaveworks/weave#3886 might have been missed which added back in one of the volume mounts used to get a uuid for the node for 2.8.1. This tripped me up yesterday where after rebooting several nodes on one Weave failed to start throwing the following somewhat cryptic error:
FATA: 2021/09/22 07:11:59.141230 All system IDs are blank
My eventual workaround to get things back up and running was to hand edit the running DaemonSet to add the Volume mount back in for mounting the file /etc/machine-id into the main Weave container.
What really confused me about this is that I'm unsure why any of the other nodes started Weave properly - this looks on the surface like a problem that should have prevented all of them from starting.
Environment:
Cloud provider or hardware configuration:
Bare metal (VMs)
OS (
printf "$(uname -srm)\n$(cat /etc/os-release)\n"
):Linux 3.10.0-1160.42.2.el7.x86_64 x86_64
NAME="CentOS Linux"
VERSION="7 (Core)"
Version of Ansible (
ansible --version
):ansible 2.10.11
Version of Python (
python --version
):Python 3.6.8
Kubespray version (commit) (
git rev-parse --short HEAD
):b83e8b0 v2.17.0
Network plugin used:
Weave
In #7181 you upgraded to Weave 2.8.0 and both shuffled around and removed a number of volume mounts from the Pod in the DaemonSet.
Subsequently you upgraded Weave to 2.8.1 for your v2.16.0 release.
However I think weaveworks/weave#3886 might have been missed which added back in one of the volume mounts used to get a uuid for the node for 2.8.1. This tripped me up yesterday where after rebooting several nodes on one Weave failed to start throwing the following somewhat cryptic error:
FATA: 2021/09/22 07:11:59.141230 All system IDs are blank
My eventual workaround to get things back up and running was to hand edit the running DaemonSet to add the Volume mount back in for mounting the file
/etc/machine-id
into the main Weave container.What really confused me about this is that I'm unsure why any of the other nodes started Weave properly - this looks on the surface like a problem that should have prevented all of them from starting.
If someone else can reproduce this to make sure it's not just my environment that caused this I'm happy to take the original Weave DaemonSet from https://github.com/weaveworks/weave/blob/master/prog/weave-kube/weave-daemonset-k8s-1.11.yaml and update the Kubespray https://github.com/kubernetes-sigs/kubespray/blob/master/roles/network_plugin/weave/templates/weave-net.yml.j2 template to match.
The text was updated successfully, but these errors were encountered: