Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Weave 2.8.1 fails to start due to missing volume mount #8003

Closed
frankfil opened this issue Sep 22, 2021 · 1 comment · Fixed by #8013
Closed

Weave 2.8.1 fails to start due to missing volume mount #8003

frankfil opened this issue Sep 22, 2021 · 1 comment · Fixed by #8013
Labels
kind/bug Categorizes issue or PR as related to a bug.

Comments

@frankfil
Copy link
Contributor

Environment:

  • Cloud provider or hardware configuration:
    Bare metal (VMs)

  • OS (printf "$(uname -srm)\n$(cat /etc/os-release)\n"):
    Linux 3.10.0-1160.42.2.el7.x86_64 x86_64
    NAME="CentOS Linux"
    VERSION="7 (Core)"

  • Version of Ansible (ansible --version):
    ansible 2.10.11

  • Version of Python (python --version):
    Python 3.6.8

Kubespray version (commit) (git rev-parse --short HEAD):
b83e8b0 v2.17.0

Network plugin used:
Weave

In #7181 you upgraded to Weave 2.8.0 and both shuffled around and removed a number of volume mounts from the Pod in the DaemonSet.

Subsequently you upgraded Weave to 2.8.1 for your v2.16.0 release.

However I think weaveworks/weave#3886 might have been missed which added back in one of the volume mounts used to get a uuid for the node for 2.8.1. This tripped me up yesterday where after rebooting several nodes on one Weave failed to start throwing the following somewhat cryptic error:

FATA: 2021/09/22 07:11:59.141230 All system IDs are blank

My eventual workaround to get things back up and running was to hand edit the running DaemonSet to add the Volume mount back in for mounting the file /etc/machine-id into the main Weave container.

What really confused me about this is that I'm unsure why any of the other nodes started Weave properly - this looks on the surface like a problem that should have prevented all of them from starting.

If someone else can reproduce this to make sure it's not just my environment that caused this I'm happy to take the original Weave DaemonSet from https://github.com/weaveworks/weave/blob/master/prog/weave-kube/weave-daemonset-k8s-1.11.yaml and update the Kubespray https://github.com/kubernetes-sigs/kubespray/blob/master/roles/network_plugin/weave/templates/weave-net.yml.j2 template to match.

@frankfil frankfil added the kind/bug Categorizes issue or PR as related to a bug. label Sep 22, 2021
@floryut
Copy link
Member

floryut commented Sep 23, 2021

I'll try to take a look, that might be on me; maybe open up a PR if there is something missing

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/bug Categorizes issue or PR as related to a bug.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants