Two VMs on 2 separate hosts, how to make networking work with Ignite and WeaveNet?

[mdundek]

Hello weaveworks team,

I am trying to spin up 2 independant k8s clusters, each having one respective worker node using ignite. I have 2 machines for this test:

Host-1 (IP: 192.168.68.130)
    |_ K8S Master 1 
    |_ K8S Master 2 

Host-2 (IP: 192.168.68.134)
    |_ K8S Worker 1 (attache to master-1)
    |_ K8S Worker 2 (attache to master-2)

Therefore, I want to set up a WeaveNet CNI network on both hosts using the instructions provided from your documentation:

# On Host-1 & Host-2

docker run -d \
  --privileged \
  --net host \
  --pid host \
  --restart always \
  -e HOSTNAME="$(hostname)" \
  -e KUBE_PEERS="192.168.68.130 192.168.68.134" \
  -v /var/lib/weave:/weavedb \
  -v /opt:/host/opt \
  -v /home:/host/home \
  -v /etc:/host/etc \
  -v /var/lib/dbus:/host/var/lib/dbus \
  -v /lib/modules:/lib/modules \
  -v /run/xtables.lock:/run/xtables.lock \
  --entrypoint /home/weave/launch.sh \
  weaveworks/weave-kube:2.5.2

I removed the file rm -rf /etc/cni/net.d/10-ignite.conflist, then started my Master-1 on Host-1, and Worker-1 on Host-2 using the following command:

# On Host-1

sudo ignite \
        --network-plugin cni \
        run weaveworks/ignite-kubeadm:latest \
        --cpus 2 \
        --memory 2GB \
        --ssh \
        --name master-1

and

# On Host-2

sudo ignite \
        --network-plugin cni \
        run weaveworks/ignite-kubeadm:latest \
        --cpus 2 \
        --memory 2GB \
        --ssh \
        --name worker-1

PS: I am not even attempting to start the K8S cluster here, I first want to get the networking part right on one of the two clusters.

Then I ssh into each VM using sudo ignite ssh master-1 and sudo ignite ssh worker-1 on each respective host.
I was expecting on Host-1, VM Master-1 to have a network interface called eth0 with an IP address that is part of a subnet shared with the VM on Host-2, VM Worker-2.
When I did a sudo ignite ps on both hosts, I got the following results:

VM ID			IMAGE					KERNEL					SIZE	CPUS	MEMORY	CREATED	STATUS	IPS		PORTS	NAME
7962498dc0401f33	weaveworks/ignite-kubeadm:latest	weaveworks/ignite-kernel:4.19.125	4.0 GB	2	2.0 GB	22m ago	Up 22m	10.32.0.2		master-1

and

VM ID			IMAGE					KERNEL					SIZE	CPUS	MEMORY	CREATED	STATUS	IPS		PORTS	NAME
a9f37cb3378c6c00	weaveworks/ignite-kubeadm:latest	weaveworks/ignite-kernel:4.19.125	4.0 GB	2	2.0 GB	3m ago	Up 3m	10.40.0.1		worker-1

When I ssh into do VM Master-1 on Host-1 and do a ifconfig, I get the following:

eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 10.32.0.2  netmask 255.240.0.0  broadcast 10.47.255.255
        inet6 fe80::4cdc:6fff:feb8:90a6  prefixlen 64  scopeid 0x20<link>
        ether 4e:dc:6f:b8:90:a6  txqueuelen 1000  (Ethernet)
        RX packets 323  bytes 28590 (28.5 KB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 755  bytes 64154 (64.1 KB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

I have a hard time doing ssh into VM Worker-1 on Host-2, it times out the first time I try, and succedes at the second attempt. Once that ssh session started, I can not do anything, the cursor is frozen, and reacts only after 30 seconds or so. I did manage to get the output of ifconfig:

eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 10.40.0.1  netmask 255.240.0.0  broadcast 10.47.255.255
        inet6 fe80::2409:8aff:fef9:500d  prefixlen 64  scopeid 0x20<link>
        ether 26:09:8a:f9:50:0d  txqueuelen 1000  (Ethernet)
        RX packets 1290  bytes 69496 (69.4 KB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 1126  bytes 59796 (59.7 KB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

When I try to ping Master-1 from Worker-1, I get a Host Unreachable.

My guess is that I am doing something wrong here, but what is it?
Could someone with more experience with WeaveNet and Ignite point me in the right direction please, that would be fantastic!

Thanks

[mdundek]

Is it possible to get some clarification on this issue please? Ignite is a fantastic tool if this works as expected (well done to the weaveworks team by the way), but I am not sure if it has been designed for those use cases to begin with.
Is this a bug, a limitation or simply a miss configuration, in which case could you point me towards some documentation or guidance to enable this scenario please?

Thanks guys

[luxas]

Hi 👋!

I will try to debug this some time during this week. Sorry, I've been heads-down on some other projects and work efforts.
It's been a while since I looked at this specifically so there might be some docs that are out of date.
Thanks for you patience.

[mdundek]

Thanks Luxas! I appreciate it very much. Looking forward to using this for some fun projects I have in mind.

[luxas]

See #634 for a possible (atm very hacky) fix. Also #633 for an explanation.