Support dual stack for K3S image

[Mahmoud-Emad]

The current image version does not enable the 'DualStack' but 'SingleStack', which provides access to the master/workers only over public IPv4. This was requested in threefoldtech/tfgrid-sdk-ts#3149 to support managing gateways on Kubernetes and can include IPv6 normally.

after some investigations, I found that we should pass the flag while running the k3s server as well as mentioned in k3s-io/k3s#4400

[PeterNashaat]

• Created new flist from k3s latest version v1.31.0+k3s1 https://hub.grid.tf/petep.3bot/k3s-ds.flist.md and testing now

[PeterNashaat]

• Trying to test the k3s dualstack flist by deploying a cluster with one worker and one master which works, but it appears it's connected with ipv4

root@m1:~# kubectl get nodes -o wide
NAME   STATUS     ROLES                  AGE     VERSION        INTERNAL-IP      EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION   CONTAINER-RUNTIME
m1     Ready      control-plane,master   3h15m   v1.31.0+k3s1   10.20.4.2        <none>        Ubuntu 24.04.1 LTS   6.1.21           containerd://1.7.20-k3s1
w1     Ready      <none>                 3h15m   v1.31.0+k3s1   10.20.5.2        <none>        Ubuntu 24.04.1 LTS   6.1.21           containerd://1.7.20-k3s1
w2     NotReady   <none>                 84m     v1.31.0+k3s1   185.69.166.162   <none>        Ubuntu 24.04.1 LTS   6.1.21           containerd://1.7.20-k3s1
w4     Ready      <none>                 16m     v1.31.0+k3s1   185.69.166.151   <none>        Ubuntu 24.04.1 LTS   6.1.21           containerd://1.7.20-k3s1

• Tried to make sure the dual stack is working by creating a micro vm w4 with the same flist but connect to the cluster with master's public ipv6

export K3S_URL=https://[2a02:1802:5e:0:b80a:a7ff:fe29:f0d0]:6443
export K3S_FLANNEL_IFACE=eth1
export K3S_DATA_DIR=/mnt/data/
export K3S_TOKEN=WDMTQ2ecuf
export EXTRA_ARGS="--data-dir $K3S_DATA_DIR --kubelet-arg=root-dir=$K3S_DATA_DIR/kubelet"
k3s agent --flannel-iface $K3S_FLANNEL_IFACE $EXTRA_ARGS

from logs worker w4 run in dual stack mode

[+] k3s: time="2024-09-29T13:07:04Z" level=info msg="Annotations and labels have been set successfully on node: w4"
[+] k3s: time="2024-09-29T13:07:04Z" level=info msg="Starting flannel with backend vxlan"
[+] k3s: I0929 13:07:04.369697     189 server.go:677] "Successfully retrieved node IP(s)" IPs=["185.69.166.151","2a02:1802:5e:0:d84b:36ff:fe34:7903"]
[+] k3s: E0929 13:07:04.369931     189 server.go:234] "Kube-proxy configuration may be incomplete or incorrect" err="nodePortAddresses is unset; NodePort connections will be accepted on all local IPs. Consider using `--nodeport-addresses primary`"
[+] k3s: I0929 13:07:04.381571     189 server.go:243] "kube-proxy running in dual-stack mode" primary ipFamily="IPv4"
[+] k3s: time="2024-09-29T13:07:04Z" level=info msg="Using dual-stack mode. The interface eth1 with ipv4 address 185.69.166.151 and ipv6 address 2a02:1802:5e:0:d84b:36ff:fe34:7903 will be used by flannel"

then it tries to connect to the master over the private ipv4

[+] k3s: E0929 13:07:04.600729     189 cleanup.go:70] "Failed to delete stale service connections" err="error deleting connection tracking state for UDP service IP: 10.43.0.10, error: conntrack command returned: \"conntrack v1.4.7 (conntrack-tools): Operation failed: invalid parameters\\n\", error message: exit status 1" IP="10.43.0.10"
[+] k3s: E0929 13:07:04.682574     189 cleanup.go:70] "Failed to delete stale service connections" err="error deleting connection tracking state for UDP service IP: 2001:cafe:43::a, error: conntrack command returned: \"conntrack v1.4.7 (conntrack-tools): Operation failed: invalid parameters\\n\", error message: exit status 1" IP="2001:cafe:43::a"
[+] k3s: time="2024-09-29T13:07:04Z" level=info msg="Connecting to proxy" url="wss://10.20.4.2:6443/v1-k3s/connect"
[+] k3s: time="2024-09-29T13:07:04Z" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp 10.20.4.2:6443: connect: no route to host"
[+] k3s: time="2024-09-29T13:07:04Z" level=error msg="Remotedialer proxy error; reconnecting..." error="dial tcp 10.20.4.2:6443: connect: no route to host" url="wss://10.20.4.2:6443/v1-k3s/connect"
.
.
.
[+] k3s: I0929 13:07:05.590558     189 iptables.go:372] bootstrap done
[+] k3s: I0929 13:07:05.590643     189 iptables.go:372] bootstrap done
[+] k3s: I0929 13:07:05.592143     189 iptables.go:372] bootstrap done
[+] k3s: time="2024-09-29T13:07:05Z" level=info msg="Connecting to proxy" url="wss://10.20.4.2:6443/v1-k3s/connect"
[+] k3s: time="2024-09-29T13:07:05Z" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp 10.20.4.2:6443: connect: no route to host"
[+] k3s: time="2024-09-29T13:07:05Z" level=error msg="Remotedialer proxy error; reconnecting..." error="dial tcp 10.20.4.2:6443: connect: no route to host" url="wss://10.20.4.2:6443/v1-k3s/connect"
[+] k3s: time="2024-09-29T13:07:06Z" level=info msg="Connecting to proxy" url="wss://10.20.4.2:6443/v1-k3s/connect"
[+] k3s: time="2024-09-29T13:07:06Z" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp 10.20.4.2:6443: connect: no route to host"
[+] k3s: time="2024-09-29T13:07:06Z" level=error msg="Remotedialer proxy error; reconnecting..." error="dial tcp 10.20.4.2:6443: connect: no route to host" url="wss://10.20.4.2:6443/v1-k3s/connect"
[+] k3s: time="2024-09-29T13:07:07Z" level=info msg="Connecting to proxy" url="wss://10.20.4.2:6443/v1-k3s/connect"
[+] k3s: time="2024-09-29T13:07:07Z" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp 10.20.4.2:6443: connect: no route to host"
[+] k3s: time="2024-09-29T13:07:07Z" level=error msg="Remotedialer proxy error; reconnecting..." error="dial tcp 10.20.4.2:6443: connect: no route to host" url="wss://10.20.4.2:6443/v1-k3s/connect"

[PeterNashaat]

• Will try another deployment to test and verify the correct config and architecture of the setup should be.

[PeterNashaat]

• Installed new cluster using my flist https://hub.grid.tf/petep.3bot/k3s-ds.flist with only public ipv6 on the master with worker that has no pub ip's eg. mycelium, ipv4, v6 or planetary.
  • info about master and worker here

• Installed nginx pod to test the cluster.

root@m1:~# cat nginx-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
spec:
  replicas: 2
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:latest
        ports:
        - containerPort: 80

• Then ran kubectl apply -f nginx-deployment.yaml

root@m1:~# kubectl get pods -o wide
NAME                                READY   STATUS    RESTARTS   AGE   IP          NODE   NOMINATED NODE   READINESS GATES
nginx-deployment-54b9c68f67-vt6xt   1/1     Running   0          34m   10.42.0.7   m1     <none>           <none>
nginx-deployment-54b9c68f67-xr5sh   1/1     Running   0          34m   10.42.1.7   w1     <none>           <none>

• Exposeing the Nginx pod

kubectl expose deployment nginx-deployment --port=80 --target-port=80 --name=nginx-service

• Adding portforward

kubectl port-forward --address :: svc/nginx-service 8080:80

• Tested from an ipv6 vm

root@imagecreator:~# curl -g -6 "http://[2a04:f340:c0:71:a41f:85ff:fe3c:a546]:8080"
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>

• Tested from a browser
  

• Modified the deployment template to deploy only in the worker node w1 and tested connection again and worked fine.

[PeterNashaat]

• The first deployment with m1 and w1 worked because both nodes had IPv4 addresses and the K3s cluster is configured for IPv4-only pod networking (10.42.x.x/16). The internal pod communication was handled over IPv4, while the external traffic (e.g., accessing the Nginx service) used IPv6 through the master node's public IPv6 address via port forwarding.

• Adding new worker w2 which only has ipv6 no private ipv4 on eth1 failed with error

ansitionTime":"2024-09-30T14:04:18Z","reason":"KubeletNotReady","message":"CSINode is not yet initialized"}
INFO[0005] Flannel found PodCIDR assigned for node w2
ERRO[0005] flannel exited: failed to find the interface: failed to find IPv4 address for interface eth1

[PeterNashaat]

• flist created https://hub.grid.tf/petep.3bot/threefolddev-k3s_v1.31.0_ds-latest.flist , the issue in testing the flist is that when using the dashboard the worker and master nodes are not in the same ipv6 private network.