IPv4/IPv6 dual-stack

[eremid]

Hello,

I'm trying to set up an IPv4/IPv6 dual-stack on Talos v1.6.3. I haven't found documentation to help me, and there are some open/closed issues, so I don't know what the status of dual-stack in Talos is.
Is there someone with a working configuration who can help me? I'm running Talos on three nodes with Flannel and MetalLB.

Here are my changes in the machineconfig to add IPv6 support :

---
machine:
  type: controlplane
  sysctls:
    net.ipv6.conf.all.forwarding: 1
  network:
    interfaces:
      - deviceSelector:
          busPath: 0*
        dhcp: true
        dhcpOptions:
          ipv6: true
cluster:
  network:
    podSubnets:
      - 10.244.0.0/16
      - 2001:db8:42:0::/56
    serviceSubnets:
      - 10.96.0.0/12
      - 2001:db8:42:1::/112

In kube-proxy daemonset :
- --cluster-cidr=10.244.0.0/16,2001:db8:42:0::/56

For now, I'm stuck without an IPv6 CIDR on my nodes :

❯ k get node   -o yaml|grep -i cidr
    podCIDR: 10.244.1.0/24
    podCIDRs:
    podCIDR: 10.244.0.0/24
    podCIDRs:
    podCIDR: 10.244.2.0/24
    podCIDRs:

Nothing in the kube-controller-manager logs :

I0131 21:06:45.173256       1 serving.go:380] Generated self-signed cert in-memory
I0131 21:06:45.931520       1 controllermanager.go:187] "Starting" version="v1.29.1"
I0131 21:06:45.931539       1 controllermanager.go:189] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
I0131 21:06:45.934271       1 requestheader_controller.go:169] Starting RequestHeaderAuthRequestController
I0131 21:06:45.934288       1 shared_informer.go:311] Waiting for caches to sync for RequestHeaderAuthRequestController
I0131 21:06:45.934316       1 configmap_cafile_content.go:202] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::client-ca-file"
I0131 21:06:45.934330       1 shared_informer.go:311] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I0131 21:06:45.934354       1 configmap_cafile_content.go:202] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file"
I0131 21:06:45.934361       1 shared_informer.go:311] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
I0131 21:06:45.935853       1 secure_serving.go:213] Serving securely on 127.0.0.1:10257
I0131 21:06:45.936126       1 leaderelection.go:250] attempting to acquire leader lease kube-system/kube-controller-manager...
I0131 21:06:45.936427       1 tlsconfig.go:240] "Starting DynamicServingCertificateController"
I0131 21:06:46.034588       1 shared_informer.go:318] Caches are synced for RequestHeaderAuthRequestController
I0131 21:06:46.034611       1 shared_informer.go:318] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
I0131 21:06:46.035152       1 shared_informer.go:318] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file

IPs :

❯ talosctl -n 10.10.10.1 get addresses
NODE         NAMESPACE   TYPE            ID                                                 VERSION   ADDRESS                                     LINK
10.10.10.1   network     AddressStatus   cni0/10.244.2.1/24                                 1         10.244.2.1/24                               cni0
10.10.10.1   network     AddressStatus   cni0/fe80::24fa:4fff:aaaa:aaaa/64                  2         fe80::24fa:4fff:aaaa:aaaa/64                cni0
10.10.10.1   network     AddressStatus   enp1s0/10.10.10.1/24                               1         10.10.10.1/24                               enp1s0
10.10.10.1   network     AddressStatus   enp1s0/2a00:ab19:99b:aaaa:aaaa:aaaa:aaaa:aaaa/64   1         enp1s0/2a00:ab19:99b:aaaa:aaaa:aaaa:aaaa:aaaa/64   enp1s0
10.10.10.1   network     AddressStatus   enp1s0/fe80::24fa:4fff:aaaa:aaaa/64                2        fe80::24fa:4fff:aaaa:aaaa/64                enp1s0
10.10.10.1   network     AddressStatus   flannel.1/10.244.2.0/32                            1         10.244.2.0/32                               flannel.1
10.10.10.1   network     AddressStatus   flannel.1/fe80::44e3:aaaa:aaaa:aaaa/64             2         fe80::44e3:aaaa:aaaa:aaaa/64                flannel.1
10.10.10.1   network     AddressStatus   lo/127.0.0.1/8                                     1         127.0.0.1/8                                 lo
10.10.10.1   network     AddressStatus   lo/::1/128                                         1         ::1/128                                     lo
10.10.10.1   network     AddressStatus   veth94e37908/fe80::ca0:aaaa:aaaa:aaaa/64            2         fe80::ca0:aaaa:aaaa:aaaa/64                  veth94e37908

Thanks a lot for your help.

[smira]

Doing proper pod/service subnets should be enough. If you applied this after the moment the cluster bootstrapped, you might need to do a manifest sync via talosctl upgrade-k8s.

@TimJones any ideas?

[TimJones]

If you're using the default Flannel CNI that comes with Talos, you will have to specifically enable IPv6 with the EnableIPv6 config. See flannel docs here: https://github.com/flannel-io/flannel/blob/master/Documentation/configuration.md

[eremid]

Hello,
So yes, with a talosctl upgrade-k8s, some of my manual changes are done with Talos.
The flannel config is updated with the EnableIPv6 and the IPv6Network.
However, without an IPv6 CIDR on my node, Flannel throws an error :

Defaulted container "kube-flannel" out of: kube-flannel, install-config (init), install-cni (init)
I0201 15:19:52.424908       1 main.go:210] CLI flags config: {etcdEndpoints:http://127.0.0.1:4001,http://127.0.0.1:2379 etcdPrefix:/coreos.com/network etcdKeyfile: etcdCertfile: etcdCAFile: etcdUsername: etcdPassword: version:false kubeSubnetMgr:true kubeApiUrl: kubeAnnotationPrefix:flannel.alpha.coreos.com kubeConfigFile: iface:[] ifaceRegex:[] ipMasq:true ifaceCanReach: subnetFile:/run/flannel/subnet.env publicIP: publicIPv6: subnetLeaseRenewMargin:60 healthzIP:0.0.0.0 healthzPort:0 iptablesResyncSeconds:5 iptablesForwardRules:true netConfPath:/etc/kube-flannel/net-conf.json setNodeNetworkUnavailable:true}
W0201 15:19:52.424967       1 client_config.go:617] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I0201 15:19:52.432135       1 kube.go:490] Starting kube subnet manager
I0201 15:19:52.432138       1 kube.go:145] Waiting 10m0s for node controller to sync
I0201 15:19:52.436326       1 kube.go:511] Creating the node lease for IPv4. This is the n.Spec.PodCIDRs: [10.244.1.0/24]
I0201 15:19:52.436353       1 kube.go:266] Error turning node "talos-2pj-0xg" to lease: Invalid IP address format
I0201 15:19:52.436368       1 kube.go:511] Creating the node lease for IPv4. This is the n.Spec.PodCIDRs: [10.244.0.0/24]
I0201 15:19:52.436376       1 kube.go:266] Error turning node "talos-d4k-ofw" to lease: Invalid IP address format
I0201 15:19:52.436393       1 kube.go:511] Creating the node lease for IPv4. This is the n.Spec.PodCIDRs: [10.244.2.0/24]
I0201 15:19:52.436406       1 kube.go:544] Creating the node lease for IPv6. This is the n.Spec.PodCIDRs: [10.244.2.0/24]
E0201 15:19:52.436622       1 runtime.go:79] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)
goroutine 10 [running]:
k8s.io/apimachinery/pkg/util/runtime.logPanic({0x19389c0?, 0x2a225e0})

Currently, I think the problem is the missing IPv6 cidr in the node definition :

❯ k describe node talos-o20-aa1|grep -i cidr
PodCIDR:                      10.244.2.0/24
PodCIDRs:                     10.244.2.0/24

Thanks for your answers.

[smira]

I think you'd need to re-join the kubelet, I don't think it would reconcile. Some changes like that are not handled good way on Kubernetes side.

[eremid]

What do you mean by "re-join the kubelet" ?
I'm on a cluster with three control plane nodes : do I need to reset (with a talosctl reset) my nodes one by one ?

[smira]

well, you can do that as well, you can also start from scratch with correct config patches; pod/service subnets is not something which can be easily always changed on the fly.

if going the reset one by one route, use talosctl -n <IP> reset --system-labels-to-wipe=EPHEMERAL --reboot

[jalim]

@eremid did you manage to get this working without tearing down the whole cluster? I'm trying to achieve the same but reseting each node one at a time doesn't seem to have achieved anything.

[RealKelsar]

I am stuck at the same Point, the Nodes not getting the IPv6 Cidr. Funny enough configuring them to big stops the controller-manner to come up.
In my case it is not flannel but cilium, but it pretty much has a similar error.

Just to be safe, I redeployed the whole cluster multiple times now.

  podCIDR: 10.244.1.0/24
  podCIDRs:
  - 10.244.1.0/24

[bernardgut]

@eremid and @RealKelsar did any of you manage to make IPv6 work with Talos ? If so can you share the configuration needed? I am on Talos/Cilium and cannot get dualstack working

[RealKelsar]

No, I just tried again, dunno I am seem to miss something very fundamental...

[RealKelsar]

I got it kind of running now, PODs get IPv6, ingress gets IPv6...

But i am not sure v6 really works....

I can not reach it yet, but that very well may be my Opnsense beeing difficult.

[bernardgut]

How did you do it ?

[skandragon]

I am also having problems with this. It seems like ipv6 should be a no brainer in 2024...

I'm not entirely sure what is broken here, but I have IPv6 configured, but now flannel won't start. My guess is this is related to the node not having something configured properly. The Talos address list includes an IPv6 address.

What's also annoyingly difficult is that my provider is lame, and only gives me three /64s, not something I can chop up. My plan was to use fc00::/7 address subnets for the pod and service addresses, and then put "proper" addresses in MetalLB.

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x10 pc=0x52a937]

goroutine 1 [running]:
math/big.(*Int).Bytes(0x0)
	/usr/local/go/src/math/big/int.go:527 +0x17
github.com/flannel-io/flannel/pkg/ip.(*IP6).ToIP(0x0)
	/build/pkg/ip/ip6net.go:82 +0x1c
github.com/flannel-io/flannel/pkg/ip.IP6Net.ToIPNet({0x0?, 0x0?})
	/build/pkg/ip/ip6net.go:175 +0x25
github.com/flannel-io/flannel/pkg/ip.EnsureV6AddressOnLink({0x0?, 0xc0005015d8?}, {0xc000142540?, 0x30?}, {0x1e611e8, 0xc00050ba40})
	/build/pkg/ip/iface.go:298 +0x52
github.com/flannel-io/flannel/pkg/backend/vxlan.(*vxlanDevice).ConfigureIPv6(0xc00010f9f0, {0x0?, 0xc00048fbd0?}, {0xc000142540?, 0x10?})
	/build/pkg/backend/vxlan/device.go:153 +0x50
github.com/flannel-io/flannel/pkg/backend/vxlan.(*VXLANBackend).RegisterNetwork(0xc00051ed38, {0x1e756f8, 0xc00048fbd0}, 0x0?, 0xc000462100)
	/build/pkg/backend/vxlan/vxlan.go:228 +0xd25
main.main()
	/build/main.go:341 +0xfc8

[imp1sh]

Wow, should I even try talos? This would be a show stopper for me...

[mgruener]

What worked for me after extending the podSubnets with an ipv6 CIDR was deleting the node object (kubectl delete node XXX) and rebooting it. After the reboot, the node object was recreated and had the proper podCIDRs.

I assume this is what @smira meant with "re-join the kubelet".

Note: I only tested this on a single node setup.