[Release-1.26] - Dual-stack not supporting <IPv6><IPv4> #8502

manuelbuil · 2023-09-29T10:35:03Z

Backport fix for Dual-stack not supporting

Dual-stack not supporting <IPv6><IPv4> #8467

ShylajaDevadiga · 2023-10-19T23:57:16Z

Validated on release-1.26 branch using commit id `43d9986`

Environment Details

Infrastructure
Cloud EC2 instance

Node(s) CPU architecture, OS, and Version:
Ubuntu 22.04

Cluster Configuration:
3 server 1 agent

Config.yaml:

$ cat config.yaml 
node-ip: <IPv6>,192.168.29.91
token: secret
write-kubeconfig-mode: 644
cluster-cidr: 2001:cafe:42:0::/56,10.42.0.0/16
service-cidr: 2001:cafe:42:1::/112,10.43.0.0/16
cluster-init: true

Steps to reproduce the issue and validate the fix

Copy config.yaml
Install k3s

Replication results:
kube-dns shows ipv4 IP

$ kubectl get svc -A
NAMESPACE     NAME             TYPE           CLUSTER-IP             EXTERNAL-IP                                            PORT(S)                      AGE
default       kubernetes       ClusterIP      2001:cafe:42:1::1      <none>                                                 443/TCP                      10m
kube-system   kube-dns         ClusterIP      10.43.0.10             <none>                                                 53/UDP,53/TCP,9153/TCP       10m
kube-system   metrics-server   ClusterIP      2001:cafe:42:1::a1e8   <none>                                                 443/TCP                      10m
kube-system   traefik          LoadBalancer   2001:cafe:42:1::3155   192.168.29.91,<ipv6>   80:31548/TCP,443:31910/TCP   9m51s

coredns, metrics-server shows SingleStack

$ kubectl describe pod -n kube-system |grep IP -A 2
IP:                   10.42.0.6
IPs:
  IP:           10.42.0.6
  IP:           2001:cafe:42::6
Controlled By:  ReplicaSet/coredns-59b4f5bbd5
Containers:
--
IP:               10.42.0.3
IPs:
  IP:           10.42.0.3
  IP:           2001:cafe:42::3
Controlled By:  Job/helm-install-traefik

Services in dual-stack are includes Singlestack for core and metrics-server

$ kubectl describe svc -n kube-system    |grep  -i family -A4 -B2
Selector:          k8s-app=kube-dns
Type:              ClusterIP
IP Family Policy:  SingleStack
IP Families:       IPv4
IP:                10.43.0.10
IPs:               10.43.0.10
Port:              dns  53/UDP
--
Selector:          k8s-app=metrics-server
Type:              ClusterIP
IP Family Policy:  SingleStack
IP Families:       IPv6
IP:                2001:cafe:42:1::a1e8
IPs:               2001:cafe:42:1::a1e8
Port:              https  443/TCP
--
Selector:                 app.kubernetes.io/instance=traefik-kube-system,app.kubernetes.io/name=traefik
Type:                     LoadBalancer
IP Family Policy:         PreferDualStack
IP Families:              IPv6,IPv4
IP:                       2001:cafe:42:1::3155
IPs:                      2001:cafe:42:1::3155,10.43.131.173
LoadBalancer Ingress:     192.168.29.91, 2600:1f1c:ab4:ee48:c46f:b416:7f91:3578

Validation Results:

$ kubectl get svc -A
NAMESPACE     NAME             TYPE           CLUSTER-IP             EXTERNAL-IP                                                                                                                                                     PORT(S)                      AGE
default       kubernetes       ClusterIP      2001:cafe:42:1::1      <none>                                                                                                                                                          443/TCP                      46m
kube-system   kube-dns         ClusterIP      2001:cafe:42:1::a      <none>                                                                                                                                                          53/UDP,53/TCP,9153/TCP       46m
kube-system   metrics-server   ClusterIP      2001:cafe:42:1::f89    <none>                                                                                                                                                          443/TCP                      46m
kube-system   traefik          LoadBalancer   2001:cafe:42:1::f666   192.168.4.203,192.168.4.220,192.168.7.237,IPv6 REDACTED ,IPv6 REDACTED   80:30927/TCP,443:30154/TCP   45m

Services in dual-stack are all using IPFamily: PreferDualStack

$ kubectl describe svc -n kube-system    |grep  -i family -A4 -B2
Selector:          k8s-app=kube-dns
Type:              ClusterIP
IP Family Policy:  PreferDualStack
IP Families:       IPv6,IPv4
IP:                2001:cafe:42:1::a
IPs:               2001:cafe:42:1::a,10.43.0.10
Port:              dns  53/UDP
--
Selector:          k8s-app=metrics-server
Type:              ClusterIP
IP Family Policy:  PreferDualStack
IP Families:       IPv6,IPv4
IP:                2001:cafe:42:1::f89
IPs:               2001:cafe:42:1::f89,10.43.220.171
Port:              https  443/TCP
--
Selector:                 app.kubernetes.io/instance=traefik-kube-system,app.kubernetes.io/name=traefik
Type:                     LoadBalancer
IP Family Policy:         PreferDualStack
IP Families:              IPv6,IPv4
IP:                       2001:cafe:42:1::f666
IPs:                      2001:cafe:42:1::f666,10.43.50.180
LoadBalancer Ingress:     192.168.4.203, 192.168.4.220, 192.168.7.237, IPv6REDACTED, IPv6REDACTED, IPv6REDACTED

Pods display ipv4 IP address

$ kubectl get pods -A -o wide
NAMESPACE     NAME                                      READY   STATUS        RESTARTS   AGE   IP          NODE               NOMINATED NODE   READINESS GATES
default       multitool-deployment-968fbb87-8w62p       1/1     Running       0          40m   10.42.3.3   ip-192-168-4-203   <none>           <none>
default       multitool-deployment-968fbb87-cdrtj       1/1     Terminating   0          40m   10.42.1.3   ip-192-168-29-91   <none>           <none>
default       multitool-deployment-968fbb87-p2q4n       1/1     Running       0          31m   10.42.0.9   ip-192-168-7-237   <none>           <none>
default       multitool-deployment-968fbb87-z99bn       1/1     Running       0          40m   10.42.2.3   ip-192-168-4-220   <none>           <none>
kube-system   coredns-59b4f5bbd5-dtmc4                  1/1     Running       0          60m   10.42.0.5   ip-192-168-7-237   <none>           <none>
kube-system   helm-install-traefik-crd-9j8zv            0/1     Completed     0          60m   10.42.0.2   ip-192-168-7-237   <none>           <none>
kube-system   helm-install-traefik-h2wr4                0/1     Completed     1          60m   10.42.0.3   ip-192-168-7-237   <none>           <none>
kube-system   local-path-provisioner-76d776f6f9-qzfb6   1/1     Running       0          60m   10.42.0.4   ip-192-168-7-237   <none>           <none>
kube-system   metrics-server-68cf49699b-l7mth           1/1     Running       0          60m   10.42.0.6   ip-192-168-7-237   <none>           <none>
kube-system   svclb-traefik-620c66ea-57dk4              2/2     Running       0          43m   10.42.3.2   ip-192-168-4-203   <none>           <none>
kube-system   svclb-traefik-620c66ea-5fj5x              2/2     Running       0          59m   10.42.0.7   ip-192-168-7-237   <none>           <none>
kube-system   svclb-traefik-620c66ea-f4xl7              2/2     Running       0          43m   10.42.2.2   ip-192-168-4-220   <none>           <none>
kube-system   svclb-traefik-620c66ea-lppbg              2/2     Running       0          43m   10.42.1.2   ip-192-168-29-91   <none>           <none>
kube-system   traefik-f75d47c4b-g6mn4                   1/1     Running       0          59m   10.42.0.8   ip-192-168-7-237   <none>           <none>

Also the IP order is ipv4, ipv6 compared to v1.28 and v1.27

$ kubectl describe pod -n kube-system |grep IP -A 2
IP:                   10.42.0.5
IPs:
  IP:           10.42.0.5
  IP:           2001:cafe:42::5
Controlled By:  ReplicaSet/coredns-59b4f5bbd5
Containers:
--
IP:               10.42.0.2
IPs:
  IP:           10.42.0.2
  IP:           2001:cafe:42::2
Controlled By:  Job/helm-install-traefik-crd

ShylajaDevadiga · 2023-10-20T00:15:29Z

@rbrtbnfgl Please take a look at the results for the Pod IP in the above validation. With passing node-ip and without, results show the same IP order.

rbrtbnfgl · 2023-10-20T16:18:32Z

Kubelet with an external cloud provider on 1.26 doesn't support a dualstack node-ip, then we can't configure the node-ip as IPv6,IPv4 to prioritize IPv6 as primary podIP as we did on 1.27 and 1.28.

brandond · 2023-10-20T16:28:30Z

Even if we can't do it via --node-ip, we should be able to do it via our cloud provider, correct? The internal and external IPs set on the node via the InternalIP and ExternalIP annotations should respect the provided order. Are we not setting these correctly in dual-stack environments?

k3s/pkg/cloudprovider/instances.go

Lines 43 to 61 in 6aef26e

 // check internal address 

 if address := node.Annotations[InternalIPKey]; address != "" { 

 for _, v := range strings.Split(address, ",") { 

 addresses = append(addresses, corev1.NodeAddress{Type: corev1.NodeInternalIP, Address: v}) 

 } 

 } else if address = node.Labels[InternalIPKey]; address != "" { 

 addresses = append(addresses, corev1.NodeAddress{Type: corev1.NodeInternalIP, Address: address}) 

 } else { 

 logrus.Infof("Couldn't find node internal ip annotation or label on node %s", node.Name) 

 } 

 // check external address 

 if address := node.Annotations[ExternalIPKey]; address != "" { 

 for _, v := range strings.Split(address, ",") { 

 addresses = append(addresses, corev1.NodeAddress{Type: corev1.NodeExternalIP, Address: v}) 

 } 

 } else if address = node.Labels[ExternalIPKey]; address != "" { 

 addresses = append(addresses, corev1.NodeAddress{Type: corev1.NodeExternalIP, Address: address}) 

 }

rbrtbnfgl · 2023-10-20T21:13:51Z

I'll check if it's configured right on the cloud provider. There are a lot of part where that needed to be changed to prioritize IPv6 some of them was done by @manuelbuil maybe we missed some part.

rbrtbnfgl · 2023-10-23T09:42:18Z

The values for internal-ip and external-ip should be ordered correctly. The issue here is that the primary podIP is the IPv4 and not IPv6 and it should be related to the kubelet configuration.

rbrtbnfgl · 2023-10-23T14:41:33Z

I checked the kubelet code where the first podIP is selected. The function sortPodIPs https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/kubelet_pods.go#L1797 it's using the first configured nodeIP on kubelet to select the podIP to show.

ShylajaDevadiga · 2023-10-23T23:46:15Z

Thanks @rbrtbnfgl. Closing issue as validated as we can't prioritise the IPv6 on the list of the pods on v1.26 and v1.25

rbrtbnfgl · 2023-10-24T07:25:58Z

Small fix if you configure kubelet-arg: "--node-ip=::" it should order the IPv6 first for the pod.

manuelbuil self-assigned this Sep 29, 2023

manuelbuil mentioned this issue Sep 29, 2023

[Release 1.26] Take IPFamily precedence based on order #8505

Merged

manuelbuil added this to the v1.26.10+k3s1 milestone Sep 29, 2023

aganesh-suse assigned ShylajaDevadiga Oct 3, 2023

rbrtbnfgl mentioned this issue Oct 12, 2023

[Release 1.26] Use IPv6 in case is the first configured IP with dualstack #8598

Merged

ShylajaDevadiga mentioned this issue Oct 23, 2023

[Release-1.25] - Dual-stack not supporting <IPv6><IPv4> #8503

Closed

ShylajaDevadiga closed this as completed Oct 23, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Release-1.26] - Dual-stack not supporting <IPv6><IPv4> #8502

[Release-1.26] - Dual-stack not supporting <IPv6><IPv4> #8502

manuelbuil commented Sep 29, 2023

ShylajaDevadiga commented Oct 19, 2023

ShylajaDevadiga commented Oct 20, 2023

rbrtbnfgl commented Oct 20, 2023

brandond commented Oct 20, 2023

rbrtbnfgl commented Oct 20, 2023

rbrtbnfgl commented Oct 23, 2023

rbrtbnfgl commented Oct 23, 2023

ShylajaDevadiga commented Oct 23, 2023

rbrtbnfgl commented Oct 24, 2023

[Release-1.26] - Dual-stack not supporting <IPv6><IPv4> #8502

[Release-1.26] - Dual-stack not supporting <IPv6><IPv4> #8502

Comments

manuelbuil commented Sep 29, 2023

ShylajaDevadiga commented Oct 19, 2023

Validated on release-1.26 branch using commit id 43d9986

Environment Details

Steps to reproduce the issue and validate the fix

ShylajaDevadiga commented Oct 20, 2023

rbrtbnfgl commented Oct 20, 2023

brandond commented Oct 20, 2023

rbrtbnfgl commented Oct 20, 2023

rbrtbnfgl commented Oct 23, 2023

rbrtbnfgl commented Oct 23, 2023

ShylajaDevadiga commented Oct 23, 2023

rbrtbnfgl commented Oct 24, 2023

Validated on release-1.26 branch using commit id `43d9986`