Dual-stack not supporting <IPv6><IPv4> #8467

manuelbuil · 2023-09-27T09:58:01Z

Environmental Info:
K3s Version:

Any

Node(s) CPU architecture, OS, and Version:

Cluster Configuration:

Any

Describe the bug:

This bug is covered by a bigger bug: rancher/rke2#4772

If we deploy k3s with networking flags using the IPFamily order <IPv6><IPv4>, we are seeing problems. There are workarounds to those problems but even then, IPv4 takes precedence over IPv6, not honoring the order

Steps To Reproduce:

Installed K3s:

Deploy k3s in dual-stack mode but using:

cluster-cidr: 2001:cafe:42:0::/56,10.42.0.0/16
service-cidr: 2001:cafe:42:1::/112, 10.43.0.0/16

And you will see several problems happening (e.g. kube-api or netpol). Defining an "advertise-address" are disabling netpol makes k3s start but then you'll see that services IPs are all ipv4

Expected behavior:

User can set networking flags as <IPv6>,<IPv4> and k3s starts without problems. Moreover, IPv6 is prioritized over IPv4

Actual behavior:

Additional context / logs:

The text was updated successfully, but these errors were encountered:

ShylajaDevadiga · 2023-10-05T15:23:19Z

Validated using commit id `e82b376` on master branch

Environment Details

Infrastructure
Cloud EC2 instance

Node(s) CPU architecture, OS, and Version:
Ubuntu 22.04

Cluster Configuration:
3 server 1 agent

Config.yaml:

$ cat /etc/rancher/k3s/config.yaml 
write-kubeconfig-mode: 644
cluster-cidr: 2001:cafe:42:0::/56,10.42.0.0/16
service-cidr: 2001:cafe:42:1::/112,10.43.0.0/16
token: <TOKEN>
cluster-init: true

Steps to reproduce the issue and validate the fix

Copy config.yaml
Install k3s

Validation results:

$ kubectl get nodes 
NAME                STATUS   ROLES                       AGE     VERSION
ip-192-168-11-226   Ready    control-plane,etcd,master   4m48s   v1.28.2+k3s-e82b3764
ip-192-168-27-219   Ready    control-plane,etcd,master   8m37s   v1.28.2+k3s-e82b3764
ip-192-168-28-253   Ready    control-plane,etcd,master   2m47s   v1.28.2+k3s-e82b3764
ip-192-168-4-94     Ready    <none>                      3m10s   v1.28.2+k3s-e82b3764

Services in dual-stack are all using IPFamily: PreferDualStack

ubuntu@ip-192-168-27-219:~$ kubectl get svc -A
NAMESPACE     NAME             TYPE           CLUSTER-IP             EXTERNAL-IP                                                                                                                                                                                                             PORT(S)                      AGE
default       kubernetes       ClusterIP      2001:cafe:42:1::1      <none>                                                                                                                                                                                                                  443/TCP                      9m16s
kube-system   kube-dns         ClusterIP      2001:cafe:42:1::a      <none>                                                                                                                                                                                                                  53/UDP,53/TCP,9153/TCP       9m13s
kube-system   metrics-server   ClusterIP      2001:cafe:42:1::9aaf   <none>                                                                                                                                                                                                                  443/TCP                      9m11s
kube-system   traefik          LoadBalancer   2001:cafe:42:1::faf9   192.168.11.226,192.168.27.219,192.168.28.253,192.168.4.94,2600:<IP1>,2600:<IP2>,2600:<IP3>,2600:<IP4>  80:30160/TCP,443:31704/TCP   8m43s
ubuntu@ip-192-168-27-219:~$ kubectl describe svc -n kube-system  kube-dns  |grep -i prefer
IP Family Policy:  PreferDualStack
ubuntu@ip-192-168-27-219:~$ kubectl describe svc -n kube-system  metrics-server  |grep -i prefer
IP Family Policy:  PreferDualStack
ubuntu@ip-192-168-27-219:~$ kubectl describe svc -n kube-system  traefik   |grep -i prefer
IP Family Policy:         PreferDualStack
ubuntu@ip-192-168-27-219:~$

Pods have both ipv4 and ipv6 IPs. Pods list ipv4. Fix coming in soon

ubuntu@ip-192-168-27-219:~$ kubectl get pods -A -o wide
NAMESPACE     NAME                                      READY   STATUS      RESTARTS   AGE     IP          NODE                NOMINATED NODE   READINESS GATES
kube-system   coredns-6799fbcd5-rgfdx                   1/1     Running     0          11m     10.42.0.6   ip-192-168-27-219   <none>           <none>
kube-system   helm-install-traefik-8wbwr                0/1     Completed   1          11m     10.42.0.3   ip-192-168-27-219   <none>           <none>
kube-system   helm-install-traefik-crd-k6fq8            0/1     Completed   0          11m     10.42.0.4   ip-192-168-27-219   <none>           <none>
kube-system   local-path-provisioner-84db5d44d9-8mc27   1/1     Running     0          11m     10.42.0.5   ip-192-168-27-219   <none>           <none>
kube-system   metrics-server-67c658944b-bl5wp           1/1     Running     0          11m     10.42.0.2   ip-192-168-27-219   <none>           <none>
kube-system   svclb-traefik-35952bc4-4fskh              2/2     Running     0          7m58s   10.42.1.2   ip-192-168-11-226   <none>           <none>
kube-system   svclb-traefik-35952bc4-924sz              2/2     Running     0          11m     10.42.0.7   ip-192-168-27-219   <none>           <none>
kube-system   svclb-traefik-35952bc4-ghcrq              2/2     Running     0          6m21s   10.42.2.2   ip-192-168-4-94     <none>           <none>
kube-system   svclb-traefik-35952bc4-mv4zj              2/2     Running     0          5m57s   10.42.3.2   ip-192-168-28-253   <none>           <none>
kube-system   traefik-7bf7d7576d-rhdhj                  1/1     Running     0          11m     10.42.0.8   ip-192-168-27-219   <none>           <none>
ubuntu@ip-192-168-27-219:~$ kubectl describe pod -n kube-system |grep IP -A 5
IP:                   10.42.0.6
IPs:
  IP:           10.42.0.6
  IP:           2001:cafe:42::6
Controlled By:  ReplicaSet/coredns-6799fbcd5
Containers:
  coredns:
    Container ID:  containerd://1ac602b5b81b249f0df3b29b870d0da06011fe6624fc153cb526b801308a334f
    Image:         rancher/mirrored-coredns-coredns:1.10.1
--
IP:               10.42.0.3
IPs:
  IP:           10.42.0.3
  IP:           2001:cafe:42::3
Controlled By:  Job/helm-install-traefik
Containers:
  helm:
    Container ID:  containerd://b7c771afd8ed3a31a9088838bb00fae00439aa90e52a61c7f8ef8e9a0b066eae
    Image:         rancher/klipper-helm:v0.8.2-build20230815

Both ipv4 as well as ipv6 ip are available

$ kubectl exec -it multitool-deployment-564b975b9c-46475 -- ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0@if7: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 8951 qdisc noqueue state UP group default 
    link/ether 22:fc:8a:d4:fc:75 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 10.42.1.3/24 brd 10.42.1.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 2001:cafe:42:2::3/64 scope global 
       valid_lft forever preferred_lft forever
    inet6 fe80::20fc:8aff:fed4:fc75/64 scope link 
       valid_lft forever preferred_lft forever

Validating pod to pod communication

ubuntu@ip-192-168-27-219:~$ kubectl exec -it multitool-deployment-564b975b9c-46475 -- ping 2001:cafe:42:3::3
PING 2001:cafe:42:3::3(2001:cafe:42:3::3) 56 data bytes
64 bytes from 2001:cafe:42:3::3: icmp_seq=1 ttl=62 time=0.977 ms
^C
--- 2001:cafe:42:3::3 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.977/0.977/0.977/0.000 ms
ubuntu@ip-192-168-27-219:~$ kubectl exec -it multitool-deployment-564b975b9c-46475 -- ping 2001:cafe:42:4::3
PING 2001:cafe:42:4::3(2001:cafe:42:4::3) 56 data bytes
64 bytes from 2001:cafe:42:4::3: icmp_seq=1 ttl=62 time=0.420 ms
64 bytes from 2001:cafe:42:4::3: icmp_seq=2 ttl=62 time=0.300 ms
^C
--- 2001:cafe:42:4::3 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1009ms
rtt min/avg/max/mdev = 0.300/0.360/0.420/0.060 ms
ubuntu@ip-192-168-27-219:~$ kubectl exec -it multitool-deployment-564b975b9c-46475 -- ping 2001:cafe:42:2::3
PING 2001:cafe:42:2::3(2001:cafe:42:2::3) 56 data bytes
64 bytes from 2001:cafe:42:2::3: icmp_seq=1 ttl=64 time=0.027 ms
64 bytes from 2001:cafe:42:2::3: icmp_seq=2 ttl=64 time=0.035 ms
^C
--- 2001:cafe:42:2::3 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1030ms
rtt min/avg/max/mdev = 0.027/0.031/0.035/0.004 ms
ubuntu@ip-192-168-27-219:~$ kubectl exec -it multitool-deployment-564b975b9c-46475 -- ping 10.42.3.3
PING 10.42.3.3 (10.42.3.3) 56(84) bytes of data.
64 bytes from 10.42.3.3: icmp_seq=1 ttl=62 time=1.96 ms
^C
--- 10.42.3.3 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 1.958/1.958/1.958/0.000 ms
ubuntu@ip-192-168-27-219:~$

rbrtbnfgl · 2023-10-13T08:24:00Z

reopened for testing

ShylajaDevadiga · 2023-10-19T19:58:46Z

Validated fix to prioritize IPv6 on the pods using commit id 5b6b968.
Config.yaml

write-kubeconfig-mode: 644
cluster-cidr: 2001:cafe:42:0::/56,10.42.0.0/16
service-cidr: 2001:cafe:42:1::/112,10.43.0.0/16
token: secret
cluster-init: true
node-ip: <IPv6>,<IPv4>

ubuntu@ip-192-168-7-237:~$ k3s -v
k3s version v1.28.2+k3s-5b6b9685 (5b6b9685)
go version go1.20.8
ubuntu@ip-192-168-7-237:~$ kubectl get svc -A
NAMESPACE     NAME             TYPE           CLUSTER-IP             EXTERNAL-IP                                           PORT(S)                      AGE
default       kubernetes       ClusterIP      2001:cafe:42:1::1      <none>                                                443/TCP                      50s
kube-system   kube-dns         ClusterIP      2001:cafe:42:1::a      <none>                                                53/UDP,53/TCP,9153/TCP       47s
kube-system   metrics-server   ClusterIP      2001:cafe:42:1::da4e   <none>                                                443/TCP                      46s
kube-system   traefik          LoadBalancer   2001:cafe:42:1::82b6   192.168.7.237,<IPv6>   80:30535/TCP,443:31068/TCP   18s
ubuntu@ip-192-168-7-237:~$ kubectl get pods -o wide -A
NAMESPACE     NAME                                      READY   STATUS      RESTARTS   AGE   IP                NODE               NOMINATED NODE   READINESS GATES
kube-system   coredns-6799fbcd5-xtqqs                   1/1     Running     0          36s   2001:cafe:42::3   ip-192-168-7-237   <none>           <none>
kube-system   helm-install-traefik-crd-c96dl            0/1     Completed   0          36s   2001:cafe:42::6   ip-192-168-7-237   <none>           <none>
kube-system   helm-install-traefik-kn7nn                0/1     Completed   1          36s   2001:cafe:42::2   ip-192-168-7-237   <none>           <none>
kube-system   local-path-provisioner-84db5d44d9-frzc4   1/1     Running     0          36s   2001:cafe:42::4   ip-192-168-7-237   <none>           <none>
kube-system   metrics-server-67c658944b-s7pcp           1/1     Running     0          36s   2001:cafe:42::5   ip-192-168-7-237   <none>           <none>
kube-system   svclb-traefik-2e9cfee1-g8k26              2/2     Running     0          20s   2001:cafe:42::7   ip-192-168-7-237   <none>           <none>
kube-system   traefik-55f65f58b-wm5j6                   1/1     Running     0          20s   2001:cafe:42::8   ip-192-168-7-237   <none>           <none>
ubuntu@ip-192-168-7-237:~$

ShylajaDevadiga · 2023-10-19T21:49:25Z

On a multi-node cluster

$ kubectl get nodes
NAME               STATUS   ROLES                       AGE     VERSION
ip-192-168-4-203   Ready    control-plane,etcd,master   3m32s   v1.28.2+k3s-5b6b9685
ip-192-168-4-220   Ready    control-plane,etcd,master   3m16s   v1.28.2+k3s-5b6b9685
ip-192-168-7-237   Ready    control-plane,etcd,master   6m51s   v1.28.2+k3s-5b6b9685

Both ipv4 as well as ipv6 ip are available, prioritizing ipv6 ip

65f58b-6fhgw                   1/1     Running     0          6m23s
ubuntu@ip-192-168-7-237:~$ kubectl describe pod -n kube-system |grep IP -A 2
IP:                   2001:cafe:42::4
IPs:
  IP:           2001:cafe:42::4
  IP:           10.42.0.4
Controlled By:  ReplicaSet/coredns-6799fbcd5
Containers:
--
IP:               2001:cafe:42::2
IPs:
  IP:           2001:cafe:42::2
  IP:           10.42.0.2
Controlled By:  Job/helm-install-traefik-crd
Containers:
--
IP:               2001:cafe:42::6
IPs:
  IP:           2001:cafe:42::6
  IP:           10.42.0.6
Controlled By:  Job/helm-install-traefik
Containers:
--
IP:                   2001:cafe:42::5
IPs:
  IP:           2001:cafe:42::5
  IP:           10.42.0.5
Controlled By:  ReplicaSet/local-path-provisioner-84db5d44d9
Containers:
--
IP:                   2001:cafe:42::3
IPs:
  IP:           2001:cafe:42::3
  IP:           10.42.0.3
Controlled By:  ReplicaSet/metrics-server-67c658944b

Re-validating services in dual-stack are all using IPFamily: PreferDualStack

$ kubectl describe svc -n kube-system    |grep  -E 'Name|Prefer'
Name:              kube-dns
Namespace:         kube-system
IP Family Policy:  PreferDualStack
Name:              metrics-server
Namespace:         kube-system
IP Family Policy:  PreferDualStack
Name:                     traefik
Namespace:                kube-system
IP Family Policy:         PreferDualStack

Validating Pod to Pod communication

$ kubectl exec -it multitool-deployment-5d779cf6bc-6gjbp bash
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.
multitool-deployment-5d779cf6bc-6gjbp:/# ping 2001:cafe:42::9
PING 2001:cafe:42::9(2001:cafe:42::9) 56 data bytes
64 bytes from 2001:cafe:42::9: icmp_seq=1 ttl=62 time=0.485 ms
64 bytes from 2001:cafe:42::9: icmp_seq=2 ttl=62 time=0.485 ms
^C
--- 2001:cafe:42::9 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1008ms
rtt min/avg/max/mdev = 0.485/0.485/0.485/0.000 ms
multitool-deployment-5d779cf6bc-6gjbp:/# ping 2001:cafe:42:3::3 
PING 2001:cafe:42:3::3(2001:cafe:42:3::3) 56 data bytes
64 bytes from 2001:cafe:42:3::3: icmp_seq=1 ttl=62 time=0.402 ms
64 bytes from 2001:cafe:42:3::3: icmp_seq=2 ttl=62 time=1.02 ms
^C

brandond · 2023-10-19T21:54:38Z

@ShylajaDevadiga I did note ServiceLB appears to still prefer IPv4 over IPv6 in the list - is that an issue?

NAMESPACE     NAME             TYPE           CLUSTER-IP             EXTERNAL-IP            PORT(S)    
kube-system   traefik          LoadBalancer   2001:cafe:42:1::82b6   192.168.7.237,<IPv6>   80:30535/TCP,443:31068/TCP   18s

Based on this it looks like the service ClusterIP is IPv6 only - is that correct, or is it actually dual-stack?

ShylajaDevadiga · 2023-10-19T22:35:03Z

@brandond service ClusterIP has both ipv6 as well as ipv4, ipv6 is only displayed in the output. Order is ServiceLB is ipv4 first. @rbrtbnfgl please advice

$ kubectl get svc -n kube-system traefik -o yaml
apiVersion: v1
kind: Service
metadata:
  annotations:
    meta.helm.sh/release-name: traefik
    meta.helm.sh/release-namespace: kube-system
  creationTimestamp: "2023-10-19T22:38:54Z"
  finalizers:
  - service.kubernetes.io/load-balancer-cleanup
  labels:
    app.kubernetes.io/instance: traefik-kube-system
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: traefik
    helm.sh/chart: traefik-21.2.1_up21.2.0
  name: traefik
  namespace: kube-system
  resourceVersion: "1681"
  uid: 994b0bf0-5634-4ca9-be83-c72372b54c48
spec:
  allocateLoadBalancerNodePorts: true
  clusterIP: 2001:cafe:42:1::dac9
  clusterIPs:
  - 2001:cafe:42:1::dac9
  - 10.43.46.165
  externalTrafficPolicy: Cluster
  internalTrafficPolicy: Cluster
  ipFamilies:
  - IPv6
  - IPv4
  ipFamilyPolicy: PreferDualStack
  ports:
  - name: web
    nodePort: 30620
    port: 80
    protocol: TCP
    targetPort: web
  - name: websecure
    nodePort: 30907
    port: 443
    protocol: TCP
    targetPort: websecure
  selector:
    app.kubernetes.io/instance: traefik-kube-system
    app.kubernetes.io/name: traefik
  sessionAffinity: None
  type: LoadBalancer
status:
  loadBalancer:
    ingress:
    - ip: 192.168.29.91
    - ip: 192.168.4.220
    - ip: 192.168.7.237
    - ip: <ipv6>
    - ip: <ipv6>
    - ip: <ipv6>

brandond · 2023-10-19T22:46:49Z

Opened ServiceLB return ingress address list in alphabetical order, instead of sorted by address family #8704 to track the separate issue with servicelb

ShylajaDevadiga · 2023-10-19T22:47:42Z

Thanks @brandond

manuelbuil self-assigned this Sep 27, 2023

manuelbuil added this to the v1.28.3+k3s1 milestone Sep 27, 2023

ShylajaDevadiga self-assigned this Oct 3, 2023

rbrtbnfgl mentioned this issue Oct 10, 2023

Use IPv6 in case is the first configured IP with dualstack #8581

Merged

rbrtbnfgl closed this as completed in #8581 Oct 13, 2023

rbrtbnfgl reopened this Oct 13, 2023

ShylajaDevadiga closed this as completed Oct 19, 2023

ShylajaDevadiga reopened this Oct 19, 2023

ShylajaDevadiga closed this as completed Oct 19, 2023

mdrahman-suse mentioned this issue Nov 17, 2023

Cluster data is not showing <IPv4> IPs in a Dual-Stack setup #8901

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Dual-stack not supporting <IPv6><IPv4> #8467

Dual-stack not supporting <IPv6><IPv4> #8467

manuelbuil commented Sep 27, 2023 •

edited by brandond

Loading

ShylajaDevadiga commented Oct 5, 2023 •

edited

Loading

rbrtbnfgl commented Oct 13, 2023 •

edited

Loading

ShylajaDevadiga commented Oct 19, 2023

ShylajaDevadiga commented Oct 19, 2023

brandond commented Oct 19, 2023 •

edited

Loading

ShylajaDevadiga commented Oct 19, 2023 •

edited

Loading

brandond commented Oct 19, 2023 •

edited

Loading

ShylajaDevadiga commented Oct 19, 2023

Dual-stack not supporting <IPv6><IPv4> #8467

Dual-stack not supporting <IPv6><IPv4> #8467

Comments

manuelbuil commented Sep 27, 2023 • edited by brandond Loading

ShylajaDevadiga commented Oct 5, 2023 • edited Loading

Validated using commit id e82b376 on master branch

Environment Details

Steps to reproduce the issue and validate the fix

rbrtbnfgl commented Oct 13, 2023 • edited Loading

ShylajaDevadiga commented Oct 19, 2023

ShylajaDevadiga commented Oct 19, 2023

brandond commented Oct 19, 2023 • edited Loading

ShylajaDevadiga commented Oct 19, 2023 • edited Loading

brandond commented Oct 19, 2023 • edited Loading

ShylajaDevadiga commented Oct 19, 2023

manuelbuil commented Sep 27, 2023 •

edited by brandond

Loading

ShylajaDevadiga commented Oct 5, 2023 •

edited

Loading

Validated using commit id `e82b376` on master branch

rbrtbnfgl commented Oct 13, 2023 •

edited

Loading

brandond commented Oct 19, 2023 •

edited

Loading

ShylajaDevadiga commented Oct 19, 2023 •

edited

Loading

brandond commented Oct 19, 2023 •

edited

Loading