Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Dual-stack IPv6 route is lost #9896

Open
LIENZHIADMIN opened this issue Feb 26, 2025 · 0 comments
Open

Dual-stack IPv6 route is lost #9896

LIENZHIADMIN opened this issue Feb 26, 2025 · 0 comments

Comments

@LIENZHIADMIN
Copy link

LIENZHIADMIN commented Feb 26, 2025 •
edited
Loading

Expected Behavior

Create an ipv6 route correctly

Current Behavior

Description:

I created a cluster with IPv4 and IPv6 dual-stack using kubeadm.

  • Kubernetes version: v1.29.5
  • Pod CIDR: 172.99.0.0/16, 2001:db8:42::/64
  • Calico version: Client Version: v3.28.0
  • os version: ubuntu22.04

After enabling the IPv4 and IPv6 dual-stack network, only the IPv4 routing rules were created. The IPv6 routing rules were created but immediately deleted.

The routing results are as follows:

ip -4 r | grep calid2c5b465b5c
172.99.1.5 dev calid2c5b465b5c scope link 

ip -6 r | grep calid2c5b465b5c
fe80::/64 dev calid2c5b465b5c proto kernel metric 256 pref medium

Calico-node logs show that initially an IPv6 IP (2001:db8:42:0:208b:133f:7967:ee05) was created, but it was later removed:

2025-02-26 08:32:55.422 [INFO][82] felix/int_dataplane.go 1891: Received *proto.ServiceRemove update from calculation graph msg=name:"chenby" namespace:"default" 
2025-02-26 08:33:10.944 [INFO][82] felix/summary.go 100: Summarising 11 dataplane reconciliation loops over 1m3.3s: avg=18ms longest=40ms (resync-filter-v6,resync-mangle-v6,resync-nat-v6,resync-raw-v6)
2025-02-26 08:33:14.679 [INFO][82] felix/int_dataplane.go 1891: Received *proto.ServiceUpdate update from calculation graph msg=name:"chenby" namespace:"default" type:"NodePort" cluster_ip:"2002:db8:42:1::a8f3" ports:<Protocol:"TCP" Port:80 NodePort:30128 > 
2025-02-26 08:33:15.705 [INFO][82] felix/calc_graph.go 507: Local endpoint updated id=WorkloadEndpoint(node=gtai01, orchestrator=k8s, workload=default/chenby-7975959574-n52d8, name=eth0)
2025-02-26 08:33:15.706 [INFO][82] felix/int_dataplane.go 1891: Received *proto.WorkloadEndpointUpdate update from calculation graph msg=id:<orchestrator_id:"k8s" workload_id:"default/chenby-7975959574-n52d8" endpoint_id:"eth0" > endpoint:<state:"active" name:"calid2c5b465b5c" profile_ids:"kns.default" profile_ids:"ksa.default.default" ipv4_nets:"172.99.1.5/32" ipv6_nets:"2001:db8:42:0:208b:133f:7967:ee05/128" > 
2025-02-26 08:33:15.706 [INFO][82] felix/endpoint_mgr.go 700: Updating per-endpoint chains. id=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"default/chenby-7975959574-n52d8", EndpointId:"eth0"}
2025-02-26 08:33:15.707 [INFO][82] felix/endpoint_mgr.go 737: Updating endpoint routes. id=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"default/chenby-7975959574-n52d8", EndpointId:"eth0"}
2025-02-26 08:33:15.707 [INFO][82] felix/table.go 611: Chain became referenced, marking it for programming chainName="cali-fw-calid2c5b465b5c" ipVersion=0x4 table="filter"
2025-02-26 08:33:15.707 [INFO][82] felix/table.go 611: Chain became referenced, marking it for programming chainName="cali-tw-calid2c5b465b5c" ipVersion=0x4 table="filter"
2025-02-26 08:33:15.708 [INFO][82] felix/endpoint_mgr.go 1430: Skipping configuration of interface because it is oper down. ifaceName="calid2c5b465b5c"
2025-02-26 08:33:15.708 [INFO][82] felix/table.go 611: Chain became referenced, marking it for programming chainName="cali-sm-calid2c5b465b5c" ipVersion=0x4 table="filter"
2025-02-26 08:33:15.708 [INFO][82] felix/endpoint_mgr.go 586: Re-evaluated workload endpoint status adminUp=true failed=false known=true operUp=false status="down" workloadEndpointID=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"default/chenby-7975959574-n52d8", EndpointId:"eth0"}
2025-02-26 08:33:15.708 [INFO][82] felix/status_combiner.go 58: Storing endpoint status update ipVersion=0x4 status="down" workload=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"default/chenby-7975959574-n52d8", EndpointId:"eth0"}
2025-02-26 08:33:15.709 [INFO][82] felix/endpoint_mgr.go 700: Updating per-endpoint chains. id=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"default/chenby-7975959574-n52d8", EndpointId:"eth0"}
2025-02-26 08:33:15.709 [INFO][82] felix/endpoint_mgr.go 737: Updating endpoint routes. id=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"default/chenby-7975959574-n52d8", EndpointId:"eth0"}
2025-02-26 08:33:15.709 [INFO][82] felix/table.go 611: Chain became referenced, marking it for programming chainName="cali-fw-calid2c5b465b5c" ipVersion=0x6 table="filter"
2025-02-26 08:33:15.709 [INFO][82] felix/table.go 611: Chain became referenced, marking it for programming chainName="cali-tw-calid2c5b465b5c" ipVersion=0x6 table="filter"
2025-02-26 08:33:15.710 [INFO][82] felix/endpoint_mgr.go 1430: Skipping configuration of interface because it is oper down. ifaceName="calid2c5b465b5c"
2025-02-26 08:33:15.710 [INFO][82] felix/table.go 611: Chain became referenced, marking it for programming chainName="cali-sm-calid2c5b465b5c" ipVersion=0x6 table="filter"
2025-02-26 08:33:15.710 [INFO][82] felix/endpoint_mgr.go 586: Re-evaluated workload endpoint status adminUp=true failed=false known=true operUp=false status="down" workloadEndpointID=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"default/chenby-7975959574-n52d8", EndpointId:"eth0"}
2025-02-26 08:33:15.710 [INFO][82] felix/status_combiner.go 58: Storing endpoint status update ipVersion=0x6 status="down" workload=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"default/chenby-7975959574-n52d8", EndpointId:"eth0"}
2025-02-26 08:33:15.755 [INFO][82] felix/status_combiner.go 78: Endpoint down for at least one IP version id=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"default/chenby-7975959574-n52d8", EndpointId:"eth0"} ipVersion=0x6 status="down"
2025-02-26 08:33:15.755 [INFO][82] felix/status_combiner.go 78: Endpoint down for at least one IP version id=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"default/chenby-7975959574-n52d8", EndpointId:"eth0"} ipVersion=0x4 status="down"
2025-02-26 08:33:15.756 [INFO][82] felix/status_combiner.go 98: Reporting combined status. id=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"default/chenby-7975959574-n52d8", EndpointId:"eth0"} status="down"
2025-02-26 08:33:15.791 [INFO][82] felix/int_dataplane.go 1314: Linux interface state changed. ifIndex=16132 ifaceName="calid2c5b465b5c" state="down"
2025-02-26 08:33:15.791 [INFO][82] felix/int_dataplane.go 1948: Received interface update msg=&intdataplane.ifaceStateUpdate{Name:"calid2c5b465b5c", State:"down", Index:16132}
2025-02-26 08:33:15.792 [INFO][82] felix/int_dataplane.go 1358: Linux interface addrs changed. addrs=set.Set{fe80::ecee:eeff:feee:eeee} ifaceName="calid2c5b465b5c"
2025-02-26 08:33:15.792 [INFO][82] felix/endpoint_mgr.go 527: Workload interface state changed; marking for status update. ifaceName="calid2c5b465b5c"
2025-02-26 08:33:15.792 [INFO][82] felix/endpoint_mgr.go 527: Workload interface state changed; marking for status update. ifaceName="calid2c5b465b5c"
2025-02-26 08:33:15.792 [INFO][82] felix/iface_monitor.go 238: Netlink address update for known interface.  addr="fe80::ecee:eeff:feee:eeee" exists=true ifIndex=16132
2025-02-26 08:33:15.792 [INFO][82] felix/endpoint_mgr.go 586: Re-evaluated workload endpoint status adminUp=true failed=false known=true operUp=false status="down" workloadEndpointID=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"default/chenby-7975959574-n52d8", EndpointId:"eth0"}
2025-02-26 08:33:15.793 [INFO][82] felix/int_dataplane.go 1314: Linux interface state changed. ifIndex=16132 ifaceName="calid2c5b465b5c" state="up"
2025-02-26 08:33:15.793 [INFO][82] felix/status_combiner.go 58: Storing endpoint status update ipVersion=0x4 status="down" workload=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"default/chenby-7975959574-n52d8", EndpointId:"eth0"}
2025-02-26 08:33:15.793 [INFO][82] felix/endpoint_mgr.go 586: Re-evaluated workload endpoint status adminUp=true failed=false known=true operUp=false status="down" workloadEndpointID=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"default/chenby-7975959574-n52d8", EndpointId:"eth0"}
2025-02-26 08:33:15.793 [INFO][82] felix/status_combiner.go 58: Storing endpoint status update ipVersion=0x6 status="down" workload=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"default/chenby-7975959574-n52d8", EndpointId:"eth0"}
2025-02-26 08:33:15.794 [INFO][82] felix/status_combiner.go 78: Endpoint down for at least one IP version id=proto.WorkloadEndpointID{
...
2025-02-26 08:33:15.798 [INFO][82] felix/status_combiner.go 58: Storing endpoint status update ipVersion=0x6 status="up" workload=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"default/chenby-7975959574-n52d8", EndpointId:"eth0"}
2025-02-26 08:33:15.800 [INFO][82] felix/status_combiner.go 81: Endpoint up for at least one IP version id=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"default/chenby-7975959574-n52d8", EndpointId:"eth0"} ipVersion=0x4 status="up"
2025-02-26 08:33:15.800 [INFO][82] felix/status_combiner.go 98: Reporting combined status. id=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"default/chenby-7975959574-n52d8", EndpointId:"eth0"} status="up"
2025-02-26 08:33:16.448 [INFO][82] felix/calc_graph.go 507: Local endpoint updated id=WorkloadEndpoint(node=gtai01, orchestrator=k8s, workload=default/chenby-7975959574-n52d8, name=eth0)
2025-02-26 08:33:16.449 [INFO][82] felix/int_dataplane.go 1891: Received *proto.WorkloadEndpointUpdate update from calculation graph msg=id:<orchestrator_id:"k8s" workload_id:"default/chenby-7975959574-n52d8" endpoint_id:"eth0" > endpoint:<state:"active" name:"calid2c5b465b5c" profile_ids:"kns.default" profile_ids:"ksa.default.default" ipv4_nets:"172.99.1.5/32" > 
2025-02-26 08:33:16.449 [INFO][82] felix/endpoint_mgr.go 700: Updating per-endpoint chains. id=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"default/chenby-7975959574-n52d8", EndpointId:"eth0"}
2025-02-26 08:33:16.449 [INFO][82] felix/endpoint_mgr.go 737: Updating endpoint routes. id=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"default/chenby-7975959574-n52d8", EndpointId:"eth0"}
2025-02-26 08:33:16.450 [INFO][82] felix/endpoint_mgr.go 1362: Applying /proc/sys configuration to interface. ifaceName="calid2c5b465b5c"
2025-02-26 08:33:16.451 [INFO][82] felix/endpoint_mgr.go 586: Re-evaluated workload endpoint status adminUp=true failed=false known=true operUp=true status="up" workloadEndpointID=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"default/chenby-7975959574-n52d8", EndpointId:"eth0"}
2025-02-26 08:33:16.451 [INFO][82] felix/status_combiner.go 58: Storing endpoint status update ipVersion=0x4 status="up" workload=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"default/chenby-7975959574-n52d8", EndpointId:"eth0"}
2025-02-26 08:33:16.451 [INFO][82] felix/endpoint_mgr.go 700: Updating per-endpoint chains. id=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"default/chenby-7975959574-n52d8", EndpointId:"eth0"}
2025-02-26 08:33:16.451 [INFO][82] felix/endpoint_mgr.go 737: Updating endpoint routes. id=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"default/chenby-7975959574-n52d8", EndpointId:"eth0"}
2025-02-26 08:33:16.452 [INFO][82] felix/endpoint_mgr.go 1362: Applying /proc/sys configuration to interface. ifaceName="calid2c5b465b5c"
2025-02-26 08:33:16.452 [INFO][82] felix/endpoint_mgr.go 586: Re-evaluated workload endpoint status adminUp=true failed=false known=true operUp=true status="up" workloadEndpointID=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"default/chenby-7975959574-n52d8", EndpointId:"eth0"}
2025-02-26 08:33:16.453 [INFO][82] felix/status_combiner.go 58: Storing endpoint status update ipVersion=0x6 status="up" workload=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"default/chenby-7975959574-n52d8", EndpointId:"eth0"}
2025-02-26 08:33:16.454 [INFO][82] felix/conntrack.go 90: Removing conntrack flows ip=2001:db8:42:0:208b:133f:7967:ee05
2025-02-26 08:33:16.482 [INFO][82] felix/status_combiner.go 81: Endpoint up for at least one IP version id=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"default/chenby-7975959574-n52d8", EndpointId:"eth0"} ipVersion=0x4 status="up"
2025-02-26 08:33:16.483 [INFO][82] felix/status_combiner.go 98: Reporting combined status. id=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"default/chenby-7975959574-n52d8", EndpointId:"eth0"} status="up"

log debug

...
2025-02-26 09:08:09.035 [DEBUG][90] felix/ipsets.go 713: No dirty IP sets. family="inet"
2025-02-26 09:08:09.036 [DEBUG][90] felix/table.go 1327: Update ended up being no-op, skipping call to ip(6)tables-restore. ipVersion=0x6 table="raw"
2025-02-26 09:08:09.036 [DEBUG][90] felix/table.go 1327: Update ended up being no-op, skipping call to ip(6)tables-restore. ipVersion=0x4 table="mangle"
2025-02-26 09:08:09.036 [DEBUG][90] felix/feature_detect_linux.go 120: Refreshing detected iptables features
2025-02-26 09:08:09.036 [DEBUG][90] felix/route_table.go 770: Create L3 route for: routetable.Target{Type:"", CIDR:ip.V6CIDR{addr:ip.V6Addr{0x20, 0x1, 0xd, 0xb8, 0x0, 0x42, 0x0, 0x0, 0x20, 0x8b, 0x13, 0x3f, 0x79, 0x67, 0xee, 0x6}, prefix:0x80}, GW:ip.Addr(nil), Src:ip.Addr(nil), DestMAC:net.HardwareAddr(nil)}
2025-02-26 09:08:09.037 [DEBUG][90] felix/route_table.go 984: Starting goroutine to delete conntrack entries ip=2001:db8:42:0:208b:133f:7967:ee06
2025-02-26 09:08:09.037 [DEBUG][90] felix/route_table.go 551: Synchronised routes on interface ifaceName="cali81f47b84923" ifaceRegex="^cali.*" ipVersion=0x6 tableIndex=254
2025-02-26 09:08:09.037 [DEBUG][90] felix/route_table.go 1003: Background goroutine yet to finish deleting conntrack entries ip=2001:db8:42:0:208b:133f:7967:ee06
2025-02-26 09:08:09.037 [DEBUG][90] felix/update_filter.go 115: Route update route={Ifindex: 16139 Dst: 2001:db8:42:0:208b:133f:7967:ee06/128 Src: <nil> Gw: <nil> Flags: [] Table: 254 Realm: 0}
2025-02-26 09:08:09.037 [DEBUG][90] felix/route_table.go 551: Synchronised routes on interface ifaceName="cali81f47b84923" ifaceRegex="^cali.*" ipVersion=0x4 tableIndex=254
2025-02-26 09:08:09.037 [INFO][90] felix/conntrack.go 90: Removing conntrack flows ip=2001:db8:42:0:208b:133f:7967:ee06
2025-02-26 09:08:09.037 [DEBUG][90] felix/update_filter.go 117: Ignoring non-local route. route={Ifindex: 16139 Dst: 2001:db8:42:0:208b:133f:7967:ee06/128 Src: <nil> Gw: <nil> Flags: [] Table: 254 Realm: 0}

ipvsadm rule

root@gtai01:/home/gtai01/lienzhi/calico# ipvsadm -L -n
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  172.17.0.1:30128 rr
  -> 172.99.1.5:80                Masq    1      0          0         
TCP  10.64.7.145:30128 rr
  -> 172.99.1.5:80                Masq    1      0          0         
TCP  10.96.0.1:443 rr
  -> 10.64.7.145:6443             Masq    1      0          0         
TCP  10.96.0.10:53 rr
  -> 172.99.1.0:53                Masq    1      0          0         
  -> 172.99.1.1:53                Masq    1      0          0         
TCP  10.96.0.10:9153 rr
  -> 172.99.1.0:9153              Masq    1      0          0         
  -> 172.99.1.1:9153              Masq    1      0          0         
TCP  10.96.175.114:5473 rr
  -> 10.64.7.145:5473             Masq    1      0          0         
TCP  10.96.200.54:80 rr
  -> 172.99.1.5:80                Masq    1      0          0         
UDP  10.96.0.10:53 rr
  -> 172.99.1.0:53                Masq    1      0          0         
  -> 172.99.1.1:53                Masq    1      0          0         
TCP  [2002:db8:42:1::a8f3]:80 rr
TCP  [2003:db8:1::1]:30128 rr
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@LIENZHIADMIN