calico node says running but shows 0/1. Errors come up in logs and

[wahahawasabi]

I installed calico v3.24.5 manifest onto my Asus TInkerboard kernel 4.19. However, even though all status shows running, there are errors under the hood that pops up.

I have config_bpf, config_ipset enabled in the kernel. Not sure if i'm missing anything else?

I have also tried with both NetworkManager enabled and disabled. With NetworkManager enabled, i've added this to the conf file:
[keyfile]
unmanaged-devices=interface-name:cali*;interface-name:tunl*;interface-name:vxlan.calico;interface-name:vxlan-v6.calico;interface-name:wireguard.cali;interface-name:wg-v6.cali

i am also currently using the below config:
sudo update-alternatives --set iptables /usr/sbin/iptables-legacy
sudo update-alternatives --set ip6tables /usr/sbin/ip6tables-legacy
sudo update-alternatives --set arptables /usr/sbin/arptables-legacy
sudo update-alternatives --set ebtables /usr/sbin/ebtables-legacy

Expected Behavior

• All calico services should be up and running, nodes are active and well.
• Logs are not spammed with errors and warning messages

Current Behavior

• calico node is running but shows 0/1.

• error message from `k logs calico-node | grep ERROR
  2022-11-27 18:30:04.529 [ERROR][16255] felix/route_rule.go 248: Failed to list routing rules, retrying... error=operation not supported ipVersion=4
  2022-11-27 18:30:06.628 [ERROR][16255] felix/table.go 1006: Failed to program iptables, loading diags before panic. error=exit status 2 ipVersion=0x4 table="filter"
  2022-11-27 18:30:06.637 [ERROR][16255] felix/table.go 1012: Current state of iptables ipVersion=0x4 iptablesState="# Generated by iptables-save v1.8.4 on Sun Nov 27 18:30:06 2022\nfilter\n:INPUT ACCEPT [1073425:495019571]\n:FORWARD ACCEPT [4716:213520]\n:OUTPUT ACCEPT [1072410:500267386]\n:KUBE-EXTERNAL-SERVICES - [0:0]\n:KUBE-FIREWALL - [0:0]\n:KUBE-FORWARD - [0:0]\n:KUBE-KUBELET-CANARY - [0:0]\n:KUBE-NODEPORTS - [0:0]\n:KUBE-PROXY-CANARY - [0:0]\n:KUBE-PROXY-FIREWALL - [0:0]\n:KUBE-SERVICES - [0:0]\n-A INPUT -m conntrack --ctstate NEW -m comment --comment "kubernetes load balancer firewall" -j KUBE-PROXY-FIREWALL\n-A INPUT -m comment --comment "kubernetes health check service ports" -j KUBE-NODEPORTS\n-A INPUT -m conntrack --ctstate NEW -m comment --comment "kubernetes externally-visible service portals" -j KUBE-EXTERNAL-SERVICES\n-A INPUT -j KUBE-FIREWALL\n-A FORWARD -m conntrack --ctstate NEW -m comment --comment "kubernetes load balancer firewall" -j KUBE-PROXY-FIREWALL\n-A FORWARD -m comment --comment "kubernetes forwarding rules" -j KUBE-FORWARD\n-A FORWARD -m conntrack --ctstate NEW -m comment --comment "kubernetes service portals" -j KUBE-SERVICES\n-A FORWARD -m conntrack --ctstate NEW -m comment --comment "kubernetes externally-visible service portals" -j KUBE-EXTERNAL-SERVICES\n-A OUTPUT -m conntrack --ctstate NEW -m comment --comment "kubernetes load balancer firewall" -j KUBE-PROXY-FIREWALL\n-A OUTPUT -m conntrack --ctstate NEW -m comment --comment "kubernetes service portals" -j KUBE-SERVICES\n-A OUTPUT -j KUBE-FIREWALL\n-A KUBE-FIREWALL ! -s 127.0.0.0/8 -d 127.0.0.0/8 -m comment --comment "block incoming localnet connections" -m conntrack ! --ctstate RELATED,ESTABLISHED,DNAT -j DROP\n-A KUBE-FIREWALL -m comment --comment "kubernetes firewall for dropping marked packets" -m mark --mark 0x8000/0x8000 -j DROP\n-A KUBE-FORWARD -m conntrack --ctstate INVALID -j DROP\n-A KUBE-FORWARD -m comment --comment "kubernetes forwarding rules" -m mark --mark 0x4000/0x4000 -j ACCEPT\n-A KUBE-FORWARD -m comment --comment "kubernetes forwarding conntrack rule" -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT\nCOMMIT\n# Completed on Sun Nov 27 18:30:06 2022\n" table="filter"
  2022-11-27 18:30:07.495 [ERROR][16383] felix/route_rule.go 248: Failed to list routing rules, retrying... error=operation not supported ipVersion=4
  2022-11-27 18:30:09.636 [ERROR][16383] felix/table.go 1006: Failed to program iptables, loading diags before panic. error=exit status 2 ipVersion=0x4 table="nat"
  2022-11-27 18:30:09.653 [ERROR][16383] felix/table.go 1012: Current state of iptables ipVersion=0x4 iptablesState="# Generated by iptables-save v1.8.4 on Sun Nov 27 18:30:09 2022\nnat\n:PREROUTING ACCEPT [4801:231792]\n:INPUT ACCEPT [87:18392]\n:OUTPUT ACCEPT [9384:565741]\n:POSTROUTING ACCEPT [14100:779261]\n:KUBE-KUBELET-CANARY - [0:0]\n:KUBE-MARK-DROP - [0:0]\n:KUBE-MARK-MASQ - [0:0]\n:KUBE-NODEPORTS - [0:0]\n:KUBE-POSTROUTING - [0:0]\n:KUBE-PROXY-CANARY - [0:0]\n:KUBE-SEP-IIBJKWFS4IXQF6SU - [0:0]\n:KUBE-SEP-KDHAQ7ISOL44PEIX - [0:0]\n:KUBE-SEP-KXZHBPU2JS5EYLHU - [0:0]\n:KUBE-SEP-M5QFEHOX6AZW2NXU - [0:0]\n:KUBE-SEP-PAMQENDCPQS4R2GF - [0:0]\n:KUBE-SEP-ROGUWICNIMTG5CZ6 - [0:0]\n:KUBE-SEP-TVFZIBJ3LXIVM424 - [0:0]\n:KUBE-SERVICES - [0:0]\n:KUBE-SVC-ERIFXISQEP7F7OF4 - [0:0]\n:KUBE-SVC-JD5MR3NA4I4DYORP - [0:0]\n:KUBE-SVC-NPX46M4PTMTKRN6Y - [0:0]\n:KUBE-SVC-TCOU7JCQXEZGVUNU - [0:0]\n-A PREROUTING -m comment --comment "kubernetes service portals" -j KUBE-SERVICES\n-A OUTPUT -m comment --comment "kubernetes service portals" -j KUBE-SERVICES\n-A POSTROUTING -m comment --comment "kubernetes postrouting rules" -j KUBE-POSTROUTING\n-A KUBE-MARK-DROP -j MARK --set-xmark 0x8000/0x8000\n-A KUBE-MARK-MASQ -j MARK --set-xmark 0x4000/0x4000\n-A KUBE-POSTROUTING -m mark ! --mark 0x4000/0x4000 -j RETURN\n-A KUBE-POSTROUTING -j MARK --set-xmark 0x4000/0x0\n-A KUBE-POSTROUTING -m comment --comment "kubernetes service traffic requiring SNAT" -j MASQUERADE --random-fully\n-A KUBE-SEP-IIBJKWFS4IXQF6SU -s 192.168.110.67/32 -m comment --comment "kube-system/kube-dns:metrics" -j KUBE-MARK-MASQ\n-A KUBE-SEP-IIBJKWFS4IXQF6SU -p tcp -m comment --comment "kube-system/kube-dns:metrics" -m tcp -j DNAT --to-destination 192.168.110.67:9153\n-A KUBE-SEP-KDHAQ7ISOL44PEIX -s 192.168.110.67/32 -m comment --comment "kube-system/kube-dns:dns-tcp" -j KUBE-MARK-MASQ\n-A KUBE-SEP-KDHAQ7ISOL44PEIX -p tcp -m comment --comment "kube-system/kube-dns:dns-tcp" -m tcp -j DNAT --to-destination 192.168.110.67:53\n-A KUBE-SEP-KXZHBPU2JS5EYLHU -s 192.168.110.66/32 -m comment --comment "kube-system/kube-dns:metrics" -j KUBE-MARK-MASQ\n-A KUBE-SEP-KXZHBPU2JS5EYLHU -p tcp -m comment --comment "kube-system/kube-dns:metrics" -m tcp -j DNAT --to-destination 192.168.110.66:9153\n-A KUBE-SEP-M5QFEHOX6AZW2NXU -s 192.168.110.67/32 -m comment --comment "kube-system/kube-dns:dns" -j KUBE-MARK-MASQ\n-A KUBE-SEP-M5QFEHOX6AZW2NXU -p udp -m comment --comment "kube-system/kube-dns:dns" -m udp -j DNAT --to-destination 192.168.110.67:53\n-A KUBE-SEP-PAMQENDCPQS4R2GF -s 192.168.110.66/32 -m comment --comment "kube-system/kube-dns:dns-tcp" -j KUBE-MARK-MASQ\n-A KUBE-SEP-PAMQENDCPQS4R2GF -p tcp -m comment --comment "kube-system/kube-dns:dns-tcp" -m tcp -j DNAT --to-destination 192.168.110.66:53\n-A KUBE-SEP-ROGUWICNIMTG5CZ6 -s 192.168.50.146/32 -m comment --comment "default/kubernetes:https" -j KUBE-MARK-MASQ\n-A KUBE-SEP-ROGUWICNIMTG5CZ6 -p tcp -m comment --comment "default/kubernetes:https" -m tcp -j DNAT --to-destination 192.168.50.146:6443\n-A KUBE-SEP-TVFZIBJ3LXIVM424 -s 192.168.110.66/32 -m comment --comment "kube-system/kube-dns:dns" -j KUBE-MARK-MASQ\n-A KUBE-SEP-TVFZIBJ3LXIVM424 -p udp -m comment --comment "kube-system/kube-dns:dns" -m udp -j DNAT --to-destination 192.168.110.66:53\n-A KUBE-SERVICES -d 10.96.0.1/32 -p tcp -m comment --comment "default/kubernetes:https cluster IP" -m tcp --dport 443 -j KUBE-SVC-NPX46M4PTMTKRN6Y\n-A KUBE-SERVICES -d 10.96.0.10/32 -p tcp -m comment --comment "kube-system/kube-dns:dns-tcp cluster IP" -m tcp --dport 53 -j KUBE-SVC-ERIFXISQEP7F7OF4\n-A KUBE-SERVICES -d 10.96.0.10/32 -p tcp -m comment --comment "kube-system/kube-dns:metrics cluster IP" -m tcp --dport 9153 -j KUBE-SVC-JD5MR3NA4I4DYORP\n-A KUBE-SERVICES -d 10.96.0.10/32 -p udp -m comment --comment "kube-system/kube-dns:dns cluster IP" -m udp --dport 53 -j KUBE-SVC-TCOU7JCQXEZGVUNU\n-A KUBE-SERVICES -m comment --comment "kubernetes service nodeports; NOTE: this must be the last rule in this chain" -m addrtype --dst-type LOCAL -j KUBE-NODEPORTS\n-A KUBE-SVC-ERIFXISQEP7F7OF4 ! -s 192.168.0.0/16 -d 10.96.0.10/32 -p tcp -m comment --comment "kube-system/kube-dns:dns-tcp cluster IP" -m tcp --dport 53 -j KUBE-MARK-MASQ\n-A KUBE-SVC-ERIFXISQEP7F7OF4 -m comment --comment "kube-system/kube-dns:dns-tcp -> 192.168.110.66:53" -m statistic --mode random --probability 0.50000000000 -j KUBE-SEP-PAMQENDCPQS4R2GF\n-A KUBE-SVC-ERIFXISQEP7F7OF4 -m comment --comment "kube-system/kube-dns:dns-tcp -> 192.168.110.67:53" -j KUBE-SEP-KDHAQ7ISOL44PEIX\n-A KUBE-SVC-JD5MR3NA4I4DYORP ! -s 192.168.0.0/16 -d 10.96.0.10/32 -p tcp -m comment --comment "kube-system/kube-dns:metrics cluster IP" -m tcp --dport 9153 -j KUBE-MARK-MASQ\n-A KUBE-SVC-JD5MR3NA4I4DYORP -m comment --comment "kube-system/kube-dns:metrics -> 192.168.110.66:9153" -m statistic --mode random --probability 0.50000000000 -j KUBE-SEP-KXZHBPU2JS5EYLHU\n-A KUBE-SVC-JD5MR3NA4I4DYORP -m comment --comment "kube-system/kube-dns:metrics -> 192.168.110.67:9153" -j KUBE-SEP-IIBJKWFS4IXQF6SU\n-A KUBE-SVC-NPX46M4PTMTKRN6Y ! -s 192.168.0.0/16 -d 10.96.0.1/32 -p tcp -m comment --comment "default/kubernetes:https cluster IP" -m tcp --dport 443 -j KUBE-MARK-MASQ\n-A KUBE-SVC-NPX46M4PTMTKRN6Y -m comment --comment "default/kubernetes:https -> 192.168.50.146:6443" -j KUBE-SEP-ROGUWICNIMTG5CZ6\n-A KUBE-SVC-TCOU7JCQXEZGVUNU ! -s 192.168.0.0/16 -d 10.96.0.10/32 -p udp -m comment --comment "kube-system/kube-dns:dns cluster IP" -m udp --dport 53 -j KUBE-MARK-MASQ\n-A KUBE-SVC-TCOU7JCQXEZGVUNU -m comment --comment "kube-system/kube-dns:dns -> 192.168.110.66:53" -m statistic --mode random --probability 0.50000000000 -j KUBE-SEP-TVFZIBJ3LXIVM424\n-A KUBE-SVC-TCOU7JCQXEZGVUNU -m comment --comment "kube-system/kube-dns:dns -> 192.168.110.67:53" -j KUBE-SEP-M5QFEHOX6AZW2NXU\nCOMMIT\n# Completed on Sun Nov 27 18:30:09 2022\n" table="nat"

• error message from k logs calico-node | grep WARNING 2022-11-27 18:24:43.542 [WARNING][1065] felix/table.go 999: Failed to program iptables, will retry error=exit status 2 ipVersion=0x4 table="filter" 2022-11-27 18:24:43.783 [WARNING][1065] felix/ipip_mgr.go 111: Failed to add IPIP tunnel device error=exit status 1 2022-11-27 18:24:43.784 [WARNING][1065] felix/ipip_mgr.go 88: Failed configure IPIP tunnel device, retrying... error=exit status 1 2022-11-27 18:24:43.986 [WARNING][1065] felix/table.go 1002: Retrying... error=exit status 2 ipVersion=0x4 table="nat" 2022-11-27 18:24:44.055 [WARNING][1065] felix/table.go 1002: Retrying... error=exit status 2 ipVersion=0x4 table="filter" 2022-11-27 18:24:44.086 [WARNING][1065] felix/table.go 1321: Failed to execute ip(6)tables-restore command error=exit status 2 errorOutput="iptables-restore v1.8.4 (legacy): Couldn't load match set':No such file or directory\n\nError occurred at line: 8\nTry `iptables-restore -h' or 'iptables-restore --help' for more information.\n" input="*nat\n:cali-fip-dnat - -\n:cali-fip-snat - -\n:cali-nat-outgoing - -\n:cali-PREROUTING - -\n:cali-POSTROUTING - -\n:cali-OUTPUT - -\n-A cali-nat-outgoing -m comment --comment "cali:flqWnvo8yq4ULQLa" -m set --match-set cali40masq-ipam-pools src -m set ! --match-set cali40all-ipam-pools dst --jump MASQUERADE --random-fully\n-A cali-PREROUTING -m comment --comment "cali:r6XmIziWUJsdOK6Z" --jump cali-fip-dnat\n-A cali-POSTROUTING -m comment --comment "cali:Z-c7XtVd2Bq7s_hA" --jump cali-fip-snat\n-A cali-POSTROUTING -m comment --comment "cali:nYKhEzDlr11Jccal" --jump cali-nat-outgoing\n-A cali-POSTROUTING -m comment --comment "cali:SXWvdsbh4Mw7wOln" --out-interface tunl0 -m addrtype ! --src-type LOCAL --limit-iface-out -m addrtype --src-type LOCAL --jump MASQUERADE --random-fully\n-A cali-OUTPUT -m comment --comment "cali:GBTAv2p5CwevEyJm" --jump cali-fip-dnat\n-I PREROUTING -m comment --comment "cali:6gwbT8clXdHdC1b1" --jump cali-PREROUTING\n-I OUTPUT -m comment --comment "cali:tVnHkvAo15HuiPy0" --jump cali-OUTPUT\n-I POSTROUTING -m comment --comment "cali:O3lYWMrLQYEMJtB5" --jump cali-POSTROUTING\nCOMMIT\n" ipVersion=0x4 output="" table="nat"

Possible Solution

• i have googled and found that I should be adding the below in as well:
  vim calico.yaml
  • name: IP_AUTODETECTION_METHOD
    value: "interface=eth0"

Steps to Reproduce (for bugs)

1. sudo kubeadm init --pod-network-cidr=192.168.0.0/16
2. sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
3. sudo chown $(id -u):$(id -g) $HOME/.kube/config
4. curl https://raw.githubusercontent.com/projectcalico/calico/v3.24.5/manifests/calico.yaml -O
5. kubectl apply -f calico.yaml

Context

I am trying to set up calico to use it.

Your Environment

• Calico version - v3.24.5
• Orchestrator version (e.g. kubernetes, mesos, rkt): kubernetes: v1.25.4 (kubectl, kubeadm, kubelet)
• Operating System and version: ubuntu Jammy, kernel 4.19.25,
• Link to your project (optional):

[lwr20]

The errors in your report look like the kernel rejecting Felix's attempts to list routes and iptables rules.
Does your node meet these kernel requirements?
https://projectcalico.docs.tigera.io/getting-started/kubernetes/requirements#kernel-dependencies

[wahahawasabi]

Thanks for getting back to me on this. It is what I suspect as well. since the kernel is custom build. But I'm new to kernel development (although i know how to add new configs) - i dont quite understand the requirements for kernel dependencies. Is there some resource that explains what set, rpfilter, addrtype, comment, conntrack, icmp, tcp, udp, ipvs, icmpv6 (if IPv6 is enabled in your kernel), mark, multiport, rpfilter, sctp, ipvs refers to in the kernel config level?

[wahahawasabi]

I managed to solve it by going back into the kernel and selecting everything related to ipv4 and ipv6. it works now. so it was a kernel issue that occurred.