Force nftables usage when Talos Ingress Firewall is enabled

[ivanfed0t0v]

There's a potential issue where Kubernetes components, such as kubelet and kube-proxy, are using legacy iptables, while the Talos Ingress Firewall is utilizing nftables. That means both x_tables and nf_tables kernel subsystems get used at the same time, which could potentially lead to unpredictable behavior or conflicts.

I've been able to track the cause of this down to kubelet base image:

kubelet/Dockerfile

Line 31 in 981b856

FROM registry.k8s.io/build-image/debian-iptables:bookworm-v1.0.0 as container

On the first invocation of iptables in that image /usr/sbin/iptables-wrapper script gets run. That script decides between _legacy and _nft version of iptables, based on the length of the already created rules. If they are empty (which I think is the case when kubelet on Talos starts up) the script chooses legacy version:

$ docker run --rm -it registry.k8s.io/build-image/debian-iptables:bookworm-v1.0.0 iptables --version
iptables v1.8.9 (legacy)

To fix the issue please consider one of the following solutions:

1. Force iptables-nft at build time in the Dockerfile:

update-alternatives --set iptables "/usr/sbin/iptables-nft"
update-alternatives --set ip6tables "/usr/sbin/ip6tables-nft"

2. Use an image with updated version of /usr/sbin/iptables-wrapper which favors nftables. For example this one:

$ docker run --rm -it registry.k8s.io/build-image/distroless-iptables:v0.5.2 iptables --version
iptables v1.8.9 (nf_tables)

[smira]

Unfortunately kubelet images run across a variety of Talos versions, and we should be compatible with upstream kube-proxy image, so this is not that easy.

On Talos side, nftables should exist if firewall was configured before kubelet start. But we should definitely investigate this, as probably preferring nftables is definitely better, even if we always create a dummy table/rule.

[ivanfed0t0v]

Upstream kube-proxy also uses that same /usr/sbin/iptables-wrapper script, so I think it should follow kubelet without issues. But yeah, I agree, that forcing nftables isn't the best way for compatibility.

As for the Talos side, maybe I can help with reproducing the issue:

1. Create single-node metal cluster with default parameters with talosctl gen config ..., apply-config ..., bootstrap ..., without any Ingress firewall rules
2. Add a simple firewall allow rule and set allowSchedulingOnControlPlanes: true in the controlplane.yaml and apply.

   version: v1alpha1
   cluster:
       allowSchedulingOnControlPlanes: true
   ---
   apiVersion: v1alpha1
   kind: NetworkDefaultActionConfig
   ingress: accept
   ---
   apiVersion: v1alpha1
   kind: NetworkRuleConfig
   name: no-ssh
   portSelector:
     ports:
       - 22
     protocol: tcp
   ingress:
     - subnet: 192.168.0.0/16

3. Run privileged alpine pod with iptables, nftables and iptables-legacy packages installed and list the rules:
   • only Talos Ingress Firewall rules are present

   # nft list ruleset
   table inet talos {
       chain ingress {
           type filter hook input priority filter; policy accept;
           iifname { "", "", "" } counter packets 368 bytes 62123 accept
           ip saddr != { 192.168.0.0/16 } tcp dport { 22 } counter packets 0 bytes 0 drop
           meta nfproto ipv6 tcp dport { 22 } counter packets 0 bytes 0 drop
       }
   }

   • No iptables-nft rules

   # iptables-nft -L
   # Warning: iptables-legacy tables present, use iptables-legacy to see them
   Chain INPUT (policy ACCEPT)
   target     prot opt source               destination         
   
   Chain FORWARD (policy ACCEPT)
   target     prot opt source               destination         
   
   Chain OUTPUT (policy ACCEPT)
   target     prot opt source               destination         

   • kube-proxy uses iptables-legacy

   # iptables-legacy -L
   Chain INPUT (policy ACCEPT)
   target     prot opt source               destination         
   KUBE-PROXY-FIREWALL  all  --  anywhere             anywhere             ctstate NEW /* kubernetes load balancer firewall */
   ...<all the kube-proxy rules>...

Reboot does not change things either, seems like kubelet rules go before firewall.

[smira]

Yep, I think iptables-nft looks for a specific rule that it recognizes, but Talos can inject one well before kubelet boots up - I will look into that.