kube-router does not work with iptables 1.8.8 (nf_tables) on host

[ncopa]

What happened?

Running kubelet on a host with iptables-1.8.8 (nf_tables mode) does not work due to kube-proxy image uses iptables-1.8.7. kube-proxy ends up replace the rule

-A KUBE-FIREWALL -m comment --comment "kubernetes firewall for dropping marked packets" -m mark --mark 0x8000/0x8000 -j DROP

with

-A KUBE-FIREWALL -m comment --comment "kubernetes firewall for dropping marked packets" -j DROP

This leads to network stop working.

What did you expect to happen?

Network continue to work regardless of version of iptables installed on the host.

How can we reproduce it (as minimally and precisely as possible)?

Try to join a worker with iptables 1.8.8 in nf_tables mode on the host.

Anything else we need to know?

Problem is that iptables-save with iptables 1.8.7 does not work with iptables rules created with iptables 1.8.8 (nf_tables).

If I on the host manually (using iptables 1.8.8) do:

iptables-save | grep -E '(Generated by|mytest)'
# Generated by iptables-save v1.8.8 (nf_tables) on Thu Sep 15 14:34:46 2022
# Generated by iptables-save v1.8.8 (nf_tables) on Thu Sep 15 14:34:46 2022
-A KUBE-FIREWALL -m comment --comment mytest -m mark --mark 0x8000/0x8000 -j DROP
# Generated by iptables-save v1.8.8 (nf_tables) on Thu Sep 15 14:34:46 2022

It shows the -m --mark 0x8000/0x8000.

If I then use nsenter to the kube-proxy pod and do the same I get:

/ # nsenter -t $(pidof kube-proxy) -m iptables-save | grep -E '(Generated by|mytest)'
# Generated by iptables-save v1.8.7 on Thu Sep 15 14:36:38 2022
# Generated by iptables-save v1.8.7 on Thu Sep 15 14:36:38 2022
-A KUBE-FIREWALL -m comment --comment mytest -j DROP
# Generated by iptables-save v1.8.7 on Thu Sep 15 14:36:38 2022

As you see, the -m --mark 0x8000/0x8000 is lost and all packages are dropped, not only the marked ones.

Possible workarounds:

• use iptables in legacy mode on host
• downgrade iptables to 1.8.7 on the host

Possible fixes:

• upgrade iptables in kube-proxy image to 1.8.8 (and make sure that it always is latest).
• change logic in kube-proxy so that it does not touch/re-inject previously created iptables rules (eg does not iptables-save | ... | iptables-restore)

Kubernetes version

$ kubectl version

W0915 14:46:03.488791    2464 loader.go:223] Config not found: /var/lib/k0s/pki/admin.conf

WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short.  Use --output=yaml|json to get the full version.
Client Version: version.Info{Major:"1", Minor:"25", GitVersion:"v1.25.0", GitCommit:"a866cbe2e5bbaa01cfd5e969aa3e033f3282a8a2", GitTreeState:"clean", BuildDate:"2022-08-23T17:44:59Z", GoVersion:"go1.19", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v4.5.7
Unable to connect to the server: dial tcp 127.0.0.1:8080: i/o timeout

I think the `Unable to connect` message is due to firewall being broken...

Cloud provider

n/a

OS version

# On Linux:
$ cat /etc/os-release
NAME="Alpine Linux"
ID=alpine
VERSION_ID=3.16.2
PRETTY_NAME="Alpine Linux v3.16"
HOME_URL="https://alpinelinux.org/"
BUG_REPORT_URL="https://gitlab.alpinelinux.org/alpine/aports/-/issues"

$ uname -a
Linux worker0 5.15.67-0-lts #1-Alpine SMP Fri, 09 Sep 2022 06:15:47 +0000 x86_64 GNU/Linu

Install tools

k0s

Container runtime (CRI) and version (if applicable)

n/a

Related plugins (CNI, CSI, ...) and versions (if applicable)

n/a

[k8s-ci-robot]

@ncopa: This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[ncopa]

/sig network

[ncopa]

The rule that gets mangled by older iptables is created here:

kubernetes/pkg/kubelet/kubelet_network_linux.go

Line 152 in bddb4ec

if _, err := iptClient.EnsureRule(utiliptables.Append, utiliptables.TableFilter, KubeFirewallChain,

[bridgetkromhout]

/assign @danwinship

[uablrek]

change logic in kube-proxy so that it does not touch/re-inject previously created iptables rules (eg does not iptables-save | ... | iptables-restore)

Iptables rules must not be added one-by one. Reason is that the update is; read-everything-to-user-space, update-one-rule, write-everything-back-to-kernel. This was one reason to invent nft, so perhaps this is not true for nf_tables mode, but please check it.

[danwinship]

kube-proxy re-creates this rule on purpose so it doesn't matter whether it's doing it one-by-one or via iptables-restore.

[danwinship]

This seems to be a bug in iptables and I don't think we can plausibly work around it. (Changing the version of iptables in the kube-proxy image would just introduce the bug in the opposite scenario, where kube-proxy has the newer version and kubelet has the older version.) The answer for now seems to be "don't use iptables 1.8.8, it's broken".

[danwinship]

Filed https://bugzilla.netfilter.org/show_bug.cgi?id=1632