Iptables not updating correctly for HostPort when using CNI chaining Portmap in Cilium

[carloscastrojumo]

As suggested by Cilium maintainers, opening the issue here for Portmap. Original issue: cilium/cilium#18227

Using Cilium as a AWS VPC CNI replacement along with Portmap CNI chaining for HostPort pods:

eni:
  enabled: true

ipam:
  mode: eni

cni:
  configMap: cilium-cni-configuration
  chainingMode: portmap

apiVersion: v1
kind: ConfigMap
metadata:
  name: cilium-cni-configuration
  namespace: cilium-system
data:
  cni-config: |
    {
      "cniVersion":"0.3.1",
      "name":"cilium",
      "plugins": [
        {
          "cniVersion":"0.3.1",
          "type": "cilium-cni",
          "enable-debug": false,
          "ipam": {
            "pre-allocate": 20
          },
          "eni": {
            "first-interface-index": 0
          }
        },
        {
          "type": "portmap",
          "capabilities": {"portMappings": true}
        }
      ]
    }

Situation:

When restarting a pod with a HostPort, in this case listening on port 8126, we noticed that sometimes, the old iptables rule is not deleted, leading to service being unavailable. We ended up with a duplication of iptables rules, pointing to the same port on the old and new pods IP.

$ iptables -tnat -L | grep 8126
CNI-HOSTPORT-SETMARK  tcp  --  ip-10-210-135-109.eu-west-1.compute.internal  anywhere             tcp dpt:8126
CNI-HOSTPORT-SETMARK  tcp  --  localhost            anywhere             tcp dpt:8126
DNAT       tcp  --  anywhere             anywhere             tcp dpt:8126 to:10.210.135.109:8126
CNI-HOSTPORT-SETMARK  tcp  --  ip-10-210-148-146.eu-west-1.compute.internal  anywhere             tcp dpt:8126
CNI-HOSTPORT-SETMARK  tcp  --  localhost            anywhere             tcp dpt:8126
DNAT       tcp  --  anywhere             anywhere             tcp dpt:8126 to:10.210.148.146:8126
CNI-DN-d47c420e88d0edf95894e  tcp  --  anywhere             anywhere             /* dnat name: "portmap" id: "fea6e792eedef96ddc9fd0e46bfe71e85ee0a8c68d47753c1c3591a38e030ee3" */ multiport dports 8126
CNI-DN-23f180f03670ec8e0fb47  tcp  --  anywhere             anywhere             /* dnat name: "cilium" id: "b2d2c6302fac04975de3cf1b56e75b070ba6e75c1a6f5976b37f5a80d06b9b1b" */ multiport dports 8126

Here, our example is a daemonset that we restarted. The previous pod was using IP 10.210.148.146 and the new pod has IP 10.210.135.109. After the restart, the old IP was picked up by an unrelated pod, which doesn't have hostPort on 8126.

$ kubectl get pods -owide -A | grep 10.210.135.109
datadog                   datadog-agent-t5lv7                                               4/4     Running            0          12m     10.210.135.109   ip-10-210-156-224.eu-west-1.compute.internal   <none>           <none>
$ kubectl get pods -owide -A | grep 10.210.148.146
data-engineering          de-superset-microservice-59f854ddf8-rzgdj                         1/1     Running            0          169m    10.210.148.146   ip-10-210-156-224.eu-west-1.compute.internal   <none>           <none>

This issue occurred for every pod in the restarted daemonset.

Here is another example of iptables -L on another host.

$ iptables -tnat -L --line-numbers
[...]
Chain CNI-DN-6cd85e8a20983778996cc (2 references)
num  target     prot opt source               destination
1    CNI-HOSTPORT-SETMARK  udp  --  ip-10-210-117-62.eu-west-1.compute.internal  anywhere             udp dpt:8125
2    CNI-HOSTPORT-SETMARK  udp  --  localhost            anywhere             udp dpt:8125
3    DNAT       udp  --  anywhere             anywhere             udp dpt:8125 to:10.210.117.62:8125
4    CNI-HOSTPORT-SETMARK  tcp  --  ip-10-210-117-62.eu-west-1.compute.internal  anywhere             tcp dpt:8126
5    CNI-HOSTPORT-SETMARK  tcp  --  localhost            anywhere             tcp dpt:8126
6    DNAT       tcp  --  anywhere             anywhere             tcp dpt:8126 to:10.210.117.62:8126

Chain CNI-DN-7666aaa327c0a203ca6a9 (2 references)
num  target     prot opt source               destination
1    CNI-HOSTPORT-SETMARK  udp  --  ip-10-210-108-129.eu-west-1.compute.internal  anywhere             udp dpt:8125
2    CNI-HOSTPORT-SETMARK  udp  --  localhost            anywhere             udp dpt:8125
3    DNAT       udp  --  anywhere             anywhere             udp dpt:8125 to:10.210.108.129:8125
4    CNI-HOSTPORT-SETMARK  tcp  --  ip-10-210-108-129.eu-west-1.compute.internal  anywhere             tcp dpt:8126
5    CNI-HOSTPORT-SETMARK  tcp  --  localhost            anywhere             tcp dpt:8126
6    DNAT       tcp  --  anywhere             anywhere             tcp dpt:8126 to:10.210.108.129:8126

Chain CNI-HOSTPORT-DNAT (2 references)
num  target     prot opt source               destination
1    CNI-DN-7666aaa327c0a203ca6a9  tcp  --  anywhere             anywhere             /* dnat name: "portmap" id: "bb8aa3f94d5a2b7c2bd02f5c3fe0261845813c1f4c04d1efba0c3a6cbadce026" */ multiport dports 8126
2    CNI-DN-7666aaa327c0a203ca6a9  udp  --  anywhere             anywhere             /* dnat name: "portmap" id: "bb8aa3f94d5a2b7c2bd02f5c3fe0261845813c1f4c04d1efba0c3a6cbadce026" */ multiport dports 8125
3    CNI-DN-6cd85e8a20983778996cc  tcp  --  anywhere             anywhere             /* dnat name: "cilium" id: "0a81533f150b72e2f06d4a8e1a3b2e900b8dbc93873de9a42eb07b24c39a92b6" */ multiport dports 8126
4    CNI-DN-6cd85e8a20983778996cc  udp  --  anywhere             anywhere             /* dnat name: "cilium" id: "0a81533f150b72e2f06d4a8e1a3b2e900b8dbc93873de9a42eb07b24c39a92b6" */ multiport dports 8125
[...]

What did we expect?

Upon pod restart, the old iptables rule should be deleted and a new one should be written with the new pod IP.

Quick fix

Flushing the chain CNI-HOSTPORT-DNAT on the host and restarting the pod with the hostPort fixed the issue.

Kernel Version

5.4.156-83.273.amzn2.x86_64

Kubernetes Version

v1.20.11 in AWS EKS

[notfromstatefarm]

+1 on this issue, experiencing the exact same thing on Google Kubernetes Engine + Dataplane V2 (cilium). When nodes with a DaemonSet using a hostPort are restarted, the iptables entries are duplicated and communication over the hostPort no longer works.

cc @parabolic

[mohag]

Still present with k8s 1.23 from RKE (using Canal)

(apparently there were not enough +1 comments posted so the bot closed this...)

[satyamab]

We may still be running into this issue. More details here. Is this still an open issue or have we potentially misconfigured something (or maybe running an older version of something in the critical path)?

Any guidance would be much appreciated!

[goyjy]

Whether this issue has been resolved. Can provide a solution? grateful