Switching AWS_VPC_CNI_NODE_PORT_SUPPORT to false breaks SNAT for Pods

[antoninbas]

What happened:
When setting AWS_VPC_CNI_NODE_PORT_SUPPORT to false for aws-node, SNAT breaks for all Pods for which the IP is assigned from a secondary network interface / ENI.

How to reproduce it (as minimally and precisely as possible):

1. Create a default EKS cluster with a single Node, and use a "small-ish" instance type (so that the primary ENI can support a smaller number of addresses). For example: eksctl create cluster -N 1 --ssh-access --node-type t3.medium.
2. Edit the aws-node Deployment to set AWS_VPC_CNI_NODE_PORT_SUPPORT to false. AWS_VPC_K8S_CNI_EXTERNALSNAT should remain set to false, so that the CNI is responsible for SNATting Pod traffic.
3. Create a Deployment with 5 replicas. There is a single Node so all 5 Pods will run on that Node. For example, kubectl apply the following YAML:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: toolbox
spec:
  replicas: 5
  selector:
    matchLabels:
      app: toolbox
  template:
    metadata:
      labels:
        app: toolbox
    spec:
      containers:
      - name: toolbox
        image: antoninbas/toolbox
        command: ["sleep", "3600"]

4. check which Pods are assigned to a secondary ENI. I do this by running ip rule list and matching the rules for route table to to Pod IPs.
5. exec into one of these Pods and ping 8.8.8.8. It will fail.

Anything else we need to know?:

This is caused by some incorrect iptables rules:

• amazon-vpc-cni-k8s/pkg/networkutils/network.go

  Lines 628 to 637 in f979630

  	iptableRules = append(iptableRules, iptablesRule{
  	name: "connmark to fwmark copy",
  	shouldExist: !n.useExternalSNAT,
  	table: "nat",
  	chain: "PREROUTING",
  	rule: []string{
  	"-m", "comment", "--comment", "AWS, CONNMARK", "-j", "CONNMARK",
  	"--restore-mark", "--mask", fmt.Sprintf("%#x", n.mainENIMark),
  	},
  	})

  is in the nat table and hence only applies to the first packet of the connection

• amazon-vpc-cni-k8s/pkg/networkutils/network.go

  Lines 529 to 538 in f979630

  	iptableRules = append(iptableRules, iptablesRule{
  	name: "connmark restore for primary ENI",
  	shouldExist: n.nodePortSupportEnabled,
  	table: "mangle",
  	chain: "PREROUTING",
  	rule: []string{
  	"-m", "comment", "--comment", "AWS, primary ENI",
  	"-i", n.vethPrefix + "+", "-j", "CONNMARK", "--restore-mark", "--mask", fmt.Sprintf("%#x", n.mainENIMark),
  	},
  	})

  should be installed if AWS_VPC_CNI_NODE_PORT_SUPPORT is true OR if AWS_VPC_K8S_CNI_EXTERNALSNAT false (default).

Additionally, mainENIRule IP rule also needs to be installed, and loose rpf_filter needs to be configured.

I will submit a PR with these changes.

Environment:
eksctl version

0.105.0-rc.0

K8s version

Server Version: version.Info{Major:"1", Minor:"22+", GitVersion:"v1.22.10-eks-84b4fe6", GitCommit:"cc6a1b4915a99f49f5510ef0667f94b9ca832a8a", GitTreeState:"clean", BuildDate:"2022-06-09T18:24:04Z", GoVersion:"go1.16.15", Compiler:"gc", Platform:"linux/amd64"}

credit also goes to my colleague @tnqn for investigating this

[jayanthvn]

Hi,

Thanks for opening the PR for this issue. I assume you are on 1.10.x since k8s version is 1.22 right? Have you tried a similar test on 1.7 branch?

[antoninbas]

@jayanthvn I am using v1.10.1-eksbuild.1. I can test with 1.7. Do you expect a different result?

BTW, I have updated my PR with a unit test (also fixed test failures for existing unit test cases which were introduced by my change).

[antoninbas]

@jayanthvn I just finished testing with v1.7.10. I can confirm that the problem does not exist with the 1.7 version.

The IP rules are different for that older version:

0:	from all lookup local
512:	from all to 192.168.84.247 lookup main
512:	from all to 192.168.83.153 lookup main
512:	from all to 192.168.89.109 lookup main
512:	from all to 192.168.92.230 lookup main
512:	from all to 192.168.72.42 lookup main
512:	from all to 192.168.88.212 lookup main
512:	from all to 192.168.83.69 lookup main
1536:	from 192.168.84.247 to 192.168.0.0/16 lookup 3
1536:	from 192.168.83.153 to 192.168.0.0/16 lookup 3
1536:	from 192.168.89.109 to 192.168.0.0/16 lookup 2
1536:	from 192.168.92.230 to 192.168.0.0/16 lookup 3
1536:	from 192.168.72.42 to 192.168.0.0/16 lookup 3
1536:	from 192.168.88.212 to 192.168.0.0/16 lookup 3
32766:	from all lookup main
32767:	from all lookup default

The rules which select a secondary route table (for Pods assigned to secondary ENIs) have an extra to clause compared to 1.10. For example:

1536:	from 192.168.84.247 to 192.168.0.0/16 lookup 3

Which means that SNATed traffic still hits the main route table.

With 1.10, the rules are as follows:

0:	from all lookup local
512:	from all to 192.168.29.56 lookup main
512:	from all to 192.168.24.134 lookup main
512:	from all to 192.168.31.135 lookup main
512:	from all to 192.168.31.223 lookup main
512:	from all to 192.168.29.27 lookup main
512:	from all to 192.168.16.158 lookup main
512:	from all to 192.168.2.135 lookup main
1024:	from all fwmark 0x80/0x80 lookup main
1536:	from 192.168.31.223 lookup 2
1536:	from 192.168.29.27 lookup 2
1536:	from 192.168.16.158 lookup 2
1536:	from 192.168.2.135 lookup 2
32766:	from all lookup main
32767:	from all lookup default

So SNATed traffic needs to have the 0x80 mark and the bypass rule (1024: from all fwmark 0x80/0x80 lookup main) is required. Otherwise it goes to secondary route tables, and it egresses through a secondary ENI (but with the source IP set to the primary IP).

[github-actions]

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 14 days

[antoninbas]

/remove stale

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.