AWS_VPC_K8S_CNI_RANDOMIZESNAT prng not working in amazon-k8s-cni:v1.4.1 (and 1.5+)

[jraby]

The iptables version shipped in the amazon-k8s-cni image does not support the --fully-random SNAT option. This option was introduced in iptables 1.6.something whereas v1.24.x is used in the amazon-k8s-cni image. (this seems to be the version from the yum repos from amazonlinux)

When setting AWS_VPC_K8S_CNI_RANDOMIZESNAT to prng using this image, it falls back to the hash random method.

Is it expected that fully-random is not working out of the box on the official image? And is there a way to get the --fully-random behavior using the official images?

Thanks.

(edit: clarified question)

[jaypipes]

Hi @jraby, yes this is the expected behaviour, AFAICT from the code:

amazon-vpc-cni-k8s/pkg/networkutils/network.go

Lines 349 to 357 in 3aacadc

	if n.typeOfSNAT == randomPRNGSNAT {
	if ipt.HasRandomFully() {
	snatRule = append(snatRule, "--random-fully")
	} else {
	log.Warn("prng (--random-fully) requested, but iptables version does not support it. " +
	"Falling back to hashrandom (--random)")
	snatRule = append(snatRule, "--random")
	}
	}

You should see a warning logged about falling back to "random" instead of "random-fully".

As for when the AL2 iptables package will be updated to a more modern version, I don't know the answer to that. Perhaps @stewartsmith would be able to answer that.

[jraby]

In the end we rolled our own image with the following pseudo diff:

 FROM amazonlinux:2
 RUN yum update -y && \
     yum install -y iproute && \
-    yum install -y iptables && \
+    yum install -y tar wget bzip2 gcc gcc-c++ make && \
     yum clean all

+# iptables 1.8.3 without nftable support (aka iptables-legacy to match what kube-proxy uses)
+RUN cd /tmp && wget http://www.netfilter.org/projects/iptables/files/iptables-1.8.3.tar.bz2 && \
+    tar xvf iptables-1.8.3.tar.bz2 && cd iptables-1.8.3 && \
+    ./configure --disable-nftables  --enable-static --disable-shared --prefix=/usr/local && \
+    make && make install
[...]

One needs to be aware that iptables >=1.8.0 defaults to nftables (so does iptables shipped with centos:8), which will break if used on a node where kube-proxy also injects rules with an older version of iptables.
The above builds iptables without that nftable support to avoid any surprise / conflict.

Fun times.

[nithu0115]

#509

[mogren]

We need awslabs/amazon-eks-ami#380 done to be able to resolve this issue.

[mogren]

Fixed in v1.5.7

[terrych0u]

I also got the same issue on the newest eks 1.16 with CNI 1.6.1 and my kube-proxy 1.16.8 keeps show log :

Not using `--random-fully` in the MASQUERADE rule for iptables because the local version of iptables does not support it

is there any way to fix it?

[lapidus79]

@mogren the same issue shows up in eks 1.16 with cni 1.6.2 and kube-proxy v1.16.8

[infa-ddeore]

@mogren the same issue shows up in eks 1.16 with cni 1.6.2 and kube-proxy v1.16.8

--random-fully support is added in 1.7.0 aws-cni so using this and above version should solve the problem