DNS Cluster IP determination inconsistent with EKS AL2 AMIs

[backjo]

Image I'm using:
Latest BottleRocket EKS-optimized image (ami-01d975726ace73c6d in us-west-2)

What I expected to happen:
The DNS IP that BottleRocket determines should be consistent with the IP address that the EKS-optimized AL2 determines.

What actually happened:
The DNS IP is not consistent. The logic in the AL2 AMI here -> awslabs/amazon-eks-ami@8864821 resolves:

DNS_CLUSTER_IP=${SERVICE_IPV4_CIDR%.*}.10

The BottleRocket image resolves:

    const DEFAULT_10_RANGE_DNS_CLUSTER_IP: &str = "172.20.0.10";
    let dns = if mac_cidr_blocks.starts_with("10.") {
        DEFAULT_10_RANGE_DNS_CLUSTER_IP
    } else {
        DEFAULT_DNS_CLUSTER_IP
    } 

How to reproduce the problem:
Run EKS-optimized AMIs on AL2 and BottleRocket with default settings on the same EKS cluster.

[webern]

Also linking this commit awslabs/amazon-eks-ami@4353bbf1e

[webern]

I have started looking at this. It seems the big difference is here:

https://github.com/awslabs/amazon-eks-ami/blob/732b6b2f4bee5b92297c416de1f8cba30a70b364/files/bootstrap.sh#L301..L309

The aws eks CLI is being used to describe the cluster and obtain the CIDR. In the event that the CIDR is not-customized, then Bottlerocket's fallback logic appears to be the same.

To replicate this, we would need to use rusoto in pluto to get this information from the EKS API. The node would also need an IAM role that permits this. These changes are more substantial than I imagined when first reading the issue, but they do open up some additional possibilities. For example, we might populate the cluster certificate and kubernetes endpoint using only the cluster name, which simplifies the userdata requirements when launching nodes.

[webern]

Sadly, I have to mark this as blocked for the moment. The field we need to replicate the EKS AL2 script is cluster.kubernetesNetworkConfig.serviceIpv4Cidr.

This field does not exist in rusoto v0.45 https://github.com/rusoto/rusoto/blob/rusoto-v0.45.0/rusoto/services/eks/src/generated.rs#L51

It was added in this commit:
rusoto/rusoto@4ca2e90

And released in v0.46
https://github.com/rusoto/rusoto/blob/rusoto-v0.46.0/rusoto/services/eks/src/generated.rs#L193

The problem with using v0.46 is that it updated to tokio v1:
https://github.com/rusoto/rusoto/blob/rusoto-v0.46.0/rusoto/services/eks/Cargo.toml#L35

And so... this issue is essentially blocked by #1269. I'll turn back to #1269 and try harder!

[webern]

This is now unblocked and I will open a PR soon.