kubelet, pluto: support IPv6 #1710

etungsten · 2021-08-11T22:23:33Z

Issue number:
Closes #1696

Description of changes:

Author: Erikson Tung <etung@amazon.com>
Date:   Mon Sep 13 15:59:59 2021 -0700

    kubernetes-1.21: patch kubelet to support ipv6 with AWS
    
    This adds a patch to the in-tree AWS cloud provider code to make kubelet
    append IPv6 addresses to the list of NodeAddresses.

Author: Erikson Tung <etung@amazon.com>
Date:   Mon Aug 9 12:27:23 2021 -0700

    pluto: support IPv6
    
    This adds support for retrieving IPv6 node IP address for IPv6 EKS
    clusters

Author: Erikson Tung <etung@amazon.com>
Date:   Fri Aug 13 11:00:47 2021 -0700

    release: enable IPv6 forwarding

The testing below has been done with IPv6 changes in aws-vpc-cni that has yet to be formally released: aws/amazon-vpc-cni-k8s#1587

Testing done:

Testing for IPv6 EKS Clusters:
Created an AMI off of the built images and launched an EC2 instance in my IPv6 EKS cluster.
The node is able to get an IPv6 internal IP address:

$ kubectl get nodes -o wide
NAME                                          STATUS   ROLES    AGE   VERSION   INTERNAL-IP                              EXTERNAL-IP      OS-IMAGE                KERNEL-VERSION   CONTAINER-RUNTIME
ip-192-168-3-236.us-west-2.compute.internal   Ready    <none>   26m   v1.21.3   2600:1f13:8ff:6600:6a76:135f:ee92:3395   54.188.180.229   Bottlerocket OS 1.2.0   5.10.50          containerd://1.4.8+bottlerocket

Can run pods without problem. The pods get IPv6 addresses assigned without problem. I can ping external ipv4 and ipv6 addresses without problem.

Testing for IPv4 EKS Clusters:
Launched node with AMI in IPv4 cluster.
Node becomes Ready with the correct node-ip and cluster-dns-ip:

$ kubectl get nodes -o wide
NAME                                           STATUS   ROLES    AGE   VERSION   INTERNAL-IP      EXTERNAL-IP     OS-IMAGE                KERNEL-VERSION   CONTAINER-RUNTIME
ip-192-168-1-232.us-west-2.compute.internal    Ready    <none>   28s   v1.21.3   192.168.1.232    54.190.54.101   Bottlerocket OS 1.2.0   5.10.50          containerd://1.4.8+bottlerocket

Can run pods without problem.

Ran conformance tests with an IPv6 cluster and all tests passed.

Terms of contribution:

By submitting this pull request, I agree that this contribution is dual-licensed under the terms of both the Apache License, version 2.0, and the MIT license.

sources/api/pluto/src/main.rs

etungsten · 2021-08-13T17:36:12Z

Push above addresses @webern 's comments.

Removed rusoto related imports from main.rs
Created new type alias for rusoto_eks::KubernetesNetworkConfigResponse for sake of brevity
Fixed unit tests

etungsten · 2021-09-10T23:12:02Z

Push above changes the approach to determining EKS cluster IP family. Instead of making a call to EKS API to determine the cluster's IP family, we check cluster-dns-ip's IP family and if it's an IPv4 we assign the instance's IPv4 IP as node-ip; If it's an IPv6 address, we assign the instance's IPv6 address to be the node-ip.

When users want to give their nodes an IPv6 node IP address for an IPv6 EKS Cluster, they need to set the cluster's DNS IP to settings.kubernetes.cluster-dns-ip via instance userdata.

This is a temporary measure until the EKS API provides ways to let us automatically determine the IP family and service IPv6 CIDR.

etungsten · 2021-09-10T23:18:47Z

Push above fixes README documentation for settings.kubernetes.node-ip and settings.kubernetes.cluster-dns-ip.

This adds a patch to the in-tree AWS cloud provider code to make kubelet append IPv6 addresses to the list of NodeAddresses.

etungsten · 2021-09-13T23:17:04Z

Push above adds the patch to kubelet for supporting ipv6 node addresses in AWS.
Tested everything and they work as expected.

README.md

packages/release/release-sysctl.conf

sources/imdsclient/src/lib.rs

sources/api/pluto/src/main.rs

etungsten · 2021-09-21T19:31:35Z

Pushes above addresses @bcressey 's comments. Tested the changes and they still work as expected.

sources/api/pluto/README.md

sources/imdsclient/src/lib.rs

bcressey · 2021-09-22T15:05:21Z

sources/api/pluto/src/main.rs

+        ))]
+        AwsInfo { source: api::Error },
+
+        #[snafu(display("Missing field '{}' in EKS kubernetes network config response", field))]


nit:

Suggested change

#[snafu(display("Missing field '{}' in EKS kubernetes network config response", field))]

#[snafu(display("Missing field '{}' in EKS network config response", field))]

etungsten · 2021-09-22T16:56:20Z

Push above addresses comments by @bcressey . Tested and things still work as expected.

sources/api/pluto/src/main.rs

etungsten · 2021-09-22T19:18:15Z

Push above addresses Matt's comment #1710 (comment)

sources/api/pluto/src/main.rs

This adds support for retrieving IPv6 node IP address for IPv6 EKS clusters

etungsten · 2021-09-22T19:52:29Z

Push above fixes the fallthrough for determining the cluster dns ip.

etungsten · 2021-09-24T23:50:08Z

Ran conformance tests with an IPv6 cluster and all tests passed.

etungsten requested review from bcressey and webern August 11, 2021 22:23

etungsten force-pushed the eks-ipv6 branch from ec1a095 to 781ba1b Compare August 11, 2021 22:30

webern reviewed Aug 12, 2021

View reviewed changes

sources/api/pluto/src/main.rs Outdated Show resolved Hide resolved

sources/api/pluto/src/main.rs Outdated Show resolved Hide resolved

etungsten force-pushed the eks-ipv6 branch 3 times, most recently from e6cd791 to 88e2b6a Compare August 13, 2021 17:34

etungsten force-pushed the eks-ipv6 branch from 6ab7fc7 to 3ccf423 Compare August 13, 2021 18:53

This comment has been minimized.

Sign in to view

etungsten force-pushed the eks-ipv6 branch from 3ccf423 to 185d735 Compare September 10, 2021 20:20

etungsten changed the title ~~pluto: support IPv6 clusters~~ pluto: support IPv6 Sep 10, 2021

etungsten force-pushed the eks-ipv6 branch from 185d735 to 20af379 Compare September 10, 2021 23:18

etungsten force-pushed the eks-ipv6 branch from 20af379 to 89cafff Compare September 10, 2021 23:22

kubernetes-1.21: patch kubelet to support ipv6 with AWS

a54e735

This adds a patch to the in-tree AWS cloud provider code to make kubelet append IPv6 addresses to the list of NodeAddresses.

etungsten force-pushed the eks-ipv6 branch from 89cafff to b5ba413 Compare September 13, 2021 23:05

etungsten changed the title ~~pluto: support IPv6~~ kubelet, pluto: support IPv6 Sep 13, 2021

etungsten requested a review from webern September 13, 2021 23:21

etungsten marked this pull request as ready for review September 14, 2021 23:21

bcressey requested changes Sep 20, 2021

View reviewed changes

README.md Outdated Show resolved Hide resolved

packages/release/release-sysctl.conf Outdated Show resolved Hide resolved

sources/imdsclient/src/lib.rs Outdated Show resolved Hide resolved

sources/api/pluto/src/main.rs Outdated Show resolved Hide resolved

etungsten force-pushed the eks-ipv6 branch 2 times, most recently from e3d6948 to 2e0ed0b Compare September 21, 2021 19:20

etungsten requested a review from bcressey September 21, 2021 19:31

bcressey reviewed Sep 22, 2021

View reviewed changes

etungsten force-pushed the eks-ipv6 branch from 2e0ed0b to baf5b96 Compare September 22, 2021 16:44

etungsten requested a review from bcressey September 22, 2021 17:03

webern reviewed Sep 22, 2021

View reviewed changes

sources/api/pluto/src/main.rs Outdated Show resolved Hide resolved

etungsten force-pushed the eks-ipv6 branch from baf5b96 to a6ab121 Compare September 22, 2021 18:43

etungsten requested a review from webern September 22, 2021 19:18

etungsten commented Sep 22, 2021

View reviewed changes

sources/api/pluto/src/main.rs Outdated Show resolved Hide resolved

etungsten added 2 commits September 22, 2021 12:37

pluto: support IPv6

5a567a7

This adds support for retrieving IPv6 node IP address for IPv6 EKS clusters

release: enable IPv6 forwarding

2bf5e10

etungsten force-pushed the eks-ipv6 branch from a6ab121 to 2bf5e10 Compare September 22, 2021 19:37

webern approved these changes Sep 23, 2021

View reviewed changes

bcressey approved these changes Sep 28, 2021

View reviewed changes

etungsten merged commit dbaa456 into bottlerocket-os:develop Sep 28, 2021

etungsten deleted the eks-ipv6 branch September 28, 2021 18:16

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

kubelet, pluto: support IPv6 #1710

kubelet, pluto: support IPv6 #1710

etungsten commented Aug 11, 2021 •

edited

Loading

etungsten commented Aug 13, 2021

This comment has been minimized.

etungsten commented Sep 10, 2021

etungsten commented Sep 10, 2021

etungsten commented Sep 13, 2021

etungsten commented Sep 21, 2021 •

edited

Loading

bcressey Sep 22, 2021

etungsten commented Sep 22, 2021

etungsten commented Sep 22, 2021

etungsten commented Sep 22, 2021

etungsten commented Sep 24, 2021 •

edited

Loading

	#[snafu(display("Missing field '{}' in EKS kubernetes network config response", field))]
	#[snafu(display("Missing field '{}' in EKS network config response", field))]

kubelet, pluto: support IPv6 #1710

kubelet, pluto: support IPv6 #1710

Conversation

etungsten commented Aug 11, 2021 • edited Loading

etungsten commented Aug 13, 2021

This comment has been minimized.

etungsten commented Sep 10, 2021

etungsten commented Sep 10, 2021

etungsten commented Sep 13, 2021

etungsten commented Sep 21, 2021 • edited Loading

bcressey Sep 22, 2021

Choose a reason for hiding this comment

etungsten commented Sep 22, 2021

etungsten commented Sep 22, 2021

etungsten commented Sep 22, 2021

etungsten commented Sep 24, 2021 • edited Loading

etungsten commented Aug 11, 2021 •

edited

Loading

etungsten commented Sep 21, 2021 •

edited

Loading

etungsten commented Sep 24, 2021 •

edited

Loading