Support IPv6DualStack

[thuck]

RKE version:
v1.0.4

Docker version:
18.09.1

Operating system and kernel:
Debian GNU/Linux 10 (buster), 4.19.0-6-amd64

Type/provider of hosts:
KVM

cluster.yml file:

nodes:
  - address: 192.168.122.254
    internal_address: 192.168.122.254
    role:
      - controlplane
      - etcd
      - worker

ignore_docker_version: false
cluster_name: cluster.local
kubernetes_version: v1.17.2-rancher1-2
addon_job_timeout: 300

services:
  kube-api:
    service_cluster_ip_range: 10.42.0.0/24
    service_node_port_range: 30000-32767
    pod_security_policy: false
    always_pull_images: false
    extra_args:
      audit-log-path: "/var/log/audit/"
      runtime-config: 'authorization.k8s.io/v1beta1=true,apps/v1beta1=true,apps/v1beta2=true,extensions/v1beta1/daemonsets=true,extensions/v1beta1/deployments=true,extensions/v1beta1/replicasets=true,extensions/v1beta1/networkpolicies=true,extensions/v1beta1/podsecuritypolicies=true'
      delete-collection-workers: 3
      feature-gates: "SCTPSupport=True,IPv6DualStack=true"

  kube-controller:
    cluster_cidr: "10.43.0.0/24,fc00::/64"
    service_cluster_ip_range: "10.42.0.0/24,fc01::/64"
    extra_args:
      feature-gates: "IPv6DualStack=true"

  kubeproxy:
    cluster_cidr: "10.43.0.0/24,fc00::/64"
    extra_args:
        feature-gates: "IPv6DualStack=true"
        proxy-mode: "ipvs"

  kubelet:
    cluster_domain: cluster.local
    cluster_dns_server: 10.42.0.3
    fail_swap_on: false
    extra_args:
      max-pods: 20
      pod-manifest-path: /etc/kubernetes/manifests
      feature-gates: "IPv6DualStack=true"
      v: 2

authorization:
  mode: rbac

network:
  plugin: calico

ingress:
  provider: none

dns:
  provider: coredns

Steps to Reproduce:
rke up
Results:
Calico fails to create an IPV6 network:
2020-02-04 15:53:28.006 [ERROR][8] startup.go 779: Invalid CIDR specified in CALICO_IPV4POOL_CIDR '10.43.0.0/24,fc00::/64'

Template for calico do not contain proper environment variable:

- name: CALICO_IPV6POOL_CIDR
  value: "fd20::0/112"
- name: IP6
  value: "autodetect"

Calico documenation

[lwolf]

I also tried to deploy dual-stack. Worked around calico issue described above by using network.plugin: none and deploying calico separately.
Faced another issue though: rke doesn't allow to set service_cluster_ip_range for kube-apiserver to more that one subset.

failed set network options: Failed to get Kubernetes Service IP: Failed to get kubernetes service IP from Kube API option [service_cluster_ip_range]: invalid CIDR address: 10.43.0.0/16,2a00:xxxx:xxxx:xxxx::/112

So, currently I have semi-working setup running: pods are dual-stack, but services are not.

the issue is in GetKubernetesServiceIP function here:

rke/pki/util.go

Line 382 in 4c1d3db

func GetKubernetesServiceIP(serviceClusterRange string) (net.IP, error) {

it used to validate service IP and can't parse array of values.

[lwolf]

Don't know rke codebase well enough, but this seems to be enough to make it work. At least I was able to deploy dual-stack cluster.

lwolf@b332b56

Can make a PR if somebody from rancher could confirm that this looks ok.

[ghakfoort]

any news on this one?

I try to create a dual stack enabled cluster. According to the Kubernetes documentation, a few settings need to be set correctly:

kube-apiserver:
--feature-gates="IPv6DualStack=true"
--service-cluster-ip-range=,
kube-controller-manager:
--feature-gates="IPv6DualStack=true"
--cluster-cidr=,
--service-cluster-ip-range=,
--node-cidr-mask-size-ipv4|--node-cidr-mask-size-ipv6 defaults to /24 for IPv4 and /64 for IPv6
kubelet:
--feature-gates="IPv6DualStack=true"
kube-proxy:
--cluster-cidr=,
--feature-gates="IPv6DualStack=true"

This can be done for the Kube-controller:
services:
kube-controller:
cluster_cidr: "10.42.0.0/24,fc00::/64"
service_cluster_ip_range: "10.43.0.0/24,fc01::/64"
extra_args:
feature-gates: "IPv6DualStack=true"
node-cidr-mask-size-ipv4: 24
node-cidr-mask-size-ipv6: 64

but it fails at the Kube-api:
services:
kube-api:
service_cluster_ip_range: "10.43.0.0/24,fc01::/64"
extra_args:
feature-gates: "IPv6DualStack=true"

I don't seem to be able to configure 2 (ipv4 and ipv6) ip ranges in the service_cluster_ip_range of the kube-api.
If I only configure the ipv4 ip range. "rke up" works correctly. But ofcourse, dual stack isn't configured well. If I manage calico manually, I'm able to assign ip addresses in both stacks to the pods and egress traffic works OK but I need to solve the whole puzzle :)

[JustInVTime]

👍 on this one. I know it's still an alpha feature in kubernetes, but (experimental) support in RKE would be very nice!

[stale]

This issue/PR has been automatically marked as stale because it has not had activity (commit/comment/label) for 60 days. It will be closed in 14 days if no further activity occurs. Thank you for your contributions.

[n1kofr]

Is there any plan / roadmap on RKE to have this feature available?

[stale]

This issue/PR has been automatically marked as stale because it has not had activity (commit/comment/label) for 60 days. It will be closed in 14 days if no further activity occurs. Thank you for your contributions.

[n1kofr]

Any news for this issue?

[WebSpider]

I'm very interested in this as well, since my cloud provider has support for IPv6 in all relevant areas

[c-mckenzie]

I'd also be very keen to get any update on this issue, especially with the upcoming promotion of dual-stack support in k8s 1.21

[ghost]

+1 we would have to abandon RKE if IPv6 Dualstack is not supported soon.

[ederqueirozdf]

RKE: v2.4.15
I don't seem to be able to configure 2 (ipv4 and ipv6) ip ranges in the service_cluster_ip_range of the kube-api.
any news about the Dualstack in the rancher? is supported?

[igorcezar]

I also tried to deploy dual-stack. Worked around calico issue described above by using network.plugin: none and deploying calico separately.
Faced another issue though: rke doesn't allow to set service_cluster_ip_range for kube-apiserver to more that one subset.

failed set network options: Failed to get Kubernetes Service IP: Failed to get kubernetes service IP from Kube API option [service_cluster_ip_range]: invalid CIDR address: 10.43.0.0/16,2a00:xxxx:xxxx:xxxx::/112

So, currently I have semi-working setup running: pods are dual-stack, but services are not.

the issue is in GetKubernetesServiceIP function here:

rke/pki/util.go

Line 382 in 4c1d3db

func GetKubernetesServiceIP(serviceClusterRange string) (net.IP, error) {

it used to validate service IP and can't parse array of values.

The problem for rke is indeed in GetKubernetesServiceIP function, as @lwolf described.

A workaround for this would be setting kube-api like this:

kube-api:
    service_cluster_ip_range: 10.43.0.0/24
    extra_args:
      service-cluster-ip-range: "10.43.0.0/24,fc01::/64"
      feature-gates: "IPv6DualStack=true"

With this,GetKubernetesServiceIP function will validate service IP without problems, and service-cluster-ip-range in extra_args will override the flag --service-cluster-ip-range in kube-apiserver component, so the cluster will have all configuration it needs to run dual stack.

This still requires calico to be deployed separately.

[superseb]

Merging into rancher/rancher#33107, if anyone can test it out, that would be helpful.

[n1kofr]

@superseb, it looks like the implementation provided supports only Calico. Any reason why?

We are using Canal as a network plugin with VXLAN. When deploying a kubernetes cluster with latest RKE 1.3.0-RC15, we got the error:

Failed to validate cluster: Network plugin [canal] does not support IPv6 (dualstack)

Our VMs are deployed on our Openstack environment, would you recommend another network plugin than Canal to support DualStack? It seems like calico supports only AWS or GCE from the documentation.

[superseb]

@n1kofr Because Flannel does not support dual stack yet, see flannel-io/flannel#248. The requirements are in rancher/rancher#33107 (comment), let me know if you run into anything and provide debug output.

[n1kofr]

Thanks for the feedback. I though Flannel was able to support DualStack when i checked the documentation:

https://github.com/flannel-io/flannel/blob/master/Documentation/configuration.md

[superseb]

Yeah no clue why docs have been merged while its not released, last release is 0.14.0 and was released before dual stack support was added.

[n1kofr]

Oh yes, thanks, I can see now that the pull request for VXLAN DualStack was merged on July 13th
flannel-io/flannel#1448

Does it mean you will enable Canal / Flannel DualStack support once a new Flannel is available with VXLAN?

[n1kofr]

@superseb I noticed that RKE 1.3.3 is now RC and includes Flannel 0.15.1 (providing DualStack support).
Does it mean RKE will now support DualStack with Canal?
If not, could you share what is the plan (if any) to support it?

[superseb]

It won't be in 1.3.3, we only enabled and tested DualStack for Calico since v1.3.0. Now that we have Flannel able to do it, we can start the work to make it configurable for Flannel and possibly Canal (no clue what the stance is from Canal on DualStack)

[superseb]

@n1kofr Please file a new issue for it

[n1kofr]

@superseb Ok, thanks and yes I will create a new issue/improvement for this.