IPv6 networks unreachable in pods

[JanKrb]

Description

As soon as I'm trying to reach any ipv6 network, I'll get a network error. This is the case when using the ip adress or AAAA-only DNS records.

While browsing through the ipv6 related issues I've found this issue #772 which looks like it is the same error. I'm not quite sure if it's the same case? It has been closed a while ago.

To reproduce, I created a new cluster with the default config. Only thing I changed is the autoscaling, but it is also happening without. I booted myself a new pod (busybox in my case) kubectl run test --image=busybox --rm -it -- sh and tried ping6 google.com.

My nodes seem to have access to ipv6.

Kube.tf file

locals {
  hcloud_token = "xxxx"
}

module "kube-hetzner" {
  providers = {
    hcloud = hcloud
  }
  hcloud_token = var.hcloud_token != "" ? var.hcloud_token : local.hcloud_token

  source = "github.com/kube-hetzner/terraform-hcloud-kube-hetzner"

  ssh_public_key = file("./keys.pub")
  ssh_private_key = file("./keys")
  
  network_region = "eu-central"

  control_plane_nodepools = [
    {
      name        = "control-plane-fsn1",
      server_type = "cpx11",
      location    = "fsn1",
      labels      = [],
      taints      = [],
      count       = 1
    },
    {
      name        = "control-plane-nbg1",
      server_type = "cpx11",
      location    = "nbg1",
      labels      = [],
      taints      = [],
      count       = 1
    },
    {
      name        = "control-plane-hel1",
      server_type = "cpx11",
      location    = "hel1",
      labels      = [],
      taints      = [],
      count       = 1
    }
  ]

  agent_nodepools = [
    {
      name        = "agent-small",
      server_type = "cpx11",
      location    = "fsn1",
      labels      = [],
      taints      = [],
      count       = 1
    },
    {
      name        = "agent-large",
      server_type = "cpx21",
      location    = "nbg1",
      labels      = [],
      taints      = [],
      count       = 1
    },
    {
      name        = "storage",
      server_type = "cpx21",
      location    = "fsn1",
      labels      = [
        "node.kubernetes.io/server-usage=storage"
      ],
      taints      = [],
      count       = 1
    },
    {
      name        = "egress",
      server_type = "cpx11",
      location    = "fsn1",
      labels = [
        "node.kubernetes.io/role=egress"
      ],
      taints = [
        "node.kubernetes.io/role=egress:NoSchedule"
      ],
      floating_ip = true
      count = 1
    },
    {
      name        = "agent-arm-small",
      server_type = "cax11",
      location    = "fsn1",
      labels      = [],
      taints      = [],
      count       = 1
    }
  ]
  
  load_balancer_type     = "lb11"
  load_balancer_location = "fsn1"
  load_balancer_disable_ipv6 = false
  load_balancer_algorithm_type = "least_connections"

  autoscaler_nodepools = [
    {
      name        = "autoscaled-small"
      server_type = "cpx21"
      location    = "fsn1"
      min_nodes   = 1
      max_nodes   = 5
      labels      = {
        "node.kubernetes.io/role": "peak-workloads"
      }
      taints = [{
         key: "node.kubernetes.io/role"
         value: "peak-workloads"
         effect: "NoExecute"
      }]  
    },
    {
      name        = "autoscaled-medium"
      server_type = "cpx31"
      location    = "fsn1"
      min_nodes   = 1
      max_nodes   = 5
      labels      = {
        "node.kubernetes.io/role": "peak-workloads"
      }
      taints = [{
         key: "node.kubernetes.io/role"
         value: "peak-workloads"
         effect: "NoExecute"
      }]  
    },
    {
      name        = "autoscaled-high"
      server_type = "cpx41"
      location    = "fsn1"
      min_nodes   = 0
      max_nodes   = 5
      labels      = {
        "node.kubernetes.io/role": "peak-workloads"
      }
      taints = [{
         key: "node.kubernetes.io/role"
         value: "peak-workloads"
         effect: "NoExecute"
      }]  
    },
    {
      name        = "autoscaled-max"
      server_type = "cpx51"
      location    = "fsn1"
      min_nodes   = 0
      max_nodes   = 5
      labels      = {
        "node.kubernetes.io/role": "peak-workloads"
      }
      taints = [{
         key: "node.kubernetes.io/role"
         value: "peak-workloads"
         effect: "NoExecute"
      }]  
    }
  ]
}

provider "hcloud" {
  token = var.hcloud_token != "" ? var.hcloud_token : local.hcloud_token
}

terraform {
  required_version = ">= 1.5.0"
  required_providers {
    hcloud = {
      source  = "hetznercloud/hcloud"
      version = ">= 1.43.0"
    }
  }
}

output "kubeconfig" {
  value     = module.kube-hetzner.kubeconfig
  sensitive = true
}

variable "hcloud_token" {
  sensitive = true
  default   = ""
}

(trimmed the comments)

Screenshots

This is my pod:

--

This is my node (k3s-control-plane):

Platform

MicroOS (openSUSE-Tumbleweed)

[mysticaltech]

@JanKrb Which version of the module are you using? Please make sure you upgrade to the latest version with terraform init -upgrade. We fixed a lot of IPv6 errors a few months back.

[mysticaltech]

Also, if your snapshots are old, you need to recreate them. See pinned discussion on upgrade.

[JanKrb]

Created an all new cluster without any existing resources. So snapshots shouldn't be the issue here. I'm using the latest module (v2.9.2) directly from GitHub.

[mysticaltech]

@M4t7e Any idea on this please?

[mysticaltech]

@JanKrb You are correct. Just confirmed it.

18:44:55 in ~/code/kube-test at ☸️  test12  
➜ kubectl run test --image=busybox --rm -it -- sh
If you don't see a command prompt, try pressing enter.
/ # 
/ # ping6 google.com
PING google.com (2a00:1450:4001:828::200e): 56 data bytes
ping6: sendto: Network is unreachable

18:48:57 in ~/code/kube-test at ☸️  test12  
➜ ssh root@159.69.0.65 -i ~/.ssh/id_ed25519 -o StrictHostKeyChecking=no
Warning: Permanently added '159.69.0.65' (ED25519) to the list of known hosts.
Last login: Tue Nov  7 17:43:17 UTC 2023 from 88.23.61.24 on ssh
test12-agent-small-sby:~ # ping6 google.com
PING google.com(fra16s50-in-x0e.1e100.net (2a00:1450:4001:810::200e)) 56 data bytes
64 bytes from fra16s50-in-x0e.1e100.net (2a00:1450:4001:810::200e): icmp_seq=1 ttl=115 time=4.60 ms
64 bytes from fra16s50-in-x0e.1e100.net (2a00:1450:4001:810::200e): icmp_seq=2 ttl=115 time=4.18 ms
64 bytes from fra16s50-in-x0e.1e100.net (2a00:1450:4001:810::200e): icmp_seq=3 ttl=115 time=4.00 ms
^C
--- google.com ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2003ms
rtt min/avg/max/mdev = 4.002/4.258/4.595/0.248 ms

[mysticaltech]

@kube-hetzner/core Any ideas folks why that would be happening. It's interesting that it gets the dns resolution right. But can't ping from within a pod.

[mysticaltech]

I think we just do not support it yet. As Hetzner private network do not support IPv6 (unless I am mistaken). More on the dual stack that we do not implement here https://docs.k3s.io/installation/network-options#dual-stack-ipv4--ipv6-networking.

@aleksasiriski @M4t7e @ifeulner What do you think?

[JanKrb]

As Hetzner private network IPv6 do not support IPv6 (unless I am mistaken).

That's right. Currently you can only create IPv4 networks on the Hetzner interface. I have asked Hetzner whether it is also possible to create IPv6 networks or whether they have any initiatives for implementation.

[M4t7e]

@mysticaltech Yes, you are totally right! IPv6 support is currently only available for ingress via LB; egress from Pods is (almost) impossible to achieve with Hetzner, as they do not support native routed IPv6.

Regarding DNS, it is not mandatory to use IPv6 to query for IPv6 DNS records. Queries over IPv4 can retrieve both A (IPv4 addresses) and AAAA (IPv6 addresses) records. It is up to the client to decide whether to request A or AAAA records, typically based on the IP protocols that are supported and if both are available, IPv6 is usually preferred.

The only way to make public IPv6 addresses work internally is via an overlay network. However, configuring K3s IPAM with IPv6 networks for individual nodes is challenging because you can only specify a single IPv6 CIDR. K3s IPAM will then autonomously create subnetworks from this CIDR and distribute them to the nodes, which will not align with the IPv6 networks assigned to the nodes by Hetzner.

An alternative is to set up a network with private IPv6 addresses. This still requires an overlay, as Hetzner does not support this configuration either. In this setup, the overlay network would manage internal routing, and at least Cilium does offer experimental support for IPv6 masquerading for Pods. This could work, but I haven't tested it and wouldn't recommend it for various reasons. Additionally, many will argue that NAT is not permissible with IPv6.

Other solutions are only theoretically possible and would require custom implementations in either K3s IPAM or HCCM IPAM to correctly assign IPv6 networks to nodes. With such an implementation, you could achieve a native IPv6 connection to the internet while using an overlay network internally. Native IPv6 support for Hetzner's networks would still be a significant improvement.

[mysticaltech]

@M4t7e, beautiful in-depth explanation, thank you! 🙏

@JanKrb Up to you, you could try with Cilium like suggested above, you need to enable it via the cni_plugin variable and can use the cilium_values variable to configure it. If you manage to make it work, please don't hesitate to PR an example to our docs (like in the readme examples section), that would be awesome.

[caniko]

Got this from Hetzner @JanKrb:

Dear Mr Can

Unfortunately we don't offer this feature in general.

There is nothing planned at this point.

They were very certain that they offered it at first but relented when I pointed out that their docs only use IPv4 examples; typical sales people... Anyway, seems Cilium is the only option.

Also, @M4t7e, why?

wouldn't recommend it for various reasons

@mysticaltech are you planning on adding support for dual-stack?

[mysticaltech]

@caniko At this moment we can't because the Hetzner private network does not support it. However, for ingress (via lb) or egress of the cluster IPv6 works now.

[ggogel]

@mysticaltech I'm seeing IPv6-related errors in my cluster. The connection times out inside a pod whenever a domain is resolved to an IPv6 address. In other words, IPv6 egress from inside pods isn't working. Do you know if this is expected? Do you know how I can resolve this?

[ggogel]

Some additional information. For the domains causing this issue, the DNS lookup response contains an AAAA and A record. For whatever reason the application decides to prefer the AAAA record. Then the connection times out. I tried different approaches like CoreDNS configuration and disabling IPv6 on the hosts but nothing was successful.

Does somebody know a workaround for this? This might even be a bug. If the cluster can't process IPv6 egress, then IPv6 should be prevented.