Replace kube proxy by cilium and ebpf

[kopiczko]

CAPVCD

Preview Give feedback

1. Use cilium as kube-proxy replacement cluster-cloud-director#104
   +0
   
2. https://github.com/giantswarm/mc-bootstrap/pull/437
3. Add notes about DNS requirement cluster-cloud-director#120
   +0
   
4. Release v0.8.1 cluster-cloud-director#118
   +0
   

CAPV

Preview Give feedback

1. Use cilium as kube-proxy replacement cluster-vsphere#38
   +0
   
2. https://github.com/giantswarm/mc-bootstrap/pull/437
3. https://github.com/giantswarm/mc-bootstrap/pull/462
4. Add notes about DNS requirement cluster-vsphere#41
   +0
   
5. Release v0.3.0 cluster-vsphere#40
   +0
   

[cornelius-keller]

@giantswarm/team-cabbage team rocket would like to do this to get rid of iptables. As this is network related, is there an impact on your roadmap or stories you are planning?

[cornelius-keller]

Feedback from @giantswarm/wg-capi-mc:

• capa, capg, capz should use cillium without kube-proxy
• this involves going to use c-groups v2
• sould not interfere with any service-mesh etc.

[kopiczko]

CC @giantswarm/team-phoenix

[weatherhog]

@cornelius-keller at the moment we in @giantswarm/team-cabbage do not have something in the direction of cilium and ebpf. We are happy to help out regarding the replacement. But do not really have capacities for that at the moment

[paurosello]

From the customer side, cGroups v2 is still a big issue, we will have many customers in v1 for a while.

[kopiczko]

From CAPO point of view:

• we are on ubuntu 20.04 and according to this cgroups v2 are enabled with 21.10. We probably could switch to 22.04. I'm not sure if this is supported by image-builder but probably will be if not
• we need to add --skip-phases=addon/kube-proxy to kubeadm init/join. This was added in February to CAPI but is still not released
• general problem: "Be aware that this will break existing service connections. It will also stop service related traffic until the Cilium replacement has been installed" (from https://docs.cilium.io/en/v1.11/gettingstarted/kubeproxy-free/)
• general: requires kernel 5.8 and has some limitations (from https://docs.cilium.io/en/v1.11/gettingstarted/kubeproxy-free/#limitations)

[kopiczko]

Check #1456

[gawertm]

not directly related but could be relevant: #1594
Cabbage was working on Hubble and Phoenix is about to enable/introduce it for all their clusters

[kopiczko]

For CAPVCD: giantswarm/cluster-cloud-director#104

This works for WCs, but it's not going to work in mc-bootstrap out of the box because we template kubeapi domain and we'll need to pass the IP during the bootstrap.

[vxav]

Sorry I didn't get that part? What do you mean?

because we template kubeapi domain

[kopiczko]

It's templated here https://github.com/giantswarm/cluster-cloud-director/blob/8fa7cd6df6d7c85566f231d6e8dfcc8a589bd961/helm/cluster-cloud-director/templates/cilium-helmrelease.yaml#L33

During the bootstrap the domain doesn't exist. My way to work around this is here https://github.com/giantswarm/mc-bootstrap/pull/437/commits/94215215e1af7766122b1cdab7b0bac3550bdbff

[kopiczko]

This is done for CAPVCD. Now CAPV

[kopiczko]

CAPV PRs in review.

[kopiczko]

Done for CAPV.