Add eBPF dataplane support for Calico CNI

docs/networking/calico.md

@@ -93,6 +93,23 @@ It is possible to configure Calico to use Typha by editing a cluster and adding
       typhaReplicas: 3
 ```
 
+### Configuring the eBPF dataplane
+{{ kops_feature_table(kops_added_default='1.19', k8s_min='1.16') }}
+
+Calico supports using an [eBPF dataplane](https://docs.projectcalico.org/about/about-ebpf) as an alternative to the standard Linux dataplane (which is iptables based). While the standard dataplane focuses on compatibility by inter-working with kube-proxy, and your own iptables rules, the eBPF dataplane focuses on performance, latency and improving user experience with features that aren’t possible in the standard dataplane. As part of that, the eBPF dataplane replaces kube-proxy with an eBPF implementation. The main “user experience” feature is to preserve the source IP of traffic from outside the cluster when traffic hits a NodePort; this makes the server-side logs and network policy much more useful on that path.
+
+For more details on enabling the eBPF dataplane please refer the [Calico Docs](https://docs.projectcalico.org/maintenance/ebpf/enabling-bpf).
+
+```yaml
+  kubeProxy:
+    enabled: false
+  networking:
+    calico:
+      bpfEnabled: true
+      bpfExternalServiceMode: Tunnel
+      bpfLogLevel: Info
+```
+
 ### Configuring WireGuard
 {{ kops_feature_table(kops_added_default='1.19', k8s_min='1.16') }}
 

docs/releases/1.19-NOTES.md

@@ -21,7 +21,9 @@ Kops will install the Cinder plugin for kops running kubernetes 1.16 or newer. I
 
 If you already have a default `StorageClass`, you should set `cloudConfig.Openstack.BlockStorage.CreateStorageClass: false` to prevent kops from installing one.
 
-## Other significant changes
+## Other significant changes by kind
+
+### General
 
 * New clusters will now have one nodes group per zone. The number of nodes now defaults to the number of zones.
 
@@ -46,7 +48,9 @@ has been updated by a newer version of kops unless it is given the `--allow-kops
 
 * Clusters using Calico with `CrossSubnet` enabled will switch to the new [awsSrcDstCheck](https://docs.projectcalico.org/reference/resources/felixconfig#spec) for disabling the [AWS source/destination checks](https://docs.aws.amazon.com/vpc/latest/userguide/VPC_NAT_Instance.html#EIP_Disable_SrcDestCheck). The previous implementation using [k8s-ec2-srcdst](https://github.com/ottoyiu/k8s-ec2-srcdst) is now deprecated.
 
-* Clusters using Calico can now  encrypt pod-to-pod traffic with [WireGuard](https://docs.projectcalico.org/security/encrypt-cluster-pod-traffic) for Ubuntu hosts. Add `spec.networking.calico. wireguardEnabled: true` to the cluster spec to enable.
+* Clusters using Calico can now enable the [eBPF dataplane](https://docs.projectcalico.org/about/about-ebpf) mode for Ubuntu 20.04 (Focal) hosts. Add `spec.networking.calico.bpfEnabled: true` and `spec.kubeProxy.enabled: false` to the cluster spec to enable.
+
+* Clusters using Calico can now encrypt pod-to-pod traffic with [WireGuard](https://docs.projectcalico.org/security/encrypt-cluster-pod-traffic) for Ubuntu hosts. Add `spec.networking.calico.wireguardEnabled: true` to the cluster spec to enable.
 
 * New clusters running Cilium now enable BPF NodePort by default if the Kubernetes version is 1.12 or newer.
 

k8s/crds/kops.k8s.io_clusters.yaml

@@ -2138,6 +2138,15 @@ spec:
                       awsSrcDstCheck:
                         description: 'AwsSrcDstCheck enables/disables source/destination checks (AWS only) Options: "DoNothing" (default) , "Enable" or "Disable"'
                         type: string
+                      bpfEnabled:
+                        description: BpfEnabled enables the eBPF dataplane mode.
+                        type: boolean
+                      bpfExternalServiceMode:
+                        description: 'BPFExternalServiceMode controls how traffic from outside the cluster to NodePorts and ClusterIPs is handled. In Tunnel mode, packet is tunneled from the ingress host to the host with the backing pod and back again. In DSR mode, traffic is tunneled to the host with the backing pod and then returned directly; this requires a network that allows direct return. Default: Tunnel (other options: DSR)'
+                        type: string
+                      bpfLogLevel:
+                        description: 'BPFLogLevel controls the log level used by the BPF programs. The logs are emitted to the BPF trace pipe, accessible with the command tc exec BPF debug. Default: Off (other options: Info, Debug)'
+                        type: string
                       chainInsertMode:
                         description: 'ChainInsertMode controls whether Felix inserts rules to the top of iptables chains, or appends to the bottom. Leaving the default option is safest to prevent accidentally breaking connectivity. Default: ''insert'' (other options: ''append'')'
                         type: string

pkg/apis/kops/networking.go

@@ -100,6 +100,18 @@ type FlannelNetworkingSpec struct {
 
 // CalicoNetworkingSpec declares that we want Calico networking
 type CalicoNetworkingSpec struct {
+	// BpfEnabled enables the eBPF dataplane mode.
+	BPFEnabled bool `json:"bpfEnabled,omitempty"`
+	// BPFExternalServiceMode controls how traffic from outside the cluster to NodePorts and ClusterIPs is handled.
+	// In Tunnel mode, packet is tunneled from the ingress host to the host with the backing pod and back again.
+	// In DSR mode, traffic is tunneled to the host with the backing pod and then returned directly;
+	// this requires a network that allows direct return.
+	// Default: Tunnel (other options: DSR)
+	BPFExternalServiceMode string `json:"bpfExternalServiceMode,omitempty"`
+	// BPFLogLevel controls the log level used by the BPF programs. The logs are emitted
+	// to the BPF trace pipe, accessible with the command tc exec BPF debug.
+	// Default: Off (other options: Info, Debug)
+	BPFLogLevel string `json:"bpfLogLevel,omitempty"`
 	// ChainInsertMode controls whether Felix inserts rules to the top of iptables chains, or
 	// appends to the bottom. Leaving the default option is safest to prevent accidentally
 	// breaking connectivity. Default: 'insert' (other options: 'append')

pkg/apis/kops/v1alpha2/networking.go

@@ -100,6 +100,18 @@ type FlannelNetworkingSpec struct {
 
 // CalicoNetworkingSpec declares that we want Calico networking
 type CalicoNetworkingSpec struct {
+	// BpfEnabled enables the eBPF dataplane mode.
+	BPFEnabled bool `json:"bpfEnabled,omitempty"`
+	// BPFExternalServiceMode controls how traffic from outside the cluster to NodePorts and ClusterIPs is handled.
+	// In Tunnel mode, packet is tunneled from the ingress host to the host with the backing pod and back again.
+	// In DSR mode, traffic is tunneled to the host with the backing pod and then returned directly;
+	// this requires a network that allows direct return.
+	// Default: Tunnel (other options: DSR)
+	BPFExternalServiceMode string `json:"bpfExternalServiceMode,omitempty"`
+	// BPFLogLevel controls the log level used by the BPF programs. The logs are emitted
+	// to the BPF trace pipe, accessible with the command tc exec BPF debug.
+	// Default: Off (other options: Info, Debug)
+	BPFLogLevel string `json:"bpfLogLevel,omitempty"`
 	// ChainInsertMode controls whether Felix inserts rules to the top of iptables chains, or
 	// appends to the bottom. Leaving the default option is safest to prevent accidentally
 	// breaking connectivity. Default: 'insert' (other options: 'append')

pkg/apis/kops/validation/validation.go

@@ -961,16 +961,26 @@ func validateNetworkingCalico(v *kops.CalicoNetworkingSpec, e kops.EtcdClusterSp
 		}
 	}
 
-	if v.ChainInsertMode != "" {
-		valid := []string{"insert", "append"}
-		allErrs = append(allErrs, IsValidValue(fldPath.Child("chainInsertMode"), &v.ChainInsertMode, valid)...)
-	}
-
 	if v.AwsSrcDstCheck != "" {
 		valid := []string{"Enable", "Disable", "DoNothing"}
 		allErrs = append(allErrs, IsValidValue(fldPath.Child("awsSrcDstCheck"), &v.AwsSrcDstCheck, valid)...)
 	}
 
+	if v.BPFExternalServiceMode != "" {
+		valid := []string{"Tunnel", "DSR"}
+		allErrs = append(allErrs, IsValidValue(fldPath.Child("bpfExternalServiceMode"), &v.BPFExternalServiceMode, valid)...)
+	}
+
+	if v.BPFLogLevel != "" {
+		valid := []string{"Off", "Info", "Debug"}
+		allErrs = append(allErrs, IsValidValue(fldPath.Child("bpfLogLevel"), &v.BPFLogLevel, valid)...)
+	}
+
+	if v.ChainInsertMode != "" {
+		valid := []string{"insert", "append"}
+		allErrs = append(allErrs, IsValidValue(fldPath.Child("chainInsertMode"), &v.ChainInsertMode, valid)...)
+	}
+
 	if v.IptablesBackend != "" {
 		valid := []string{"Auto", "Legacy", "NFT"}
 		allErrs = append(allErrs, IsValidValue(fldPath.Child("iptablesBackend"), &v.IptablesBackend, valid)...)

upup/models/cloudup/resources/addons/networking.projectcalico.org/k8s-1.16.yaml.template

@@ -1,4 +1,6 @@
-# Pulled and modified from: https://docs.projectcalico.org/v3.16/manifests/calico-typha.yaml
+# Pulled and modified from:
+#   https://docs.projectcalico.org/v3.16/manifests/calico-bpf.yaml
+#   https://docs.projectcalico.org/v3.16/manifests/calico-typha.yaml
 
 ---
 # Source: calico/templates/calico-config.yaml
@@ -11,6 +13,12 @@ metadata:
   labels:
     role.kubernetes.io/networking: "1"
 data:
+  {{- if .Networking.Calico.BPFEnabled }}
+  # Set these to the IP and port of your API server; In BPF mode, we need to connect directly to the
+  # API server because we take over kube-proxy's role.
+  kubernetes_service_host: "{{ .MasterInternalName }}"
+  kubernetes_service_port: "443"
+  {{- end }}
   # You must set a non-zero value for Typha replicas below.
   typha_service_name: "{{- if .Networking.Calico.TyphaReplicas -}}calico-typha{{- else -}}none{{- end -}}"
   # Configure the backend to use.
@@ -3758,6 +3766,19 @@ spec:
               name: kubernetes-services-endpoint
               optional: true
           env:
+            {{- if .Networking.Calico.BPFEnabled }}
+            # Overrides for kubernetes API server host/port. Needed in BPF mode.
+            - name: KUBERNETES_SERVICE_HOST
+              valueFrom:
+                configMapKeyRef:
+                  name: calico-config
+                  key: kubernetes_service_host
+            - name: KUBERNETES_SERVICE_PORT
+              valueFrom:
+                configMapKeyRef:
+                  name: calico-config
+                  key: kubernetes_service_port
+            {{- end }}
             - name: KUBERNETES_NODE_NAME
               valueFrom:
                 fieldRef:
@@ -3785,6 +3806,19 @@ spec:
               name: kubernetes-services-endpoint
               optional: true
           env:
+            {{- if .Networking.Calico.BPFEnabled }}
+            # Overrides for kubernetes API server host/port. Needed in BPF mode.
+            - name: KUBERNETES_SERVICE_HOST
+              valueFrom:
+                configMapKeyRef:
+                  name: calico-config
+                  key: kubernetes_service_host
+            - name: KUBERNETES_SERVICE_PORT
+              valueFrom:
+                configMapKeyRef:
+                  name: calico-config
+                  key: kubernetes_service_port
+            {{- end }}
             # Name of the CNI config file to create.
             - name: CNI_CONF_NAME
               value: "10-calico.conflist"
@@ -3836,6 +3870,19 @@ spec:
               name: kubernetes-services-endpoint
               optional: true
           env:
+            {{- if .Networking.Calico.BPFEnabled }}
+            # Overrides for kubernetes API server host/port. Needed in BPF mode.
+            - name: KUBERNETES_SERVICE_HOST
+              valueFrom:
+                configMapKeyRef:
+                  name: calico-config
+                  key: kubernetes_service_host
+            - name: KUBERNETES_SERVICE_PORT
+              valueFrom:
+                configMapKeyRef:
+                  name: calico-config
+                  key: kubernetes_service_port
+            {{- end }}
             # Use Kubernetes API as the backing datastore.
             - name: DATASTORE_TYPE
               value: "kubernetes"
@@ -3916,6 +3963,18 @@ spec:
               value: "true"
 
             # kops additions
+            # Enable source/destination checks for AWS
+            - name: FELIX_AWSSRCDSTCHECK
+              value: "{{- if and (eq .CloudProvider "aws") (.Networking.Calico.CrossSubnet) -}}Disable{{- else -}} {{- or .Networking.Calico.AwsSrcDstCheck "DoNothing" -}} {{- end -}}"
+            # Enable eBPF dataplane mode
+            - name: FELIX_BPFENABLED
+              value: "{{ .Networking.Calico.BPFEnabled }}"
+            # Controls how traffic from outside the cluster to NodePorts and ClusterIPs is handled
+            - name: FELIX_BPFEXTERNALSERVICEMODE
+              value: "{{- or .Networking.Calico.BPFExternalServiceMode "Tunnel" }}"
+            # Controls the log level used by the BPF programs
+            - name: FELIX_BPFLOGLEVEL
+              value: "{{- or .Networking.Calico.BPFLogLevel "Off" }}"
             # Controls whether Felix inserts rules to the top of iptables chains, or appends to the bottom
             - name: FELIX_CHAININSERTMODE
               value: "{{- or .Networking.Calico.ChainInsertMode "insert" }}"
@@ -3934,9 +3993,6 @@ spec:
             # Enable Prometheus process metrics collection
             - name: FELIX_PROMETHEUSPROCESSMETRICSENABLED
               value: "{{- or .Networking.Calico.PrometheusProcessMetricsEnabled "true" }}"
-            # Enable / Disable source/destination checks in AWS
-            - name: FELIX_AWSSRCDSTCHECK
-              value: "{{- if and (eq .CloudProvider "aws") (.Networking.Calico.CrossSubnet) -}}Disable{{- else -}} {{- or .Networking.Calico.AwsSrcDstCheck "DoNothing" -}} {{- end -}}"
             # Enable WireGuard encryption for all on-the-wire pod-to-pod traffic
             - name: FELIX_WIREGUARDENABLED
               value: "{{ .Networking.Calico.WireguardEnabled }}"