Provide service loadbalancing function to Pods via openflow in antrea-agent

[tnqn]

Describe what you are trying to solve
Given recent code change and some ongoing designs, I want to propose we improve the priority of implementing kube-proxy via openflow in antrea, reasons as below:

• NoEncap and passthrough mode are hacked and complex because of the dependency of K8s kube-proxy.
  Currently only K8s kube-proxy can do DNAT for service traffic, so packets must be sent to host network and DNATed, then be sent back to OVS to go through the egress network policies, and finally sent to target Pods.
  In encap mode, the last step is simple as the packets will be sent to a tunnel port directly.
  In noEncap and passthrough mode, the last step is sending the packets back to host network again, and because the same packet reenters the host network but needs another routing decision, various mechanism are introduced to make it work:

  1. The first packet was marked in OVS
  2. Routing policy is introduced to match the above mark, select a user-define route table, and route the DNATed packet back to OVS
  3. The src mac of the egress filtered packet is set to a special value in OVS before sending back to host network again
  4. A special src mac check is added to IPtables raw table to skip going through conntrack for the reentry packets as it’s not valid to see same packet that was sent out to a device back to it. See more details at https://github.com/suwang48404/antrea/blob/cloud/docs/pass-through.md#handling-pod-to-service.

• For traceflow work, it will not be as simple as the packet to service will leave OVS midway and then come back, how to trace the packet even on same host will be a problem, there might be various workaround to make it work but may not worth.

• For windows platform, the encap mode may continue work as k8s kube-proxy can work with OVS pipeline in the same way as linux platform. However, for noEncap mode and passthrough mode, the above mechanism like routing policy, iptables raw table bypassing conntrack may not be available on windows. Even there are, the efforts of making them work may be more than implementing kube-proxy via openflow.

• For performance and implementation consistency consideration, adding a DNAT table in the OVS pipeline is clean and straightforward, the service traffic will only be processed once in OVS, instead of two times in OVS and one time in iptables.

Describe the solution you have in mind
antrea-agent watches K8s Service and Endpoints/EndpointSlice and implements DNAT function in DNAT table of OVS pipeline.

Describe how your solution impacts user flows
Higher throughput when accessing service should be expected.

Describe the main design/architecture of your solution
@wenyingd @weiqiangt can add more detail design.

Alternative solutions that you considered
None

Test plan

• It should pass K8s conformance tests for services.

Additional context
Task LIst:

• Add enhancements in OpenFlow control interface to support NAT actions. Support NAT functions in Antrea #489
• Add support for Group entry in OpenFlow control interface.
• Watch Service and Endpoint on agent and invoke OpenFlow client to realize Service LoadBalancing.

[weiqiangt]

Design document

[weiqiangt]

Here is the slides used in Community Meeting: https://drive.google.com/file/d/1IL6zdDYZuQZklBOz1MKKZifmbi7XWTXL/view?usp=sharing