New TrafficRoute design

[lahabana]

Description

With new matching.

Some open questions:

• Can we collapse MeshGatewayRoute and trafficRoute?
• Should this include localityAwareLB?

Easy to include features:

• Support absence of header via present_match in TrafficRoute #4691
• Support full TrafficRoute functionality for MeshGatewayRoute #4690
• Add policy configuration for Fallback in TrafficRoute policy #3338

Related issues

• Expose config.cluster.v3.Cluster.LeastRequestLbConfig.active_request_bias in TrafficRoute policy #2422
• Introduce MeshGRPCRoute policy #3325
• Proper support for locality weighted load balancing #2689
• Design MeshTLS policy #4622
• Add localityAwareLoadBalancing to TrafficRoute #3456
• Expose hash_policy in new TrafficRoute #5309

[lahabana]

We could go with something very similar to GatewayAPI routes because it handles well filters, destinations, split and mirrors.
If that's a the case we'd have different policies per protocol (probably fine).

https://gateway-api.sigs.k8s.io/references/spec/#gateway.networking.k8s.io/v1alpha2.HTTPRoute

Some open questions:

• How do you set LoadBalancer config in the route (a filter would be weird)? Maybe that's something for gatewayAPI improvement?
• Same question with LocalityAwareLB.

[michaelbeaumont]

This is a summary of the current status and open questions.

• MeshTrafficRoute refers to a new, targetRef-based TrafficRoute.

MeshTrafficRoute

We can imagine an evolution of TrafficRoute using targetRef:

spec:
  targetRef: ... # from where requests are coming
  to:
    - targetRef: ... # to where requests are going
  default: ...

Routes are implemented in the client-side config so we would use spec.targetRef for the request source plus spec.to, as opposed to spec.from/spec.targetRef.

MeshGatewayRoute

Conceptually, a MeshGateway proxy is a service like any other that sits at the edge of our mesh.

The use cases for routing and modifying requests coming from MeshGateways to services and services to services are presumably the same, so we can certainly come up with a spec that covers both and MeshTrafficRoute could eventually subsume MeshGatewayRoute.

We can target a MeshGateway using a MeshTrafficRoute:

spec:
  targetRef:
    kind: MeshService
    name: # the configured kuma.io/service of the MeshGateway
    # or 
    kind: MeshGateway
    name: edge-gateway  
  to:
    - targetRef: ...
  default: ...

Gateway API's HTTPRoute

The Gamma initiative is the working group looking at Gateway API for mesh use cases. If we want to support using upstream HTTPRoute for mesh communication we should try to be conformant with the Gamma spec.

Request/response modification

All request/response modification via HTTPRoute is done with HTTPRoute.spec.rules[].filters.

HTTPRoute includes some common filters like header and path modification.

Any Kuma-specific filter can be configured by setting HTTPRouteFilterType to ExtensionRef and pointing extensionRef to a Kuma resource. This would allow us to support any arbitrary modifications using Gateway API.

parentRef/service targeting

Gamma proposes using an HTTPRoute parentRef of Service to configure routing for requests sent to that Service.

Unfortunately this presents a fundamental incompatibility at the moment with how we would implement MeshTrafficRoute as a targetRef policy. The incompatibility is that there's no way to specify a reference from which Kuma service a request should come for an HTTPRoute. I.e., HTTPRoute is not as powerful as MeshTrafficRoute in terms of targeting the source of a request.

There is currently an open PR for using the Namespace of the HTTPRoute for this purpose. Still, it wouldn't be as powerful as a targetRef-based policy.

Other config

HTTPRoute provides only matches, filters and backendRefs. There is also backendRefs.filters for backend-specific filtering.

There is no other extension point for HTTPRoute than filters so any additional configuration, such as load balancer options must appear under filters.

The name filters is perhaps confusing for something like load balancing configuration but with ExtensionRef it is powerful, flexible and extensible enough to support any arbitrary routing configuration. It doesn't really assume anything about the referenced resource.

It would allow us to apply load balancing for the entire route, for a subset of requests with matches as well as for a subset of backendRefs. For example, to use round robin for all requests.

spec:
  ...
  rules:
    - matches: []
      filters:
        - type: ExtensionRef
          extensionRef:
            group: kuma.io/v1alpha1
            kind: RouteConfig
            name: load-balancer-common
            # potential inlined load-balancer-common:
            # spec:
            #   loadBalancer: RoundRobin
      backendRefs: []