CrossMesh MeshGateway listener does not work when MeshGatewayRoute hostnames are set

[ttreptow]

What happened?

I have a MeshGateway with a listener set up with no hostname set (defaults to '*') and crossMesh enabled. I'm trying to add MeshGatewayRoutes that have hostnames specified.

When the below configuration is set, the gateway proxy throws errors like

[2023-10-19 21:15:58.269][26][warning][config] [source/extensions/config_subscription/grpc/grpc_subscription_impl.cc:128] gRPC config for type.googleapis.com/envoy.config.listener.v3.Listener rejected: Error adding/updating listener(s) vault:HTTP:8080: error adding listener '10.200.12.80:8080': filter chain '' has the same matching rules defined as ''

Looking at the generated Envoy config, I see that it is trying to set two identical filter chains for that listener.

Digging into the code, I think what is happening is that RedistributeWildcardRoutes is creating two host name configs, one for the route's host name (assuming you have one route with one hostname set) and one for the wildcard.
The problem with having multiple hosts is that crossMesh enabled sets the protocol to HTTPS with no TLS mode set, then in configureTls the switch block never sets envoy_listeners.MatchServerNames(hostnames...), as it does for other https modes.

The net effect is that the filter chains have identical matching blocks (since they are missing hostname matching) which causes the error.

MeshGateway:

apiVersion: kuma.io/v1alpha1
kind: MeshGateway
mesh: mesh1
metadata:
  name: cross-mesh
  namespace: kuma
spec:
  conf:
    listeners:
      - crossMesh: true
        port: 8080
        protocol: HTTP
        tags:
          some: tag

Note that the hostname is not set and crossMesh is true

MeshGatewayRoute:

piVersion: kuma.io/v1alpha1
kind: MeshGatewayRoute
mesh: mesh1
metadata:
  name: test
  namespace: kuma
spec:
  conf:
    http:
      hostnames:
      - myservice.external.mesh
      rules:
      - backends:
        - destination:
            kuma.io/service: test_test_svc_8000
        matches:
        - path:
            match: PREFIX
            value: /
  selectors:
  - match:
      kuma.io/service: edge-gateway
      some: tag

MeshGatewayInstance:

apiVersion: kuma.io/v1alpha1
kind: MeshGatewayInstance
metadata:
  name: edge-gateway
  namespace: gateway
spec:
  replicas: 1
  serviceType: ClusterIP
  tags:
    kuma.io/service: edge-gateway

The namespace is tagged with kuma.io/mesh: mesh1 and the gateway dataplace does seem to get the correct mesh

[michaelbeaumont]

While this is a bug because we're sending invalid configuration, it's not clear exactly how the hostnames field of a route attached to a cross mesh listener without a hostname should work.
@ttreptow How are you reaching or expecting to reach the cross-mesh gateway at this hostname? At the moment, when not setting the listener hostname, mesh DNS is configured such that the cross-mesh gateway is reachable at a single address without wildcards.

The TLS SNI of connections to the mesh gateway is a Kuma internal SNI, so we can't match on e.g. myservice.external.mesh.

[ttreptow]

I see, I think I misunderstood the purpose of the listener hostname and mesh gateway hostnames. What I'm trying to do is call a service from one mesh from a pod in another mesh using host routing.

Because MeshGateway only has one hostname, this means that (unless I'm misunderstanding) I have one vhost and have to use path routing in the MeshGatewayRoutes to select the services (or use one gateway per exposed service). Ideally, I'd like to use MeshGatewayRoutes to define exported hosts for the other mesh to call, e.g. servce1.mesh1.mesh, etc.

FWIW, I was able to get cross-mesh host routing working using the MeshGatewayRoute hostnames: ttreptow@6abf451#diff-ad441c05e85b5598449acabae1ac6a08e6dcd3017ec295b06bfa87f4633c05ed (obviously very experimental, I've only tested my specific use case so there could be unintended side-effects)

[michaelbeaumont]

@ttreptow

You definitely didn't misunderstand the purpose of the hostnames, that's how they work of course for external gateways. It's a very reasonable feature request and we could also handle adding VIPs for hostnames in routes. Feel free to open a PR with what you have!

The only issue will be matching the SNI in the Envoy config but I could potentially imagine a way around that by adding tags to the used SNI name.

EDIT: that is to say, I suspect all of your MeshGatewayRoutes will be used for all requests to the cross mesh gateway, no matter what hostnames they have in them.