Support cluster-local Gateways

[evankanderson]

Currently, the GatewayAddress field supports only external (LoadBalanced) Gateway definitions. I'd like to suggest adding a ClusterLocal or ClusterIP AddressType with Extended support (like the existing IPAddress and Hostname support).

One of the early feature requests for Knative was the ability to deploy an application using Knative's HTTP routing support, but make it only available within the cluster. I want to be able to specify both the "internal" (service.namespace.svc) and "external" (service.namespace.example.com) Gateways using the same GatewayClass on the cluster, but ensure that the "internal" service is only reachable within the cluster. This would greatly simplify deployment for users over the instructions we have today.

[evankanderson]

I see a bunch of ⬆️ responses -- would this one be better as an issue if we agree that we should add this to the spec?

[robscott]

I think a discussion is good for this, just need more comments to work through what this would look like. I've added some questions in a separate comment.

[robscott]

I agree that there's broad interest in being able to request a cluster-local Gateway. I'm less sure what the best mechanism for this is. Some questions:

1. Could we solve network-local Gateways with the same approach as cluster-local Gateways? Maybe a different value for the same config?
2. Is an address type the best way to configure this? It seems like address type is somewhat orthogonal to what the request is here? Currently when addresses are specified in spec, it's a way of requesting a specific address, not just an address type. Generally we've assumed that most implementations support a single kind of address or a small subset. Maybe instead we should be looking at some kind of net new field that describes the scope of the requested Gateway. I think @shaneutt has suggested something similar in the past.

[evankanderson]

My thought was that you would request a cluster-local IP as:

addresses:
- type: ClusterIP

Or

addresses:
- type: ClusterIP
  value: 10.96.11.72

If you wanted to request a specific value.

I'm fine with a different implementation, though.

[youngnick]

I like this idea, because it means that we can use the type field with a domain-prefixed string to allow people to try things out, and the only change that would need to be made is making the behavior clear when you only set a type. I don't think it's too big a change from what we have today.

[shaneutt]

As Rob suggested, I have a similar use case floating around which I brought up in the community sync a few weeks back: we would potentially want the ability to differentiate on the addresses such that we could programmatically decide which network (the cluster network in the most common case) we want to communicate with the Gateway on. +1 from me, I would love to see a GEP to this effect.

[brianehlert]

If I am following:

Is this about associating "Policy" to a Service (what we can also think of as API Gateway type capabilities that meshes implement), and then it is where that Policy is implemented? (edge vs sidecar for example)
And trying to force that into a model of the current Gateway API?
I have had this concept of a Policy that has an application point associated to it. The references to this policy are where it is applied, edge, sidecar, both.

To me the question comes; how do you define the representation of the internal application of the policy? Since you don't have to implement a Mesh using sidecars.

Simply, this service has this Policy applied and it is up to the implementation to ensure that the Policy is applied.

If I am not following the concept but try to interpret this literally, this seems to define a gateway that represents the service endpoint and thus dictate not just an implementation but almost the topology of the implementation.

[youngnick]

It's not originally about mesh usecases at all, the Knative folks need a ClusterIP Gateway basically for hairpinning traffic. But the mesh use cases have defiintely made this much more complex a design than if we were only considering north-south traffic.

[mikemorris]

how do you define the representation of the internal application of the policy? Since you don't have to implement a Mesh using sidecars.

For the use case you're describing (which as @youngnick mentioned is different than the original intent of this discussion), I've envisioned using the Policy Attachment model to either target a Service directly, similar to how HTTPRoute targets Service for GAMMA E/W routing, or for other use cases, to do something like #1493, targeting a namespace (or perhaps optionally a Gateway of spec.type=egress if supported by an implementation) for consumer policy or egress routing, to support multiple models and allow implementations to enforce via sidecar where applicable, but without encoding the assumption of a sidecar into the spec.

[evankanderson]

Hey, there was some interesting discussion about this, but I don't see that it went anywhere (despite being up-voted).

We have some practical use cases in Knative for this: https://knative.dev/docs/serving/services/private-services/

(The annotation isn't 100% awesome, but the general functionality has been widely requested -- I think it was the first or second customer feature request during the Knative private beta before the project was publicly announced.)

CC'ing @dprotaso , who I'm hoping can drive the discussion to feature and resolution.

[dprotaso]

I'm going to capture some of this discussion in a GEP - I've created the corresponding issue here: #1651