Add Support for Managing the Envoy Proxy Service

[danehans]

As of v0.2.0, the Kube Infra Manager manages a Service of type LoadBalancer to perform L4 load-balancing across the managed Envoy proxy fleet. Some users may wish to use Envoy for internal-only proxying, configuring various load-balancer settings, etc. Envoy Gateway provides the EnvoyProxy API type for configuring the Envoy proxy infrastructure. This API should be expanded to support common Service configuration requirements.

[bmetzdorf]

Thanks @danehans. I'm specifically interested in the integration with 3rd party cloud providers, beginning with AWS and their NLB. Something like these annotations will be needed on the Service:

service.beta.kubernetes.io/aws-load-balancer-backend-protocol: tcp
service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: "ip"
service.beta.kubernetes.io/aws-load-balancer-type: "external"

Since Envoy Gateway will handle TLS and HTTP(S) itself, we just need TCP and UDP traffic forwarded by the NLB. And in order to not pay the node port double hop penalty, we could use NLB IP based targeting and use Proxy Protocol to retain client IP information (UDP/QUIC TBD).

[arkodg]

Thanks @danehans. I'm specifically interested in the integration with 3rd party cloud providers, beginning with AWS and their NLB. Something like these annotations will be needed on the Service:

service.beta.kubernetes.io/aws-load-balancer-backend-protocol: tcp
service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: "ip"
service.beta.kubernetes.io/aws-load-balancer-type: "external"

Since Envoy Gateway will handle TLS and HTTP(S) itself, we just need TCP and UDP traffic forwarded by the NLB. And in order to not pay the node port double hop penalty, we could use NLB IP based targeting and use Proxy Protocol to retain client IP information (UDP/QUIC TBD).

@bmetzdorf this should be addressed with #377

[danehans]

Currently, an ELB gets created for the Envoy service when running EG in AWS. We should probably set service.beta.kubernetes.io/aws-load-balancer-backend-protocol: tcp to put the ELB in TCP mode and service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: "*" to enable proxy protocol by default (when running in AWS). Or should we default to NLB with the annotations specified by @bmetzdorf? As I mentioned in #377, I prefer creating an API for managing the Envoy network endpoints instead of passing service annotations. @LukeShu @skriss @youngnick @AliceProxy @Xunzhuo thoughts?

[danehans]

I've created a design doc to help faciliate discussion for supporting this use case. @bmetzdorf @arkodg @youngnick @skriss @LukeShu @AliceProxy @Xunzhuo PTAL.

[arkodg]

I've created a design doc to help faciliate discussion for supporting this use case. @bmetzdorf @arkodg @youngnick @skriss @LukeShu @AliceProxy @Xunzhuo PTAL.

@danehans took a look at this review, was unable to leave comments in there due to limited permissions, so leaving them here - the doc proposes creating high level load balancer providers such as aws

• I worry such level of vendor specificity will trigger the project supporting and maintaining many
  such loadbalancer types/providers and a lot of time might get spent managing such driver code
• taking the aws case, there are many user inputted annotations such as service.beta.kubernetes.io/load-balancer-source-ranges that EG will not be able to compute beforehand

[danehans]

took a look at this review, was unable to leave comments in there due to limited permissions

Anyone should now be able to comment in the doc.

I worry such level of vendor specificity will trigger the project supporting and maintaining many
such loadbalancer types/providers and a lot of time might get spent managing such driver code

It does require more maintenance but provides a more consistent UX. Otherwise, users can freely pass annotations that may be invalid. We could validate the annotations but that can turn into a difficult matrix to support.

taking the aws case, there are many user inputted annotations such as service.beta.kubernetes.io/load-balancer-source-ranges that EG will not be able to compute beforehand

EG can perform some validation for this use case. For example, if a user specifies an invalid range, e.g. 1.2.3.4/34. In this instance, EG would not apply the config and surface an EnvoyProxy status condition that states the provided CIDR is invalid.

[danehans]

@bmetzdorf do you have any thoughts or preferences on the approach to resolving this issue?

[bmetzdorf]

Hi @danehans and @arkodg,

I would be leaning a bit towards not managing and validating these vendor specific configurations. There's a good argument to be made that having an API for attributes of vendor specific LoadBalancer types is technically the cleaner approach, but there's also a very good argument to be made that this could (and probably would) lead to proliferation and a lot of managing effort for maybe not so much gain. If a vendor decided to support a new annotation, then envoy gateway users would have to wait for a new release if we did not allow passing annotations through.

I can imagine (although I don't know) that the original designers of the K8s API (in general but also specifically for Ingress and Service) probably faced the same question: Should we include every configuration possible from every vendor as a first class citizen in the API and maintain and version all of them or should we use annotations as a generic way instead?

[youngnick]

I think there's a tradeoff here that we should be explicit about.

As @bmetzdorf and @arkodg noted above, having fully-fledged fields in a CRD has the disadvantage that adding support for new features requires an EG release. The advantage of having fully fledged fields is that EG can validate the information or use it to trigger other behavior. Blindly copying annotations around places a support burden on users.

On the other hand, having a map[string]string of annotations to add to loadbalancer services has a couple of downsides:

• It becomes difficult to check for annotations that may interact poorly with other EG features
• It ties the implementations strongly to LB services. What happens if, in the future, you can create a Gateway with a TCP route to replicate the functionality of a cloud-provider LB service? We have no insight into what the fields are, so we are unable to help.

The advantages here are the speed of development and the level of control given to users.

I don't have strong feelings either way here, except for the note that annotations are very sticky. Once people are using them, we will never be able to remove them. They're forever.

[arkodg]

@youngnick EG would provide an API for the user to append an annotation to the managed Envoy Proxy service annotation field, so EG is not inferring any meaning from the actual value of the annotation.
The only thing we need to be careful about here, is to ensure user cannot overwrite an EG generated annotations (annotations is one example, also can apply to other non scalar fields like labels)