Add k8s annotations to the Envoy Proxy Service

[arkodg]

Description:
Extend the Envoy Proxy API to allow a user to add more k8s annotations to the Envoy Proxy service.

Relates to #368 (comment)

[optional Relevant Links:]

Any extra documentation required to understand the issue.

[Xunzhuo]

assign, quite busy recently, will work on this tomorrow.

[danehans]

@Xunzhuo thanks for offering to help with this issue. Since this is a "Backlog" milestone issue, I suggest working on RC-2 issues since they're more pressing.

[danehans]

Relates to #368 (comment)

IMHO this issue is very different from #368. #368 was required because EG needs a mechanism to correlate a Gateway with its managed infra resources, e.g. Service. If the managed resources were in the same ns as the Gateway, an ownerReference could be used and the owning ns/name label would be unneeded.

I'm not sure if passing annotations to the Service is the best approach for solving this problem. I think it can be error-prone as opposed to creating an API. For example, what if LoadBalancer annotations are specified for an internal (ClusterIP) Service or GCP-specific LoadBalancer annotations are specified when EG runs in AWS?

Thoughts @LukeShu @skriss @youngnick @AliceProxy @Xunzhuo?

[ericgoode]

I have a slightly different use case. I would like to have the Gateway annotate its LoadBalancer service in order to register with external-dns when the gateway is created. Right now I can manually annotate the service after it is created, but then any time a change is made to the Gateway the annotation is removed from the Service.

So there is two ways that this can be approached. Either A) The gateway manages the annotations that are put onto the Service object or B) the Gateway leaves any existing annotations in place when making changes to the Gateway.

"kubectl annotate service envoy-default-eg-64656661 -n envoy-gateway-system "external-dns.alpha.kubernetes.io/hostname=myhostname.com" will annotate the service and external-dns will see this and register the DNS name with the provider.

[danehans]

@ericgoode External DNS supports Gateway API, so I would expect it to work natively with EG. Have you tested the integration yet?

xref: https://github.com/kubernetes-sigs/external-dns/blob/master/docs/tutorials/gateway-api.md

[ericgoode]

@ericgoode External DNS supports Gateway API, so I would expect it to work natively with EG. Have you tested the integration yet?

xref: https://github.com/kubernetes-sigs/external-dns/blob/master/docs/tutorials/gateway-api.md

Thank you for pointing this out. I have tested it and it works. My use case listed above for this issue is void.

[arkodg]

We currently have infrastructure defined in the GatewayClass level tied to the EnvoyProxy CRD using paramsRef https://gateway.envoyproxy.io/v0.3.0/api/config_types.html#kubernetesdeploymentspec
So an annotations knob within a services field makes sense to me.
cross linking with kubernetes-sigs/gateway-api#1653 (comment) where @youngnick highlighted that a notion of infrastructure could be added to the Gateway resource.
Prefer if we started off adding to the EnvoyProxy CRD and later incorporated the upstream proposal