Deploy Istio to the utility cluster

[upodroid]

This PR does a few things:

We are using Istio in the Utility cluster because:

1. We need auth, grpc(required for multicluster observability) loadbalancing which I can't easily do with GCE Ingress
2. We deploy an oauth2 proxy and secure the argocd UI. Its now available for viewing over the internet for specific users.
3. We'll be deploying Atlantis next for terraform automation and that needs public endpoints with traffic restricted to GitHub webhooks

Fixed a bug in istio/istio#53245

[BenTheElder]

We need auth, grpc(required for multicluster observability) loadbalancing which I can't easily do with GCE Ingress

Can you elaborate on the grpc bit for future reference? What exactly are we doing here?

We deploy an oauth2 proxy and secure the argocd UI. Its now available for viewing over the internet for specific users.

kubectl proxy ? /shrug

Somewhat concerned about keeping this patched.
We could use the managed addon?

[ameukam]

We'll be deploying Atlantis next for terraform automation and that needs public endpoints with traffic restricted to GitHub webhooks

Did we got consensus on this ? we explored this possibility but I don't think we made a decision to rollout Atlantis or did I miss it ?

[upodroid]

Can you elaborate on the grpc bit for future reference? What exactly are we doing here?

https://medium.com/@yakuphanbilgic3/thanos-for-multi-cluster-monitoring-in-aws-eks-clusters-using-s3-storage-09c684e05e2b Thanos with multiple clusters. We need a unified grafana observability stack with all the metrics from all the K8s clusters and Managed Prometheus will be insanely expensive.

Somewhat concerned about keeping this patched.

All of the programs involved are go programs and much easier than nginx. Its also one of the reasons why I kept the main prow cluster stack simple with minimal external components. There are alot of powerful programs available that can make our lives easier.

Did we got consensus on this ? we explored this possibility but I don't think we made a decision to rollout Atlantis or did I miss it ?

I'm planning on starting the exploration for that. FYI, atlantis is the Terraform PR automation that we'll be configuring on this repo. https://www.runatlantis.io/

[BenTheElder]

All of the programs involved are go programs and much easier than nginx. Its also one of the reasons why I kept the main prow cluster stack simple with minimal external components. There are alot of powerful programs available that can make our lives easier.

nginx is one binary and a couple libraries with a very stable API though, so I'm not sure that's a winning comparison.

I'm not current on the state of Istio, I last used it a few years ago and it was a pretty large install and non trivial overhead that seemed aimed at use cases we haven't really needed here (like mtls)

I trust that you've thought about these though :-)

I'm planning on starting the exploration for that. FYI, atlantis is the Terraform PR automation that we'll be configuring on this repo. https://www.runatlantis.io/

I think Arnaud is objecting to this phrasing because it sounds like we've made a collective decision but we haven't spent much time on this yet. Versus maybe not necessarily disagreeing that this is something we should do. At least, that's how I read it.

[BenTheElder]

Also nginx is only sitting in front of some of our pods and despite on breaking changes to worry about we have not done a good job keeping it patched so ... that's actually why I'm a little concerned about deploying this versus using the cluster provided gateway implementation and doing metrics collection with an agent. More to keep updated, but we don't have a lot of ops bandwidth.

[BenTheElder]

/lgtm
/approve
/hold

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: BenTheElder, upodroid

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• kubernetes/OWNERS [BenTheElder,upodroid]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[upodroid]

/hold cancel