Prow: Migrate to ExternalSecrets

[lentzi90]

Our current setup requires admins to create files with credentials and other secrets locally in the kustomizations before applying changes. This process is error prone and makes automation hard since an automation tool would also need to have access to all secrets even when only touching non-sensitive parts.

We should migrate to ExternalSecrets instead. This is the same that is used for k/k prow. It has integration with OpenStack so we should be able to store the secrets there. In practice what we need to do is to remove the secrets from the kustomizations and introduce ExternalSecrets instead. The ExternalSecrets are just references to secrets stored in the external storage (openstack for us). So they can be committed in git. Admins would then need to make sure the secrets are available in openstack before attempting a deployment.

[lentzi90]

/triage accepted

[tuminoid]

This is a stepping stone for automating Prow cluster updates via GitOps 👍