Add retries for issuing initial serving cert

[SgtCoDFish]

This enables istio-csr to be installed concurrently with cert-manager and to continue trying to operate until the initial serving cert is available

[wallrj]

I haven't tested this, but it looks like it should allow the istio-csr process to:

1. first wait for a ConfigMap to exist
2. wait for the ConfigMap to contain all of: issuer-name, issuer-kind, and issuer-group
3. create cert-manager CertificateRequest resources (with the given issuer-ref) until one of them gets signed.
4. Give up and exit with an error if this process is performed more than 250 times.

I don't really understand why the serving certificate for the istio-csr gRPC service has to use the same cert-manager issuer is used for the CRs it generates for the istio workload certificates. In fact I would expect the serving certificate for the gRPC server to be signed by a different (platform) CA than the mTLS certificates of the workloads running on the platform.

I'd have thought that the istio-csr Pod could just mount a Secret containing the serving certificate and have that Secret populated by a cert-manager Certificate, as and when cert-manager gets installed.
I suppose this would cause the Istio-csr helm deployment with --wait to fail...but nevertheless that would be a simple and easily understandable process for installing istio-csr in parallel with cert-manager.

I just don't know enough about how istio-csr works. Don't want to hold you up though so I'll add an lgtm and we can do thorough testing when you release the alpha.

/approve
/lgtm
/hold incase there's anything else to be changed.

[cert-manager-prow]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: wallrj

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [wallrj]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[SgtCoDFish]

/unhold

Thanks for the review Richard! I'm going to unhold and merge so I can get an alpha release out to test this more.

I don't really understand why the serving certificate for the istio-csr gRPC service has to use the same cert-manager issuer is used for the CRs it generates for the istio workload certificates. In fact I would expect the serving certificate for the gRPC server to be signed by a different (platform) CA than the mTLS certificates of the workloads running on the platform.

You're definitely onto something here. I think istio-csr does conflate a few things together with issuance, and there might be value in revisiting that. I think people with more istio experience than me could helpfully chime in.

For now, it seems like the current setup works - but I do think this deserves more thought!