Automate imagePullSecrets for private images for flux

[samiyaakhtar]

Currently, when we use a private image for Project Jackson, we see that flux is able to deploy the clusters, but since it doesn't have access to the image itself, it can't poll changes to the image. This step is necessary in order to automate ACR image triggers for the gitops pipeline.

• Deploy registry secrets for flux to have access to the ACR using an SP. We can reuse the SP that is being currently used. Given this PR, we should be able to configure flux to use the SP directly
• Provide guidance or SP creation command which includes ACR access as part of the SP. Since the terraform scripts don't create the SP for the user, it's not possible to grant access to the ACR after an SP has been created.

[timfpark]

@jmspring We should look at automating granting access to ACR (or at least documenting how to do it) such that the cluster has access when it spins up.

[jmspring]

@timfpark - this link describes options. https://github.com/MicrosoftDocs/azure-docs/blob/master/articles/container-registry/container-registry-auth-aks.md

[andrebriggs]

@samiyaakhtar do the instructions here help to unblock? Should be the same as what @jmspring linked.

[samiyaakhtar]

Yes, although this issue is related to flux having access to ACR, which is separate from AKS having access

[samiyaakhtar]

From more testing and quick discussion, we need to automate these on top of what we already have:

1. Deploy registry secrets for flux to have access to the ACR using an SP. We can reuse the SP that is being currently used. (Flux uses imagePullSecrets to access the ACR and will not have access to it unless this secret is deployed, even if AKS has access. Without the secret, it's able to deploy the manifest but can't poll changes from registry)
2. Give this SP access to the ACR, which can be used by the AKS cluster too (and the above secret, which grants flux access to ACR)

[dtzar]

Second checkbox should be a new item on the backlog :)

[samiyaakhtar]

@dtzar Done #255