Add support for Azure Workload Identity

[pjbgf]

Azure Workload Identity is the next generation of workload contextual authentication, which replaces the existing Azure Pod Identity.

The features in which Flux has Azure Workload Identity supported and documented:

• OCI Repository Authentication
• Bucket Repository Authentication
• Helm OCI Repository Authentication
• Secrets decryption with Azure KeyVault
• Container Registry Authentication

Relates to #3003.

[weisdd]

@pjbgf If FluxCD team accepts the usage of azidentity v1.3.0-beta.2, then we can quickly add support for Workload Identity to several controllers by bumping deps:

• Support for workload identity was added to github.com/Azure/azure-sdk-for-go/sdk/azidentity in v1.3.0-beta.1 (Support workload identity federation Azure/azure-sdk-for-go#15615 (comment)). The latest version at the time of writing is v1.3.0-beta.2.
• Even though the "stable" version is expected to be released around April after the Workload Identity extension is promoted to GA (Support of Azure workload identity? Azure/azure-sdk-for-go#19765 (comment)), the current one is already functional.
• The support for Workload Identity is available through WorkloadIdentityCredential and DefaultAzureCredential.
• In FluxCD source-controller, helm repository controller relies on fluxcd/pkg/oci package to authenticate against ACR when provider is set to azure. The package relies on DefaultAzureCredential. So, to add support for Workload Identity there, it's enough to bump dependencies (like here). I did a PoC in my sandbox, and it worked just fine (you can find logs below).
• Although I haven't tested it yet, I think some other controllers (like OCI repository controller, image reflector controller) would work as well. Bucket controller has its own logic for authentication, so it'll require some adjustments, but I guess we don't have to modify all controllers at once :)

Please, let me know what you think. If you would accept such change, I can prepare a PR with all the instructions and a terraform test lab.

Logs

source-controller-57774ccfc5-64bvd manager {"level":"info","ts":"2023-01-14T23:33:34.841Z","msg":"logging in to Azure ACR for fluxcdtestacr123.azurecr.io/charts","controller":"helmrepository","controllerGroup":"source.toolkit.fluxcd.io","controllerKind":"HelmRepository","HelmRepository":{"name":"charts","namespace":"flux-system"},"namespace":"flux-system","name":"charts","reconcileID":"8c61bb40-7a54-40a4-b7b0-e3a1fbe9b875"} source-controller-57774ccfc5-64bvd manager {"level":"info","ts":"2023-01-14T23:33:35.400Z","msg":"Helm repository is ready","controller":"helmrepository","controllerGroup":"source.toolkit.fluxcd.io","controllerKind":"HelmRepository","HelmRepository":{"name":"charts","namespace":"flux-system"},"namespace":"flux-system","name":"charts","reconcileID":"8c61bb40-7a54-40a4-b7b0-e3a1fbe9b875"} source-controller-57774ccfc5-64bvd manager {"level":"info","ts":"2023-01-14T23:38:35.444Z","msg":"logging in to Azure ACR for fluxcdtestacr123.azurecr.io/charts","controller":"helmrepository","controllerGroup":"source.toolkit.fluxcd.io","controllerKind":"HelmRepository","HelmRepository":{"name":"charts","namespace":"flux-system"},"namespace":"flux-system","name":"charts","reconcileID":"bb59958b-0c44-4b35-871f-f7679c37d097"} source-controller-57774ccfc5-64bvd manager {"level":"info","ts":"2023-01-14T23:39:01.314Z","msg":"logging in to Azure ACR for fluxcdtestacr123.azurecr.io/charts","controller":"helmchart","controllerGroup":"source.toolkit.fluxcd.io","controllerKind":"HelmChart","HelmChart":{"name":"default-podinfo","namespace":"flux-system"},"namespace":"flux-system","name":"default-podinfo","reconcileID":"a0faa635-370a-4ad8-bb37-e60e16f3607c"} source-controller-57774ccfc5-64bvd manager {"level":"info","ts":"2023-01-14T23:39:02.021Z","msg":"pulled 'base' chart with version '0.3.2'","controller":"helmchart","controllerGroup":"source.toolkit.fluxcd.io","controllerKind":"HelmChart","HelmChart":{"name":"default-podinfo","namespace":"flux-system"},"namespace":"flux-system","name":"default-podinfo","reconcileID":"a0faa635-370a-4ad8-bb37-e60e16f3607c"}

[somtochiama]

Workload identity has been added in each of the relevant controllers: See: fluxcd/kustomize-controller#813, fluxcd/source-controller#1048, fluxcd/image-reflector-controller#363.

[Poltergeisen]

How can we use this for Azure devops git repositories?

[hatfarm]

@Poltergeisen I was wondering the same thing! Have you figured it out? I don't think there's a way, currently, to do it. I've thought about putting together a PR, but am not familiar with Go (I AM pretty familiar with the AZDO method of hooking this up).

[aryan9600]

Hello, support for Azure Devops Git repositories is planned and being worked on.

[hatfarm]

@aryan9600 Is this something I can help with? Anything I can follow a little closer? I would love to get involved, but mostly want to be aware of when this might be landing.

[andywilde]

Hi, I'm a bit confused over the Azure documentation for this feature. I'm trying to migrate from pod-identity to workload-identity for image automation with ACR and I've implemented the patches as suggested here Workload Indentity for image-reflector-controller . I then see in Configure image automation authentication that AAD Pod Identity is a prerequisite? Is the latter documentation out of date?

[somtochiama]

Hey @andywilde ,

That documentation is for using cronjobs to generate short-lived credentials

While native authentication mechanisms are available, using a cron job is the preferred way of syncing image repository credentials for multi-tenancy as the controller cannot natively get access to the image repository.

If you have set up workload identity on your aks cluster and have labelled the image-reflector-controller and pod correctly, workload identity should work. The only extra you need is .spec.provider: azure in your ImageRepository manifest.
See more at: https://fluxcd.io/flux/components/image/imagerepositories/#provider

[andywilde]

Hi @somtochiama, do you have an example of ImageRepository? With this code:

apiVersion: image.toolkit.fluxcd.io/v1beta1
kind: ImageRepository
metadata:
  name: podinfo
  namespace: flux-system
spec:
  image: xxx.azurecr.io/stefanprodan/podinfo
  interval: 5m0s
  provider: azure

I get the error .spec.provider: field not declared in schema. Does API version require an update?

[somtochiama]

yes, what version of flux are you on?

[andywilde]

2.0.0

[somtochiama]

yes, please upgrade to v1beta2

[andywilde]

I guess this errors mean the authentication is fixed but I have to configure something else?

{"level":"error","ts":"2023-10-13T16:17:05.213Z","msg":"referenced ImageRepository has not been scanned yet","controller":"imagepolicy","controllerGroup":"image.toolkit.fluxcd.io","controllerKind":"ImagePolicy","ImagePolicy":{"name":"podinfo","namespace":"flux-system"},"namespace":"flux-system","name":"podinfo","reconcileID":"8051531b-7e30-4c65-a653-5fff7df48e54","error":"DependencyNotReady"}

[somtochiama]

what do you see when you run flux get image repository?

[andywilde]

The error was No matching federated identity record found for presented assertion so I created federated credential on the MI mapped to system:serviceaccount:flux-system:image-reflector-controller and that seemed to do the trick. Output from flux get image repository is now:

NAME    LAST SCAN               SUSPENDED       READY   MESSAGE
podinfo 2023-10-13T16:45:12Z    False           True    successful scan: found 4 tags

With this working and based on the images in our ACR:

I was expecting the image in the HR release to be updated in the git repo from

 image:
      repository: xxx.azurecr.io/stefanprodan/podinfo
      tag: 6.4.0
      pullPolicy: Always

to 6.5.0

I guess this means I need to configure something else?

[somtochiama]

Do you have image automation set up? https://fluxcd.io/flux/guides/image-update/

[andywilde]

It's working now. The problem was I hadn't added the policy marker:

tag: 6.4.0 # {"$imagepolicy": "flux-system:podinfo:tag"}

@somtochiama thanks for all your help. Have a great weekend!

[Poltergeisen]

Hello, support for Azure Devops Git repositories is planned and being worked on.

@aryan9600 do you happen to know how long it might take? Or is there an issue I can subscribe to that tracks that functionality?

[b-rand]

Hello, support for Azure Devops Git repositories is planned and being worked on.

@aryan9600 do you happen to know how long it might take? Or is there an issue I can subscribe to that tracks that functionality?

@aryan9600 bumping this question back up to you since this is a topic of interest for my team as well.

[aryan9600]

@Poltergeisen @b-rand hello, i was on vacation and then busy with kubecon. i have resumed work on this. the PR which enables support for this is: fluxcd/pkg#664. there's a lot of work still remaining before the new package can be used in the controllers. i'm targeting the next to next minor release of flux, i.e. flux v2.3.0