feat(oci): Add support for Azure Workload Identity

[weisdd]

As I pointed out here, we can add support for Azure Workload Identity to several controllers that rely on the oci package by just bumping the azidentity library to v1.3.0-beta.2. That is possible because the oci package itself relies on NewDefaultAzureCredential in case credentials are not passed explicitly while the updated function offers support for Workload Identity:

pkg/oci/auth/azure/auth.go

Lines 65 to 69 in 946d9ac

	// Use default credentials if no token credential is provided.
	// NOTE: NewDefaultAzureCredential() performs a lot of environment lookup
	// for creating default token credential. Load it only when it's needed.
	if c.credential == nil {
	cred, err := azidentity.NewDefaultAzureCredential(nil)

The question is whether FluxCD team would accept the usage of v1.3.0-beta.2 (which is very likely to land to v1.3.0 as is) or would rather prefer to wait for several months till the Workload Identity AKS extension turns GA (as explained below, azidentity will have v1.3.0 tag only after the extension is considered GA).

This PR is meant to either have the change accepted or, if not, explicitly indicate the fact that the project is waiting for v1.3.0.

Details

• Support for workload identity was added to github.com/Azure/azure-sdk-for-go/sdk/azidentity in v1.3.0-beta.1 (Support workload identity federation Azure/azure-sdk-for-go#15615 (comment)). The latest version at the time of writing is v1.3.0-beta.2:
  • https://github.com/Azure/azure-sdk-for-go/releases/tag/sdk%2Fazidentity%2Fv1.3.0-beta.1
  • https://github.com/Azure/azure-sdk-for-go/releases/tag/sdk%2Fazidentity%2Fv1.3.0-beta.2
• Even though the "stable" version is expected to be released around April after the Workload Identity extension is promoted to GA (Support of Azure workload identity? Azure/azure-sdk-for-go#19765 (comment)), the current one is already functional.
• The support for Workload Identity is available through WorkloadIdentityCredential and DefaultAzureCredential.
• I believe the change should be safe, because Workload Identity authentication process will get triggered only in case several environment variables are available in a container with a FluxCD controller (code). Those variables get added to a pod by a Mutating Webhook only when special labels and annotations are present (docs), so there should be no difference for those users who don't have those labels/annotations on the respective k8s resources of FluxCD controllers.
• I did a PoC with the source-controller, and it worked just fine (logs are listed below). I believe that will be the case for OCI repository controller and image reflector controller as well. Bucket controller has its own logic for authentication, so it'll require some adjustments before it supports Workload Identity, but I guess we don't have to modify all controllers at once :)

Logs

source-controller-57774ccfc5-64bvd manager {"level":"info","ts":"2023-01-14T23:33:34.841Z","msg":"logging in to Azure ACR for fluxcdtestacr123.azurecr.io/charts","controller":"helmrepository","controllerGroup":"source.toolkit.fluxcd.io","controllerKind":"HelmRepository","HelmRepository":{"name":"charts","namespace":"flux-system"},"namespace":"flux-system","name":"charts","reconcileID":"8c61bb40-7a54-40a4-b7b0-e3a1fbe9b875"} source-controller-57774ccfc5-64bvd manager {"level":"info","ts":"2023-01-14T23:33:35.400Z","msg":"Helm repository is ready","controller":"helmrepository","controllerGroup":"source.toolkit.fluxcd.io","controllerKind":"HelmRepository","HelmRepository":{"name":"charts","namespace":"flux-system"},"namespace":"flux-system","name":"charts","reconcileID":"8c61bb40-7a54-40a4-b7b0-e3a1fbe9b875"} source-controller-57774ccfc5-64bvd manager {"level":"info","ts":"2023-01-14T23:38:35.444Z","msg":"logging in to Azure ACR for fluxcdtestacr123.azurecr.io/charts","controller":"helmrepository","controllerGroup":"source.toolkit.fluxcd.io","controllerKind":"HelmRepository","HelmRepository":{"name":"charts","namespace":"flux-system"},"namespace":"flux-system","name":"charts","reconcileID":"bb59958b-0c44-4b35-871f-f7679c37d097"} source-controller-57774ccfc5-64bvd manager {"level":"info","ts":"2023-01-14T23:39:01.314Z","msg":"logging in to Azure ACR for fluxcdtestacr123.azurecr.io/charts","controller":"helmchart","controllerGroup":"source.toolkit.fluxcd.io","controllerKind":"HelmChart","HelmChart":{"name":"default-podinfo","namespace":"flux-system"},"namespace":"flux-system","name":"default-podinfo","reconcileID":"a0faa635-370a-4ad8-bb37-e60e16f3607c"} source-controller-57774ccfc5-64bvd manager {"level":"info","ts":"2023-01-14T23:39:02.021Z","msg":"pulled 'base' chart with version '0.3.2'","controller":"helmchart","controllerGroup":"source.toolkit.fluxcd.io","controllerKind":"HelmChart","HelmChart":{"name":"default-podinfo","namespace":"flux-system"},"namespace":"flux-system","name":"default-podinfo","reconcileID":"a0faa635-370a-4ad8-bb37-e60e16f3607c"}

[weisdd]

@stefanprodan I see that the test job failed complaining about diff in oci/tests/integration/go.mod (https://github.com/fluxcd/pkg/actions/runs/3937973714/jobs/6740711323#step:5:1), I've just run make all and pushed the updated go.sum and go.mod for the integration test. I guess all checks will pass this time.

[weisdd]

@phillebaba might be something of your interest since we had a brief discussion on workload identity for external-dns and cert-manager.

[stefanprodan]

@weisdd can you please rebase your PR with the main branch.

[weisdd]

@stefanprodan done

[weisdd]

@stefanprodan I've just rebased my branch again to resolve another round of merge conflicts (azidentity got recently updated in main).

[hiddeco]

Will ensure this gets in after #472, and before we tag the new package version.

[stefanprodan]

@hiddeco I think we need to do the same in kustomize-controller for SOPS decryption to work with workload identity.

[hiddeco]

@stefanprodan please fill an issue for this in kustomize-controller as it would require changing some additional logic as well I think (https://github.com/fluxcd/kustomize-controller/blob/main/internal/sops/azkv/config.go#L67), and I won't be able to make time for it this week.

[hiddeco]

Need to test if the bump of azidentity causes issues in other places where we e.g. make use of another Azure API subset.

[stefanprodan]

@hiddeco I think we should delay this further, I don't feel comfortable with shipping this in Flux v0.40.0 without any E2E tests.

[hiddeco]

Matches my thinking as I am concerned we would have issues with https://github.com/fluxcd/source-controller/blob/main/pkg/azure/blob.go#L87 as well.