Support for AAD Workload Identities

[ulucinar]

What problem are you facing?

Using AD service principal credentials is not the recommended way for AKS workloads.

How could Crossplane help solve your problem?

We can consider adding support for AAD workload identities in Crossplane Azure providers.

[jbw976]

@ulucinar - is this a duplicate of either #164 or #292?

If so, can you please merge/clean-up these issues? that would be very helpful! :)

[ulucinar]

Hi @jbw976,
My current understanding regarding these issues is as follows:

• (feat): Azure MSI authentication #164 asks for supporting authentication via Azure Managed Identities (formerly known as Managed Service Identities (MSI)) in provider-azure. This will allow us, for instance, to use ProviderConfigs that refer to no credential secrets.
• [Proposal] Use System Managed Identity OR User Managed Identity for AKSCluster #292 asks for provisioning AKS clusters using a system-assigned or user-assigned managed identity instead of creating a service principal as part of the provisioning flow like we currently do. As explained here, an AKS cluster needs an identity to provision additional resources like load balancers or managed disks. [Proposal] Use System Managed Identity OR User Managed Identity for AKSCluster #292 is about making the AKS cluster use a system-assigned or a user-assigned managed identity for that.
• This issue is similar to (feat): Azure MSI authentication #164, i.e., it asks for supporting a new means of authentication for provider-azure against Azure AD/Resource Manager, but using workload identities this time. Using workload identities requires exchanging (volume-projected) Kubernetes service account tokens with Azure AD access tokens as detailed here.

Another related concept is Azure AD pod identities but since they are being replaced with Azure workload identities, I don't think it makes sense to support them in Crossplane azure providers.

Hope this clarifies the contexts of these related issues.

[ldardick]

While native support is not yet supported, I found an alternative to use a Workload Identity with the Azure provider. This requires using the azwi-proxy sidecar which is meant to be used to migrate older applications from Pod Identity (now deprecated) to Workload Identity.

First, a User Assigned Identity should be created, and the cluster configured with the OIDC issuer enabled and the mutating admission webhook installed. There is a step-by-step explanation here.

Then, we need to configure a ProviderConfig for the Azure provider defining the Client ID as it was using a User Assigned Identity for the kubelet identity (Reference here)

apiVersion: azure.upbound.io/v1beta1
kind: ProviderConfig
metadata:
  name: [provider-config-name]
spec:
  credentials:
    source: UserAssignedManagedIdentity
  clientID: [client-id]
  subscriptionID: [subscription-id]
  tenantID: [tenant-id]

The tricky part is the following. When installing the Azure provider, Crossplane will create a Deployment for the controller and a Service Account. We need to modify those as defined below:

For the Deployment:

• Add the azure.workload.identity/inject-proxy-sidecar: 'true' annotation
• Add the azure.workload.identity/use: 'true' label

For the Service Account

• Add the azure.workload.identity/client-id: [client-id] annotation, replacing [client-id] with the ID of the user assigned identity
• Add the azure.workload.identity/use: 'true' label

Finally, we should configure the federation for the User Assigned Identity as explained in the link above using the name of the namespace and the name of the Service Account.
Mind that this behavior may be overridden if the Provider is upgraded.

[chatelain-io]

Following @ldardick explanation, here's a configuration example. It uses a user assigned identity and import an existing ResourceGroup. It will work even if you update the provider.

---
apiVersion: v1
kind: ServiceAccount
metadata:
  labels:
    azure.workload.identity/use: "true"
  annotations:
    azure.workload.identity/client-id: <client-id>
    azure.workload.identity/tenant-id: <tenant-id>
  name: <service-account-name>
  namespace: crossplane-system

---
apiVersion: pkg.crossplane.io/v1alpha1
kind: ControllerConfig
metadata:
  name: provider-azure-family-config
spec:
  metadata:
    annotations:
     azure.workload.identity/inject-proxy-sidecar: 'true'
    labels:
      azure.workload.identity/use: "true"
  serviceAccountName: <service-account-name>
  args:
  - --enable-management-policies
---
apiVersion: pkg.crossplane.io/v1
kind: Provider
metadata:
  name: provider-azure-family
spec:
  package: xpkg.upbound.io/upbound/provider-family-azure:v0.36.0
  controllerConfigRef:
    name: provider-azure-family-config
---
apiVersion: azure.upbound.io/v1beta1
kind: ProviderConfig
metadata:
  name: default
  labels:
    azure.workload.identity/use: "true"
spec:
  clientID: <client-id>
  credentials:
    source: UserAssignedManagedIdentity
  subscriptionID: <subscription-id>
  tenantID: <tenant-id>

---
apiVersion: azure.upbound.io/v1beta1
kind: ResourceGroup
metadata:
  name: <rg-name>
  annotations:
    crossplane.io/external-name: <existing-rg-name>
spec:
  managementPolicies: ["Observe"]
  forProvider: {}
  providerConfigRef:
    name: default

[gravufo]

Any news on official support for this? The workaround may work for now, but Microsoft has stated that this path is only a workaround and is not supported for production use.
Also, would that include Functions support? Let's say I need to fetch information from resources in Azure from within my function, I would need an identity and from my understanding, the function runs as a separate pod which means we need to inject the proper serviceAccount. That said, from my quick search there didn't seem to be a way to inject a serviceAccount name or labels on the function definition.
Thanks!

[chatelain-io]

I was able to make it work without injecting the proxy as sidecar by using the OIDCTokenFile

---
apiVersion: v1
kind: ServiceAccount
metadata:
  annotations:
    azure.workload.identity/client-id: <client-id>
    azure.workload.identity/tenant-id: <tenant-id>
  name:  <service-account-name>
  namespace: crossplane-system
---
apiVersion: pkg.crossplane.io/v1beta1
kind: DeploymentRuntimeConfig
metadata:
  name: provider-azure-family-config
spec:
  deploymentTemplate:
    spec:
      selector: {}
      template:
        metadata:
          labels:
            azure.workload.identity/use: "true"
        spec:
          serviceAccountName: <service-account-name>
          containers:
            - name: package-runtime
              args:
                - --enable-external-secret-stores
                - --enable-management-policies
---
apiVersion: pkg.crossplane.io/v1
kind: Provider
metadata:
  name: provider-azure-family
spec:
  package: xpkg.upbound.io/upbound/provider-family-azure:v0.41.0
  runtimeConfigRef:
    name: provider-azure-family-config
---
apiVersion: azure.upbound.io/v1beta1
kind: ProviderConfig
metadata:
  name: workload-identity-provider-config
spec:
  credentials:
    source: OIDCTokenFile
  oidcTokenFilePath: /var/run/secrets/azure/tokens/azure-identity-token
  clientID: <client-id>
  subscriptionID: <subscription-id>
  tenantID: <tenant-id>
---
apiVersion: azure.upbound.io/v1beta1
kind: ResourceGroup
metadata:
  name: <rg-name>
  annotations:
    crossplane.io/external-name: <existing-rg-name>
spec:
  managementPolicies: ["Observe"]
  forProvider: {}
  providerConfigRef:
    name: workload-identity-provider-config

[gravufo]

Forgot to comment back, but I also did the same. Works flawlessly. I guess this issue can be closed.

[illrill]

Saved my day @chatelain-io! Should probably be documented before closing, e.g. here:

• https://docs.upbound.io/providers/provider-azure/authentication/#user-assigned-managed-identity
• https://docs.crossplane.io/latest/getting-started/provider-azure/

[The-Real-Adeel]

Hi, Question regarding workload identities.

If say we want to utilize multiple workload identities... how would we go about it?

So far the examples are only showing using one workload identity for the providers.

We want to separate our pods to have different capabilities by assigning different service accounts with their own workload identities. Some managing network, others managing say storage.

Is there a way to get multiple workload identities with different RBACs able to create different things within the same cluster using crossplane?

[patst]

Hi, Question regarding workload identities.

If say we want to utilize multiple workload identities... how would we go about it?

So far the examples are only showing using one workload identity for the providers.

We want to separate our pods to have different capabilities by assigning different service accounts with their own workload identities. Some managing network, others managing say storage.

Is there a way to get multiple workload identities with different RBACs able to create different things within the same cluster using crossplane?

That is a use case we have as well.
We have a multi-tenant cluster with ProviderCredentials for each tenant.
Ideally we would have a ManagedIdentity per tenant and can use this.

It is possible to attach multiple identities to a Service Account (see https://azure.github.io/azure-workload-identity/docs/faq.html#how-to-federate-multiple-identities-with-a-kubernetes-service-account ).
The provider credentials would then need to specify only the identity-id to be used and this id must be passed on during AAD authentication (e.g. in the DefaultCredentials constructor)

[avo-sepp]

I think the ability to use multiple workload identities is desirable for us too. I think a new feature request should be opened up if there isn't a good solution for this as is. I have not discovered it yet.

[gravufo]

My guess is that it should already work with multiple identities. You can assign multiple identities to a single service account: https://azure.github.io/azure-workload-identity/docs/faq.html#how-to-federate-multiple-identities-with-a-kubernetes-service-account

From there, you simply need to have multiple ProviderConfig that specify the different clientIDs you wish to use.
Do note that this is completely theoretical. If someone tries it before me, it would be good to report back on whether it works or not.

[patst]

just one update:
@chatelain-io was completely right and I had an error about how workload identity works.

The oidc file (e.g. located at /var/run/secrets/azure/tokens/azure-identity-token) is issued by the kubernetes clusters oidc issuer.
This file can be used to get access token from EntraID for different UserAssignedIdentities (all which have the correct federated credentials configured for the serviceaccount of the provider pod).

I was able to use different identities configured in different ProviderConfigs.

I think there is no implementation required on top, maybe the documentation could have an usage example

[chatelain-io]

@patst Yup, in my example I set the client id and the tenant id on the service account but you don't have to do this.

It is only required to set azure.workload.identity/use: "true" label on the deploymentTemplate.

It will then use the clientid, tenantId of the ProviderConfig.

My only concern about this, is that the ProviderConfig is not namespaced, anyone can reference it which could be dangerous...

[finnlander]

Hello,

Hopping into the discussion as I'm also trying to find a solution to the same problem...

Disclaimer: I'm not that fluent with AKS internal and mechanisms, so please correct me if I'm wrong here 🙂 .

The current Azure provider Authorization options documented here doesn't really recommend one way over the other, but to me it seems all of them has some drawbacks (as such):

• The service account (client creds) based approach has the evident problem of needing to manage the credentials (and having possibly the long expiration period).

• The system-assigned and user-assigned mechanisms have that "fine-print" that is stated out loud in the "user-assigned" section:

You must use the user-assigned managed identity as the kubelet identity of your AKS cluster.

Please correct if I'm wrong here, but that seems to mean that the identity that is used for Crossplane (Azure provider), which requires quite wide set of permissions to the cloud (create and delete all the managed resources), is also shared with all the nodes (that being the kubelet identity). And this sounds something that the "security best practices" (repeating the "least privileges principle" mantra) would strongly advice against.

I verified my assumption with a test cluster and when trying to create a simple ResourceGroup resource with the system-assigned managed identity approach, crossplane provider fails with an error related to "insufficient permissions to read resource groups" (i.e. requiring more permissions for the managed identity).

Therefore, I've lead myself into believing that the OIDC federation based setup is the best alternative, considering the security aspects,

I.e. having:

• the OIDC provider enabled in AKS,
• a separate managed identity with the "contributor" role (or whatever fine-grained permissions) to allow managing the external resources,
• "federated-credential" binding with the managed identity and a specific service account in the crossplane-system's namespace (that is then associated with the crossplane pods)
• using those "workload-identity" annotations mentioned in the above posts (with the azure's workload identity webhook that acts on them).

(i.e. setting up the workload identity based authentication)

... .

As this way the "Contributor" level permissions are scoped only with the specific namespace+service account (and not available for all the kubelets).

I can hopefully get it working with the manifests provided above (thanks @chatelain-io 🙇), but I would also really like to see such approach being the first one mentioned in the "Authentication" document (as "workload identity based authentication" is also the one officially recommended by Microsoft), with perhaps stating the possible security implications of using the other means. Of course not, if I'm completely mistaken here (not being an expert with AKS and such matters).

p.s. If I wouldn't have found this great discussion, I'd probably fall back into using the client credentials (as I see it's tradeoffs more acceptable than elevating permissions for the kubelet identity).

p.p.s. Thank you all for the effort you put into Crossplane 💚

[Speeddymon]

@patst Yup, in my example I set the client id and the tenant id on the service account but you don't have to do this.

It is only required to set azure.workload.identity/use: "true" label on the deploymentTemplate.

It will then use the clientid, tenantId of the ProviderConfig.

My only concern about this, is that the ProviderConfig is not namespaced, anyone can reference it which could be dangerous...

@chatelain-io Make an XProviderConfig resource (Crossplane Kubernetes Provider creates Object, deploys ProviderConfig) and the XProviderConfig resource will be able to get a claim in a namespace. It's a workaround but it might help you.

[MaxAnderson95]

@chatelain-io Thanks for this! One thing I found missing was the azure.workload.identity/use: "true" annotation on the service account.

[chatelain-io]

@chatelain-io Thanks for this! One thing I found missing was the azure.workload.identity/use: "true" annotation on the service account.

This annotation is not required on the service account but on the pod of the provider.

See ms documentation.

And it is set on the deploymentruntimeconfig.

The annotation tells aks to inject the proper environment variables in the pod spec so the WI works properly.

[chatelain-io]

@patst Yup, in my example I set the client id and the tenant id on the service account but you don't have to do this.

It is only required to set azure.workload.identity/use: "true" label on the deploymentTemplate.

It will then use the clientid, tenantId of the ProviderConfig.

My only concern about this, is that the ProviderConfig is not namespaced, anyone can reference it which could be dangerous...

@chatelain-io Make an XProviderConfig resource (Crossplane Kubernetes Provider creates Object, deploys ProviderConfig) and the XProviderConfig resource will be able to get a claim in a namespace. It's a workaround but it might help you.

It doesn't help if you do not set the right rbac in a multi tenant scenario. Even if you create a composition, the resource (ProviderConfig) is still created and cluster scoped, anyone can reference it from any namespace if they know the name.

[patst]

@patst Yup, in my example I set the client id and the tenant id on the service account but you don't have to do this.
It is only required to set azure.workload.identity/use: "true" label on the deploymentTemplate.
It will then use the clientid, tenantId of the ProviderConfig.
My only concern about this, is that the ProviderConfig is not namespaced, anyone can reference it which could be dangerous...

@chatelain-io Make an XProviderConfig resource (Crossplane Kubernetes Provider creates Object, deploys ProviderConfig) and the XProviderConfig resource will be able to get a claim in a namespace. It's a workaround but it might help you.

It doesn't help if you do not set the right rbac in a multi tenant scenario. Even if you create a composition, the resource (ProviderConfig) is still created and cluster scoped, anyone can reference it from any namespace if they know the name.

In a multitenant scenario you must not allow the tenants to create ManagedResources directly.
They only should be allowed to create Claims, which are namespace-scoped and won't allow them to use a ProviderConfig of their choice, because you can patch it in the Composite.

See https://docs.crossplane.io/latest/guides/multi-tenant/#namespaces-as-an-isolation-mechanism

[Speeddymon]

In a multitenant scenario you must not allow the tenants to create ManagedResources directly. They only should be allowed to create Claims, which are namespace-scoped and won't allow them to use a ProviderConfig of their choice, because you can patch it in the Composite.

Exactly this. You disallow creation of the ManagedResources by not granting the role allowing their creation to human users; only to the Crossplane ServiceAccounts. Then the claims make the managed resources (via the Crossplane ServiceAccounts) and can only reference the correct providerconfig.

You can use admission control like Azure Policy, Kyverno or Gatekeeper with Crossplane to apply further controls which would prevent the automation from being abused into applying a wrong ProviderConfig reference to a ManagedResource, and if you need to give human users the ability to apply ManagedResources directly, then definitely use admission control to restrict what's allowed to be input into the ManagedResource provider config reference.

[keymon]

We also want a multi tenant model, but we have some additional requirements:

1. We are running crossplane in AWS EKS, not Azure. Still, we want to automate azure. This is not a problem for federated credentials, we can assign the OIDC of EKS and federate this way. But we won't get the projected token automatically added, it must be specified as a projected volume and env vars set accordingly.

2. We wanted tenants to "bring your own Subscription". The tenant will have a k8s SA, and a identity is setup in their subscription to allow that SA.