Add support for Azure workload identity authentication for auto-unsealing

[cldmstr]

Is your feature request related to a problem? Please describe.
The vault auto-unseal mechanism should support Azure federated identities to connect to an Azure keyvault when this option is used.

Describe the solution you'd like
Vault should implement and use the Azure federated identities credentials when available to connect to Azure resources (keyvault in particular).
I created an issue on the go-kms-wrapping repo with a suggested implementation that we are currently running successfully in our cluster.

Describe alternatives you've considered
We could install vault on a VM setup or run the AAD pod identity v1 solution, which works with vault. However, the first one does not fit with our infrastructure concept and the second solution has been deprecated by Azure.

Additional context
We run Azure AKS clusters and use the new Azure workload identity (AAD pod identity v2) to connect k8s service accounts with Azure identities. This uses the Azure federated identity mechanism (magic) to inject the Azure credentials into the pods. Running vault in this configuration and using Azure keyvault for the auto unseal currently doesn't work, as the authentication mechanism isn't supported by vault.

[dhduvall]

There's a workaround: configure the vault pod(s) to use the proxy sidecar by adding the azure.workload.identity/inject-proxy-sidecar: "true" annotation to the pod.

Unfortunately, there's a bug in the WI webhook where it doesn't give the sidecar's init container the right securityContext: Azure/azure-workload-identity#697. The fix is straightforward and not too much of a pain to build for yourself if you don't want to wait for its integration.

[cldmstr]

@dhduvall Thanks for the tip. I'll definitely give the workaround the try. That will definitely make our lives easier, then we can use the official vault releases and don't have to manually patch each time to upgrade.

[UXabre]

This PR actually seems that it addresses the issue: #16611
Perhaps we can help revive this PR and see this pushed through?

[cldmstr]

@UXabre Yes and no. That issue addresses the same Azure authentication method, but as far as I can tell for the vault storage configuration. I need the same/similar code, but for the seal configuration to keep the auto unseal keys in Azure keyvault. The code for connecting to key stores lives in a different repo, but would essentially be the same.

[cldmstr]

Digging into the issue a bit further, all the github.com/Azure/go-autorest/autorest libraries used to connect to vault are going out of support on the 31. March 2023, so the whole azure authentication should probably be migrated to the new libraries as well.

[cldmstr]

Unfortunately, the side-car doesn't really fit our setup well. I created a PR for the library here:
hashicorp/go-kms-wrapping#155

[brightgong]

Hi @cldmstr i can install vault in aks and auto unseal with azure key vault, i use helm to install the vault, here is my config and you can have a try:

helm config:

server:
  enabled: true
  service:
    enabled: true
  annotations:
    azure.workload.identity/inject-proxy-sidecar: "true"
  extraLabels:
    azure.workload.identity/use: "true"
  serviceAccount:
    extraLabels:
      azure.workload.identity/use: "true"
    annotations:
      azure.workload.identity/client-id: "your managed identity client it" 

content for server.ha.raft.config

ui            = true
disable_mlock = true

listener "tcp" {
  tls_disable     = 1
  address         = "[::]:8200"
  cluster_address = "[::]:8201"
  telemetry {
    unauthenticated_metrics_access = true
  }
}

storage "raft" {
  path = "/vault/data/"
  retry_join {
    leader_api_addr = "http://vault-cluster.vault.svc.cluster.local:8200"
  }
}

seal "azurekeyvault" {
  vault_name = "${azure_keyvault_name}"
  key_name   = "${azure_keyvault_key_name}"
}

telemetry {
  prometheus_retention_time = "30s"
  disable_hostname          = true
}

here is the doc i'm refer: https://azure.github.io/azure-workload-identity/docs/topics/service-account-labels-and-annotations.html

and before installing vault you need to config in azure side for:

1. create your manage identity
2. create the federation credential for your aks in manage identity, for my case the namespace is vault , the sa is vault-cluster
3. create your azure key vault, i'm using RBAC to access
4. grand key vault access permission for your managed identity, the role name is Key Vault Crypto Officer I used
5. create a key in azure key vault
6. install azure-workload-identity into your aks, here is the sample code i used:

data "azurerm_client_config" "current" {}

resource "kubernetes_namespace" "azure-workload-identity-system" {
  metadata {
    name = "azure-workload-identity-system"
  }
}

# https://azure.github.io/azure-workload-identity/docs/introduction.html
resource "helm_release" "azure-workload-identity-system" {
  depends_on = [
    kubernetes_namespace.azure-workload-identity-system
  ]
  name       = "workload-identity-webhook"
  namespace  = kubernetes_namespace.azure-workload-identity-system.id
  chart      = "workload-identity-webhook"
  version    = "1.0.0"
  repository = "https://azure.github.io/azure-workload-identity/charts"
  wait       = true
  set {
    name  = "azureTenantID"
    value = data.azurerm_client_config.current.tenant_id
  }
}

now you can install your vault with helm, for my code, the helm use vault-cluster as app name and namespace is vault
then for vault pod, you can see there will have 3 containers: vault, azwi-proxy, and azwi-proxy-init
then you can shell to the vault and do 'vault operator init' to see if it unseal and generate the recovery keys and root token

Good luck~~

[MasterBroda]

@brightgong's solution works great but will break if you plan to use Azure Secrets Engine as the client ID passed via the annotation takes precedence over the client ID passed to the Secret Engine's config. This is because the annotation injects it as an environment variable.
To get around this problem, pass the client ID in the seal config rather than the annotation.

        seal "azurekeyvault" {
          vault_name = "xyz"
          key_name   = "zbc"
          client_id  =  "abc
        }

Ref - https://support.hashicorp.com/hc/en-us/articles/12719296376339

[jtv8]

It's worth repeating the warning from the Microsoft docs about the workload identity proxy sidecar:

The migration sidecar is not supported for production use. This feature is meant to give you time to migrate your application SDK's to a supported version, and not meant or intended to be a long-term solution. The migration sidecar is only available for Linux containers, due to only providing pod-managed identities with Linux node pools.

I notice Vault is still using ADAL for authentication, which has been deprecated and superseded by MSAL. See: https://learn.microsoft.com/en-us/azure/active-directory/develop/msal-migration

Apart from adding the possibility to support workload identity authentication, this would seem to be a bigger issue as ADAL is no longer receiving security fixes as of June 30 this year. Is there an issue somewhere tracking this?

Edit re: that last question - I see @cldmstr's PR above addresses this for auto-unsealing but I think this would also need to be addressed separately for physical/azure: https://github.com/hashicorp/vault/blob/main/physical/azure/azure.go

[MueChr]

Vault 1.15.0 should support Workload Identity.

auth/azure: Add support for azure workload identity authentication (see issue
#18257). Update go-kms-wrapping dependency to include PR
#155 [https://github.com//pull/22994]

Has someone successfully running the vault with Azure Key Vault and Workload Identity, without using proxy sidecar?

We have the problem, that the pod is not ready and no logs are written (log level already changed to trace).

[cldmstr]

@MueChr Yes, the relevant change to go-kms-wrapping was merged and included. We have it working in a cluster with that change and it works well.

[cldmstr]

This issue can be closed, as it is fixed with that PR.

[cldmstr]

@jtv8 Yes this would need to be addressed for that case as well, as the authentication is handled separately for both cases.

[UXabre]

Hi @cldmstr, I'm a bit confused? I see that some things got merged but, AFAICT, it still uses the older method of trying to fetch the token. Do you have some explanation on how you (or anyone) got auto-unseal working? The latest documentation also doesn't seem to mention it (or I am misinterpreting it)...

What I did so far:

• Enabled workload identity on the pod (set label azure.workload.identity/use: 'true')
• Enabled identity on SA (set label azure.workload.identity/use: 'true')
• Added ClientID to SA

I get this error:
error parsing Seal configuration: error fetching Azure Key Vault wrapper key information: azure.BearerAuthorizer#WithAuthorization: Failed to refresh the Token for request to https://xxx.vault.azure.net/keys/vault-master-key/?api-version=7.1: StatusCode=400 -- Original Error: adal: Refresh request failed. Status Code = '400'. Response body: {"error":"invalid_request","error_description":"Identity not found"} Endpoint http://169.254.169.254/metadata/identity/oauth2/token?api-version=2018-02-01&client_id=####&resource=https%3A%2F%2Fvault.azure.net

I don't think it should be ADAL bein used?

I am running hashicorp/vault:1.15.2

Any pointers are welcome, I think I'm forgetting something painfully obvious here...

[cldmstr]

@UXabre Sorry, it's been a long time since I made this PR and I'm actually not working for the customer I initiated it for any more, so don't really have a test-bed anymore. However, they have recently updated to vault 1.15.x and it works in their setup, so should be possible.

The Azure authentication magic tries to determine what auth method to use based on the ENV variables and probably falls back to ADAL when it finds nothing else. It should be using federated auth. It could also be that the ADAL variables are set and this has precedence over the federated auth. I'm assuming something isn't quite working with the workload identity setup in the cluster, though.

If it's working you will find the following variables on the running pods, those are the ones I look for when debugging issues with the setup:
AZURE_CLIENT_ID
AZURE_TENANT_ID
AZURE_AUTHORITY_HOST
AZURE_FEDERATED_TOKEN_FILE
Additionally, there should be a file mount with the path in AZURE_FEDERATED_TOKEN_FILE

Did you also federate the SA with a service principal that has the roles you want to give the SA? (Assuming yes, but asking because you didn't mention that step.)

[UXabre]

Hi @cldmstr , I've checked these ENV variables and they are in and yes, your assumption is right that I did add the correct roles to the service principal.

I did some more digging into the code and found that https://github.com/hashicorp/vault/blob/main/physical/azure/azure.go hasn't really changed in the last months (I'd expect it would)

also, looking at the PR #16611 it seems we discussed something related to that.

The only support I currently see, is for auto-auth in vault agent: https://developer.hashicorp.com/vault/docs/agent-and-proxy/autoauth/methods/azure#example-configuration
It is the only thing specifically mentioning it

Also, here it is mentioned that something is not entirely according to plan:
#22994 (comment)

So... my guess is that we're not there...yet but very close

EDIT: it seems that the latter was added from 1.15.5, I'm running against 1.15.2 so maybe...just maybe

EDIT2: ...yes! 1.15.5 works as expected!!

[adrianlzt]

So then, this PR is fixed as Terraform 1.15.5 allows to auto-unseal using workload identity, but it is still not implemented for azure storage, right?

[cldmstr]

@adrianlzt Yes, I had made a PR for the auto-unseal, but the azure storage uses different authentication code which would need the same change.