Error when deploying azurerm_container_group with CMK using key_vault_key_id setting

[dawsonar802]

Is there an existing issue for this?

• I have searched the existing issues

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment and review the contribution guide to help.

Terraform Version

1.4.0

AzureRM Provider Version

3.72.0

Affected Resource(s)/Data Source(s)

azurerm_container_group

Terraform Configuration Files

resource "azurerm_container_group" "azure_ci" {
  name = lower("ci-${var.abv_location}-${var.env}-${var.current_index}-${var.service}")
 
  location            = var.location
  resource_group_name = var.resource_group_name
  ip_address_type     = "Private"
  os_type             = "Linux"
  tags                = var.tags
  key_vault_key_id    = var.key_vault_key_id
  zones               = var.zones
  subnet_ids          = [var.ci_subnet_id]
 
  dynamic "container" {
    for_each = var.container
 
    content {
      name   = lower("${var.abv_location}-${var.env}-${container.key}-${var.service}")
      image  = container.value.image
      cpu    = container.value.cpu
      memory = container.value.memory
 
      ports {
        port     = container.value.port
        protocol = container.value.protocol
      }
 
      dynamic "volume" {
        for_each = var.create_volume ? [1] : []
 
        content {
          name       = lower("${var.abv_location}-${var.env}-${container.key}-${var.service}-${container.value.vol_name}")
          mount_path = container.value.vol_path
          read_only  = container.value.vol_read_only
          share_name = container.value.vol_share
 
          storage_account_name = container.value.vol_stg_name
          storage_account_key  = container.value.vol_stg_primary_key
        }
      }
 
      environment_variables        = container.value.ci_env
      secure_environment_variables = container.value.ci_env_pvt
 
    }
  }
 
  dynamic "image_registry_credential" {
    for_each = var.image_registry_credential.server == "" ? [] : [1]
    content {
      username                  = var.image_registry_credential.username == null ? null : var.image_registry_credential.username
      password                  = var.image_registry_credential.password == null ? null : var.image_registry_credential.password
      user_assigned_identity_id = var.image_registry_credential.user_assigned_identity_id == null ? null : var.image_registry_credential.user_assigned_identity_id
      server                    = var.image_registry_credential.server
    }
  }
 
  identity {
    type         = var.identity.type
    identity_ids = var.identity.identity_ids
  }
 
  diagnostics {
    log_analytics {
      workspace_id  = var.log_analytics_workspace_workspace_id
      workspace_key = var.log_analytics_workspace_key
    }
  }
}

Debug Output/Panic Output

{

    "status": "Failed",

    "error": {

        "code": "InternalServerError",

        "message": "Encountered an internal server error. The tracking activity id is 'cc386ab1-e5a8-4f34-b518-43d65d0c45c9', correlation id is 'ca2c32c5-ec39-471e-b94f-1f8f988fbe02'."

    }

}

Expected Behaviour

Deploy the resource with CMK and linked to the specified KV.

Actual Behaviour

A generic error is given:

{

    "status": "Failed",

    "error": {

        "code": "InternalServerError",

        "message": "Encountered an internal server error. The tracking activity id is 'cc386ab1-e5a8-4f34-b518-43d65d0c45c9', correlation id is 'ca2c32c5-ec39-471e-b94f-1f8f988fbe02'."

    }

}

Steps to Reproduce

No response

Important Factoids

No response

References

No response

[dawsonar802]

I have reached out to Microsoft on this and opened a support cause. The support team escalated to the backend product team who provided the following information and asked that I open a GH Issue.

They believe this is a bug on TF. They enabled TF Debug (example tutorial: https://docs.aiven.io/docs/tools/terraform/howto/terraform-logging) to get the JSON in the ARM PUT request:

2023-09-14T11:42:20.384-0600 [DEBUG] provider.terraform-provider-azurerm_v3.72.0_x5.exe: AzureRM Request:
PUT /subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourceGroups/acirepro/providers/Microsoft.ContainerInstance/containerGroups/acirepro?api-version=2023-05-01 HTTP/1.1
Host: management.azure.com
User-Agent: Go/go1.20.5 (amd64-windows) go-autorest/v14.2.1 hashicorp/go-azure-sdk/containerinstance/2023-05-01 HashiCorp Terraform/1.3.9 (+https://www.terraform.io) Terraform Plugin SDK/2.10.1 terraform-provider-azurerm/3.72.0 pid-222c6c49-1b0a-5959-a213-6608f9eb8820
Content-Length: 1051
Content-Type: application/json; charset=utf-8
X-Ms-Correlation-Request-Id: 33e8624d-5cc7-6ef2-24c0-dff1b534de94
Accept-Encoding: gzip

{"identity":{"type":"UserAssigned","userAssignedIdentities":{"/subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourceGroups/acirepro/providers/Microsoft.ManagedIdentity/userAssignedIdentities/myACIId":{}}},"location":"eastus","name":"acirepro","properties":{"containers":[{"name":"hello-world","properties":{"command":[],"environmentVariables":[],"image":"mcr.microsoft.com/azuredocs/aci-helloworld:latest","ports":[{"port":80,"protocol":"TCP"}],"resources":{"requests":{"cpu":0.5,"memoryInGB":1.5}}}}],"encryptionProperties":{"keyName":"acikey","keyVersion":"698ab1c778ef4d9981480e922d33ed47","vaultBaseUrl":"https://acikvcmk.vault.azure.net/"},"ipAddress":{"autoGeneratedDomainNameLabelScope":"Unsecure","ports":[{"port":80,"protocol":"TCP"}],"type":"Private"},"initContainers":[],"osType":"Linux","restartPolicy":"Always","sku":"Standard","subnetIds":[{"id":"/subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourceGroups/acirepro/providers/Microsoft.Network/virtualNetworks/acivnet/subnets/acisubnet"}],"volumes":[]},"tags":{},"zones":[]}: timestamp=2023-09-14T11:42:20.384-0600
2023-09-14T11:42:21.383-0600 [DEBUG] provider.terraform-provider-azurerm_v3.72.0_x5.exe: AzureRM Response for https://management.azure.com/subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourceGroups/acirepro/providers/Microsoft.ContainerInstance/containerGroups/acirepro?api-version=2023-05-01:
HTTP/2.0 500 Internal Server Error
Content-Length: 216
Cache-Control: no-cache
Content-Type: application/json; charset=utf-8
Date: Thu, 14 Sep 2023 17:42:10 GMT
Expires: -1
Pragma: no-cache
Strict-Transport-Security: max-age=31536000; includeSubDomains
X-Content-Type-Options: nosniff
X-Ms-Correlation-Request-Id: 33e8624d-5cc7-6ef2-24c0-dff1b534de94
X-Ms-Failure-Cause: service
X-Ms-Ratelimit-Remaining-Subscription-Resource-Requests-Pt1h: 299
X-Ms-Ratelimit-Remaining-Subscription-Resource-Requests-Pt5m: 99
X-Ms-Ratelimit-Remaining-Subscription-Writes: 1199
X-Ms-Request-Id: eastus:9e90d311-25e5-4802-b683-ad058bff5abe
X-Ms-Routing-Request-Id: EASTUS:20230914T174210Z:6929ec2b-c0e9-4f7b-a6f0-f3b749c648c1

As you can see TF uses latest ACI API 2023-05-01, which supports CMK via managed identity plus RBAC, but according to docs the encryptionProperties block needs the identity id: https://learn.microsoft.com/en-us/rest/api/container-instances/2023-05-01/container-groups/create-or-update?tabs=HTTP#encryptionproperties. This is true for all APIs equal or newer than 2022-09-01, which is the one they have used in their ARM repro that worked:

So, TF is still handling this encryptionProperties block per API definition equal or older than 2021-10-01 when only Access Policies were supported to access KV: https://learn.microsoft.com/en-us/rest/api/container-instances/2021-10-01/container-groups/create-or-update?tabs=HTTP#encryptionproperties. Please log a bug to TF to address this. Also, this comment is no longer true, only for Azure Government clouds:

Our CMK docs need to be updated to show RBAC is supported if you use API 2022-09-01 or newer and you add the identity field to the encyptionProperties block. In fact, this is the version that is used in the complete JSON example, while the extract example uses 2019-12-01 which fits the Access Policy instructions. They have already logged a GitHub issue for this: https://github.com/MicrosoftDocs/azure-docs/issues/114089#issuecomment-1703836056

[magodo]

@dawsonar802 Would you please provide a minimal self-contained configuration so that I can reproduce this locally? The reason I'm asking this is because the existing acctest for the CMK works fine, where it didn't set the identity.

[dawsonar802]

@magodo, the code mentioned in the initial post is what I am using. MS deployed using some different code, but I was unable to access that from the file download site. Identity is needed from my understanding and the identities needs access on the KV via RBAC. That should get you the same error. Let me know if that helps.

[dawsonar802]

@stephybun / @magodo, just so I understand, the resource now should work with RBAC instead of only Access Policy permissions on the KV correct? We just need to pass in the identity that has access? It seems confusing that we could not just use the existing identity block and the system or user assigned identity that we are already creating/passing in. Could you please clarify. It is also somewhat confusing and misleading having this issue closed when the 3.74.0 release is not available and I am not able to confirm the issue is actually resolved.

[magodo]

@dawsonar802

If I understand the API correctly, previously it only supports to assign needed key vault access policy to RP owned SP named "Azure Container Instance Service". Then you just need to specify the key_vault_key_id for the azurerm_container_group, in which case the RP will use its own SP to access the key vault key for wrapping/unwrapping the encryption key. See this for the example config.

With #23332, users are able to specify the identity used to access the key vault key. This identity can be the same user assigned identity, or one of the identities set to the azurerm_container_group (via identity), though I haven't tried the case that using an identity not assigned to the azurerm_container_group yet. In order to make sure the user assigned identity has the proper permission, ensure to grant it also via the key vault access policy. I haven't tried the case that not granting the key vault acess policy at all, since per my test, I've got the exact same error as you've provided above, and it is resolved by specifying the new key_vault_user_assigned_identity_id.

Regarding why this issue is closed, it is the process we are practicing: closing the issue and mark the milestone that the fix will be shipped, instead of close a bunch of issues after the release. You are welcome to reopen this issue if the issue remains for you. Whilst, it would be helpful that you can provide a minimal self-contained config (with all the values interpolated) that can reproduce this issue.

[dawsonar802]

@magodo, understood. Yes, RBAC is the main issue, as it is recommended to move away from Access Policies and to use RBAC. I did test with giving the 'Azure Container Instance Service' various RBAC roles, but it still would not work. So maybe once the key_vault_user_assigned_identity_id is in place it will work with RBAC as well. Is that something you would be able to test?

[kennethgp]

@dawsonar802 RBAC will work as long as the new key_vault_user_assigned_identity_id fills identity field in the encryptionProperties JSON block TF will send ARM, and provided that user identity has enough role assigment over KV, e.g. Key Vault Crypto Officer.

With current TF version no RBAC role over identity or 'Azure Container Instance Service' SP will work, because the encryptionProperties block lacks the identity field that instruct the RP not to use Access Policies.

[dawsonar802]

Thanks @kennethgp . I will give it a try once the new version is released.

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.