keyvault: populating the Key Vaults cache using the KeyVault Resource Provider

[tombuildsstuff]

There's been a plethora of attempts to fix this issue over the years, so this one requires a little background to under the choices involved here.

Background

When provisioning Nested Items within a Key Vault, that is:

• Certificates (using the unversioned URI https://myvault.vault.azure.net/certificates/mycert1 - or the versioned URI https://myvault.vault.azure.net/certificates/mycert1/abc123).
• Keys (using the unversioned URI https://myvault.vault.azure.net/keys/mykey1 - or the versioned URI https://myvault.vault.azure.net/keys/mykey1/abc123).
• Secrets (using the unversioned URI https://myvault.vault.azure.net/secrets/mysecret1 - or the versioned URI https://myvault.vault.azure.net/secrets/mysecret1/abc123).
• Storage Accounts (using the URI https://myvault.vault.azure.net/storage/account1).

These have to be provisioned using the Data Plane endpoint (that is, the API hosted at https://{account}.vault.azure.net) rather than the Resource Manager API due to a disparity in the available functionality.

This means that Terraform needs to be able to access the Key Vault Data Plane API in order to be able to both provision items in it, but also to refresh it's state.

It's worth noting that it's possible for a Key Vault to be available but not accessible - for example when a key Vault has Public Network Access disabled, or where the IP Address where Terraform is being run doesn't have access to the Key Vault.

In addition it's possible for the Key Vault to be available but (temporarily) inaccessible - which are typically DNS issues (particularly around Private Link where the DNS replication can take some time to update).

As such Terraform needs a reliable manner of determining when a Key Vault is available and accessible - and when it's available but inaccessible - and when it's unavailable (i.e. deleted) in order to be able to surface the correct behaviour.

Ultimately this allows us to support:

1. When a Key Vault has been deleted outside of Terraform, the next Plan will detect that both the Key Vault and it's Nested Items require recreation/should become untracked.
2. When a Key Vault is available but inaccessible - we should raise an error stating we can't connect to the Key Vault, rather than marking the Nested Items as untracked - since these items aren't deleted, removing from the state and attempting to recreate them will lead to an error during recreation, due to the requires import check.

Problem

When refreshing the Terraform State we have access to only the Resource ID - which for a Nested Item is in the format:

https://vault1.vaults.azure.com/secrets/example1/abc123

Which can get parsed into:

• Vault Name: vault1
• Type: secrets
• Name: example1
• Version (optional): abc123

However in order to be able to access the Key Vault via the Resource Manager API we need to know both the Subscription ID and Resource Group name where the Key Vault exists.

There's three ways of doing this:

1. Using the Resources API within the Microsoft.Resources Resource Provider (which we're doing today).
   • We currently use this API in the Provider - however unfortunately this API returns outdated information (see Intermittent Error: Unable to locate Storage Account when RAGRS/RAZGRS account kind is used #15048, Azure KeyVault Error: Provider produced inconsistent result after apply #11059 etc) - whilst this can be fixed by Azure Support clearing the cache it's not a great workaround.
2. Using the Resources API within the Microsoft.ResourceGraph Resource Provider.

• Whilst suggested this removes the caching related issues, unfortunately we've encountered outdated information through this endpoint - so we're not convinced this would improve matters.

3. Using the ListBySubscriptions API within the Key Vault Resource Provider.
   • From memory: this endpoint suffers from a low rate limit (5x/second) - although the Key Vault Rate Limit documentation only refers to a limit for Managed HSMs?
   • We've previously used this Endpoint within the Provider - however this implementation shipped with a bug requiring that this endpoint was repeatedly called, which was a problem in larger Configurations.

Options 1 and 3 have shipped in the provider, however each has been investigated - ultimately these endpoints are used to populate a cache of the Key Vault Name (the key) to MetaData about the Key Vault (e.g. KeyVaultId, DataPlaneBaseUri, ResourceGroup).

This means that once the Key Vault has been obtained once per Provider Instantiation (which happens once at Plan and once at Apply time) - it's cached within the Provider.

However both implementations have a caching issue where when a Key Vault is not present within the cache - it's obtained from the Azure API one at a time. Whilst this does have a minor performance benefit when provisioning a single Key Vault within a Subscription - it means that users provisioning against multiple Key Vaults can hit up against the Timeout for a given Resource (context deadline exceeded).

Solution

This PR changes how the cache is populated, such that when looking up a Key Vault we now:

1. Determine the cache Cache Key for this Key Vault
2. Check if this Key Vault is present in the Cache, returning it if so.
3. If not, we retrieve ALL of the Key Vaults within the specified Subscription and populate ALL of them into the Cache.
4. Re-check the cache for the Key Vault by it's Cache Key - returning it if so.
5. If not we return nil so that Resources and Data Sources can surface as needed.

It's worth worth noting that the Cache is populated/evicted as Key Vaults as Provisioned/Deleted - as such whilst we are hitting the ListBySubscription endpoint which (from memory) has a lower rate-limit - since we're fully populating the cache in this implementation means this should only happen one-time.

---

FWIW this is almost the same approach that we use for Storage Accounts too, now that the caching issue has been resolved.

Notes

• Supersedes azurerm_key_vault: use Resource Graph API to search keyvault resource from keyvault name #23413
• Supersedes azurerm_key_vault_*: add TryCacheKeyvaultID function to speed up get keyvault id from keyvault URL #23405
• Should fix azurem_key_vault_key: Refreshing state with context deadline exceeded #23404
• Should fix Getting "Unable to determine the Resource ID for the Key Vault at URL "https://vaultname.vault.azure.net/" #20283
• Should fix Unable to determine the Resource ID for the Key Vault at URL #19322

[jackofallops]

Thanks @tombuildsstuff - Just a couple goimports to fix-up, but otherwise LGTM 👍

[asifkd012020]

Does this feature solve #9738 ?

[phoehnel]

@AndreasMWalter guess you'll like this one ;-)

[github-actions]

I'm going to lock this pull request because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active contributions.
If you have found a problem that seems related to this change, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.