Announcement: v3.0.0 of the Azure Provider (Now Available)

[opslivia]

Announcement: Upcoming v3 of the Azure Provider

Version 3.0 of the Azure Provider will be a major release, this means this will include breaking changes - in addition to new features and enhancements available in regular releases.

During the development of v2.0 of the Azure Provider we used Feature Toggling to allow us to continue shipping releases (to support the many new capabilities of Azure) while working on major features and changes for the 2.0 release. This approach worked well, allowing us to iteratively add new features for 2.0 and subsequently allowing users to try these out - which provided invaluable feedback. Subsequently, we’re using the same approach for 3.0.

Opting into the Beta

The 3.0 Beta is now available! Complete instructions for opting in here.

Pinning your Provider Version

You can pin the version of the Provider using the required_providers block in Terraform 0.13 and later. Note: earlier versions of Terraform require a different approach; however, we’d recommend upgrading to Terraform 1.0.

terraform {
  required_providers {
    azurerm = {
      source  = "hashicorp/azurerm"
      version = "= 2.98.0"
    }
  }
}

.. or to any 2.x release:

terraform {
  required_providers {
    azurerm = {
      source  = "hashicorp/azurerm"
      version = "~> 2.0"
    }
  }
}

More information on how to pin the version of a Terraform Provider can be found on the Terraform Website.

What’s Planned for 3.0?

While there’s many features will make up v3 of the Azure Provider, we’re looking to add support for the following new major features:

• New Data Sources and Resources for App Service / Function Apps and their supporting resources.
• Key Vault - Soft Delete Recovery/Purging for Certificates, Keys and Secrets.
• Switching to Microsoft Graph instead of the deprecated Azure Active Directory Graph API. Includes using MSAL instead of ADAL for authentication.

In addition - since this is a major version of the Azure Provider - we intend to update some existing behaviors, including:

• Application Gateway
  • We'll be updating the behavior of the nested items to be Sets instead of Lists where required, meaning that the order of these items no longer matters. Note that if you're referencing these nested items within your Terraform Configuration, then this may require some code changes.
• API Management
  • Terraform will now remove the Default API and Products for API Management when creating a new API Management instance, which is consistent with the behavior for other Terraform Providers.
• Resource Groups
  • Terraform will now check for Resources nested within a Resource Group prior to deletion of the resource group. If any items are found, an error will be raised. This behavior is configurable today in the features block, but disabled by default. In 3.0, this behavior will be enabled by default.
• Storage
  • We’ll switch to using Resource Manager (management plane) for all resources. (the required APIs have not yet been updated)
  • Switching to use AzureAD / Resource Manager for nested item configuration - this allows provisioning these when Private Link is enabled, however requires different permissions which are not granted by default. (the required APIs have not yet been updated)
  • The field allow_blob_public_access will be renamed to allow_nested_items_to_be_public to resolve confusion about what this field does. This field specifies whether items within the Storage Account (such as Containers and Blobs) can opt-in to being made public (for example at the Container or Blob level) - and not that all resources within this Storage Account are public by default.
• Behavioral changes:
  • All Resources: the Resource ID will now be validated at Import time to ensure the correct resource is being imported, and return the expected format upon a mismatch. While we do this for most resources today, we’ll be doing this for everything going forward. This ensures that, for example, a Virtual Machine ID is specified rather than the VM Extension ID (which is nested under a Virtual Machine ID).
  • All Resources which use min_tls_version: updating the default minimum TLS version to be 1.2.
  • Resources with a (Availability) zones field: updating the behavior to be consistent across the Provider.
    • Availability Zones will be represented across all resources which support it using the field zones - in addition the values No-Zone and ZoneRedundant will be removed in favour of being explicit.
    • Where this field is Optional and Computed - this field will no longer be Computed. This means that if you wish to use the Availability Zone defaulted by Azure, you will need to use Terraform’s ignore_changes functionality to ignore changes to this field.
    • Where this field is Optional (and isn’t defaulted by Azure) - omitting this field (or specifying a value of null) will deploy this without any Zones.
    • Where this field is Required, there are no changes.
    • Zone Redundant resources can be provisioned by specifying all of the Availability Zones for that particular Azure Region. We’ll also introduce a new azurerm_availability_zones data source to provide this information.
  • Resources with a (Managed) identity block: updating the behavior to be consistent across the Provider.
    • This means that the presence of an identity block means a Managed Identity should be assigned to this Resource - and the omission of an identity block (or a null value) means that no Managed Identity should be assigned to this Resource.

Feature details: New Data Sources / Resources for App Service & Function Apps

While it’s possible to provision App Services and Function Apps in Terraform today, the design and behavior of the App Service platform has changed over the years. These resources require some refinement.

Similar to the changes for Virtual Machines in v2.0 of the Azure Provider, we’re intending to introduce more granular resources for App Service to better represent the functionality available in Azure.

Some of these new Data Sources and Resources are available as an opt-in Beta today from version 2.79 of the Azure Provider onwards - and we encourage you to try them out.

While we don’t anticipate making changes to these resources at this time - since these are in Beta, it’s possible. As such, we recommend not using this Beta in a Production environment at this time.

More information on the specific changes for App Service / Function Apps can be found in the documentation

Feature details: Key Vault - Soft Delete Recovery/Purging for Certificates, Keys and Secrets

Previously, soft delete has only been available for a Key Vault resource as a whole. Now, you’ll be able to soft delete the nested items within a Key Vault: certificates, keys, and secrets.

Feature details: Switching to Microsoft Graph

The AzureRM provider needs to query the graph API in order to retrieve important information about the principal account it’s using to authenticate. We currently use the Azure Active Directory Graph API for this, but this API is deprecated and has been replaced by Microsoft Graph, so in version 3.0 of the provider we’ll be switching to the newer API. When using service principal authentication, this will require changes to API permissions that are granted to that principal and this will be covered in our upgrade guide.

We’ll also be enabling Microsoft Graph via an opt-in beta prior to version 3.0 and we encourage you to try this when it’s available, as part of your preparation for the new major version.

Feature details: Switching to use MSAL for authentication instead of ADAL

Authentication to APIs such as Resource Manager is currently performed using the ADAL library which yields legacy v1 authentication tokens. We’ll move to use v2 tokens in version 3.0 of the provider. In practice this change will not yield any noticeable behavioral differences; however, since this underpins the way the provider authenticates to Azure services, we’ll be making this change in a major release.

Frequently Asked Questions

Why is the behavior of Resource Groups changing?

Most resources in Azure are provisioned within a Resource Group, which (amongst other things) allows for easy grouping and allows scoping permissions to resources within this Resource Group.

The default behavior of the Azure Platform (and Terraform today) when deleting a Resource Group is to delete all of the items within this Resource Group.

While this feature makes it easy to clean-up any resources - since resources can be provisioned outside of Terraform, it means that these Resources are untracked by Terraform - and so Terraform is unable to show these Resources in the Plan during the deletion of the Resource Group.

Terraform has a feature-flag for this behavior available within the Features block today - which will check during the deletion of the Resource Group for any Resources within this Resource Group, and raise an error (stating these must be removed first) if any Resources are found. In v2.x versions of the Azure Provider this is disabled by default - however v3.0 will flip this default to be enabled by default.

This behavior can be configured using the features block as shown below, both in v2.x and v3.0 of the Azure Provider:

provider "azurerm" {
  features {
    resource_group {
      prevent_deletion_if_contains_resources = false
    }
  }
}

resource "azurerm_resource_group" "example" {
  name = "example-resources"
  location = "West Europe"
}

Setting this value to true will check for nested Resources during deletion of the Resource Group; setting this to false will skip this check.

When will v3 of the Azure Provider be available?

Version 3.0 of the Azure Provider is being built iteratively using Feature Toggles within the v2.x codebase. This means that functionality will be added over time until the 3.0 Release is completed.

This Feature Toggle approach has been used during the development of v2.0 of the Azure Provider and worked well - this allows users to opt-into functionality coming in v3.0 of the Provider in a 2.x release.

Since this is being worked on iteratively and the majority of behavior can be enabled prior release, v3.0 of the Azure Provider will launch when it’s features have been completed, tested, and sufficient time has pass to allow for feedback - currently we’re targeting early 2022 (however this is subject to change).

How can I opt into the Beta?

The 3.0 Beta is now available! Complete instructions for opting in here.

Note that functionality in the Beta is subject to change, including breaking changes - so while we’d appreciate your feedback, we’d recommend against using this in a production environment at this point in time.

Disclosures

The product-development initiatives in this document reflect HashiCorp's current plans and are subject to change and/or cancellation in HashiCorp's sole discretion.

[russellhart]

Is there a discussion for allow_blob_public_access changes? The new name, allow_nested_items_to_be_public, is also confusing. allow_nested_items_to_have_anonymous_access would have more clarity as I also see this causing confusion. Public implies connectivity from the internet but this setting controls allowing anonymous or authenticated access.

[tombuildsstuff]

@russellhart yeah we can potentially update the name for that field, although I'm still not sure that allow_nested_items_to_have_anonymous_access covers the behaviour of that field fully, since it allows nested items to opt-into being public? The challenge is picking something that's clear but succinct, so we're definitely open to changes, it's just finding the right name 😅

[oWretch]

Along with the move to Sets for Application Gateway, can the same change be done for rules blocks inside Firewall Application/Network/NAT rule collection groups? Happy to assist with the work.

[oWretch]

Thoughts on the mentioned azurerm_availability_zones data source - would it be possible (worthwhile) to do a more generic azurerm_regions data source that can be used to gather information on the various regional settings, similar to azuread_application_published_app_ids data source?

That way, zones could be accessed using data.azurerm_regions.regions.result["useast"].availability_zones

[tombuildsstuff]

👋

Last week's release of the Azure Provider (v2.97) contains the ability to opt-into 3.0 Beta using a feature-flag, details can be found here: https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/guides/3.0-beta and the (WIP) upgrade guide can be found here: https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/guides/3.0-upgrade-guide

[djryanj]

Can you comment on the rationale behind changing the lists to sets for Application Gateway? I presume it's similar in nature to what's discussed here (in that case, with respect to the AWS provider, and specifically, that the backend API doesn't actually order those configuration items so having e.g., a zeroth item in a list doesn't make sense). Regardless, I feel like this is actually a step backwards; finding things in sets seems somewhat difficult and makes writing modules difficult. As an alternative, I propose #15745 which I realize might be difficult given the timelines and likely lack of any development towards that. If I thought I could contribute I would and may, but I won't be able to provide anything meaningful for a while.

[tombuildsstuff]

@djryanj see this issue for more context on the Application Gateway List to Set changes - #6896

As mentioned in #15745 unfortunately we can't split the Application Gateway components into their own Resources (due to the API behaviour/conflicts) - but we could look to add data sources for the various components if that'd be helpful? From memory you should be able to use Terraform Core's tolist functionality on the output to convert the Set to a List to query the items within it - however that'll reintroduce the issue found in #6896 where the ordering can differ to any configurations.

[djryanj]

@tombuildsstuff - that makes sense, thanks for clarifying. I suppose the awkwardness of working with a Set is probably better than the alternative mentioned in those other issues.

My primary issue is around later using a azurerm_network_interface_application_gateway_backend_address_pool_association resource, which requires a backend_address_pool_id, which is difficult to obtain programmatically (e.g., via a backend pool name) as it stands. Data resources might help here, but that would force any apply to be done in two stages or two modules/configs. If you can think of another improvement that might help there, I'm all ears.

[tombuildsstuff]

👋 hey folks

Over the past few months we've been working to integrate the changes mentioned above into the Provider and we're now at a point where we've finished up all of the major changes needed for version 3.0 of the Azure Provider.

Thanks for all of the feedback around 3.0 - both above and from the 3.0 and App Service Betas - it's been really helpful and (particularly with App Service) has helped inform the design and catch a number of bugs.

Since this is a meta issue tracking the changes coming in 3.0 and those changes are now complete - I'm going to close this issue, however as this issue is assigned to the 3.0 Milestone when the 3.0 release is available a comment will be posted to let you know.

Thanks!

[github-actions]

This functionality has been released in v3.0.0 of the Terraform Provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template. Thank you!

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.