feat: Optimized deployment resolution runtime to reduce unnecessary w…

…ait time (#2461)

## Description

- Changed the way the removal logic tries to resolve deployment names to
their actual resources
- Up until now the logic was as follows
- One deployment name after another (up to 3 as per 3 retries) where
passed into the script
- The script would then, for each deploymentname respectively search for
the deployment
  - If it didn't find it, it would retry after one minute up to 40 times
- This meant, for 3 failed deployments that fail in a way that they
don't create a deployment (e.g. in case the template failed already the
initial validation, it would wait 3 x 40 minutes = 120 minutes
- This, updated approach works as follows
  - All 3 deployment names are passed into the script
  - For each round of trying, it searches for all 3 deployment names
- If at least one isn't found, it will try again, as before after one
minute to a maximum of 40 times
  - Each found name will be skript by the retries
- This way, the script in its entirety only works 40 minutes in total
and as such will make sure it will definitely wait a maximum of 40
minutes for each deployment to show up

> Note: To test a failing deployment, I used a long-running deployment
script which I would maniupulate in Azure during deployment, and also
delete one of the retries before the removal would start. This is why in
the log you can see that the removal logic finds attempts 2 & 3 but not
1, which is what was intended.
>```bicep
> // In dependencies.bicep of WAF-aligned test
>resource deploymentScript
'Microsoft.Resources/deploymentScripts@2023-08-01' = {
>  name: 'testScript'
>  location: location
>  identity: {
>    type: 'UserAssigned'
>    userAssignedIdentities: {
>      '${managedIdentity.id}': {}
>    }
>  }
>  kind: 'AzurePowerShell'
>  properties: {
>    azPowerShellVersion: '9.7'
>    retentionInterval: 'P1D'
>    scriptContent: 'Start-Sleep 180'
>  }
>}
>```

## Pipeline Reference

<!-- Insert your Pipeline Status Badge below -->

| Pipeline |
| -------- |
|
[![avm.res.network.virtual-network](https://github.com/Azure/bicep-registry-modules/actions/workflows/avm.res.network.virtual-network.yml/badge.svg?branch=users%2Falsehr%2FdeploymentWaitOptimization&event=workflow_dispatch)](https://github.com/Azure/bicep-registry-modules/actions/workflows/avm.res.network.virtual-network.yml)
(failed deployment test) |
|
[![avm.res.key-vault.vault](https://github.com/Azure/bicep-registry-modules/actions/workflows/avm.res.key-vault.vault.yml/badge.svg?branch=users%2Falsehr%2FdeploymentWaitOptimization&event=workflow_dispatch)](https://github.com/Azure/bicep-registry-modules/actions/workflows/avm.res.key-vault.vault.yml)
(successful deployment test) |

Test run with a faulty module (VNET):
https://github.com/Azure/bicep-registry-modules/actions/runs/9613612546
Test run with a working module (KeyVault):
https://github.com/Azure/bicep-registry-modules/actions/runs/9536144913

## Type of Change

<!-- Use the checkboxes [x] on the options that are relevant. -->

- [x] Update to CI Environment or utilities (Non-module affecting
changes)
- [ ] Azure Verified Module updates:
- [ ] Bugfix containing backwards-compatible bug fixes, and I have NOT
bumped the MAJOR or MINOR version in `version.json`:
- [ ] Someone has opened a bug report issue, and I have included "Closes
#{bug_report_issue_number}" in the PR description.
- [ ] The bug was found by the module author, and no one has opened an
issue to report it yet.
- [ ] Feature update backwards compatible feature updates, and I have
bumped the MINOR version in `version.json`.
- [ ] Breaking changes and I have bumped the MAJOR version in
`version.json`.
  - [ ] Update to documentation

## Example log: **Successful deployments run**
```
  VERBOSE: Invoke task with
  VERBOSE: {
    "SubscriptionId": "***",
    "ManagementGroupId": "***",
    "TemplateFilePath": "/home/runner/work/bicep-registry-modules/bicep-registry-modules/avm/res/key-vault/vault/tests/e2e/waf-aligned/main.test.bicep",
    "DeploymentNames": "a-r-kv-v-waf-aligned-t1-20240616T1206450259Z"
  }
  
  VERBOSE: Handling resource removal with deployment names
  VERBOSE: - a-r-kv-v-waf-aligned-t1-20240616T1206450259Z
  VERBOSE: Total number of deployment target resources after fetching deployments [18]
  VERBOSE: Total number of deployment target resources after pre-filtering (duplicates) & ordering items [18]
  VERBOSE: Total number of deployment target resources after formatting items [18]
  VERBOSE: Total number of deployments after filtering all dependency resources [18]
  VERBOSE: Total number of deployments after final ordering of resources [18]
  VERBOSE: - Remove [/subscriptions/***/resourceGroups/dep-***-keyvault.vaults-kvvwaf-rg/providers/Microsoft.KeyVault/vaults/***kvvwaf002/providers/Microsoft.Authorization/locks/myCustomLockName]
```

## Example log: **Failed deployments run**
```
  VERBOSE: Invoke task with
  VERBOSE: {
    "ManagementGroupId": "***",
    "SubscriptionId": "***",
    "DeploymentNames": [
      "a-r-n-vn-waf-aligned-t1-20240621T1206370696Z",
      "a-r-n-vn-waf-aligned-t2-20240621T1206433769Z",
      "a-r-n-vn-waf-aligned-t3-20240621T1206013450Z"
    ],
    "TemplateFilePath": "/home/runner/work/bicep-registry-modules/bicep-registry-modules/avm/res/network/virtual-network/tests/e2e/waf-aligned/main.test.bicep"
  }
  
  VERBOSE: Handling resource removal with deployment names
  VERBOSE: - a-r-n-vn-waf-aligned-t1-20240621T1206370696Z
  VERBOSE: - a-r-n-vn-waf-aligned-t2-20240621T1206433769Z
  VERBOSE: - a-r-n-vn-waf-aligned-t3-20240621T1206013450Z
  VERBOSE: Found & resolved deployment [a-r-n-vn-waf-aligned-t2-20240621T1206433769Z]
  VERBOSE: Found & resolved deployment [a-r-n-vn-waf-aligned-t3-20240621T1206013450Z]
  VERBOSE: No deployment found by name(s) [@{Name=a-r-n-vn-waf-aligned-t1-20240621T1206370696Z}] in scope [subscription]. Retrying in [60] seconds [1/40]
  VERBOSE: No deployment found by name(s) [@{Name=a-r-n-vn-waf-aligned-t1-20240621T1206370696Z}] in scope [subscription]. Retrying in [60] seconds [2/40]
```

avm/utilities/pipelines/e2eValidation/resourceRemoval/helper/Get-DeploymentTargetResourceList.ps1

@@ -188,7 +188,8 @@ function Get-DeploymentTargetResourceList {
  [string] $ManagementGroupId,
 
  [Parameter(Mandatory = $true)]
- [string] $Name,
+ [Alias('Name', 'Names', 'DeploymentName')]
+ [string[]] $DeploymentNames,
 
  [Parameter(Mandatory = $true)]
  [ValidateSet(
@@ -207,31 +208,57 @@ function Get-DeploymentTargetResourceList {
  )
 
  $searchRetryCount = 1
- do {
-  $innerInputObject = @{
-  Name = $name
- Scope  = $scope
- ErrorAction = 'SilentlyContinue'
+ $resourcesToRemove = @()
+ $deploymentNameObjects = $DeploymentNames | ForEach-Object {
+ @{
+ Name = $_
+ Resolved = $false
  }
- if (-not [String]::IsNullOrEmpty($resourceGroupName)) {
- $innerInputObject['resourceGroupName'] = $resourceGroupName
- }
- if (-not [String]::IsNullOrEmpty($ManagementGroupId)) {
- $innerInputObject['ManagementGroupId'] = $ManagementGroupId
+ }
+
+ do {
+ foreach ($deploymentNameObject in $deploymentNameObjects) {
+
+ if ($deploymentNameObject.Resolved) {
+ # Skip further invocations for this deployment name if deployment was already found
+ continue
+ }
+
+ $innerInputObject = @{
+ Name = $deploymentNameObject.Name
+ Scope = $scope
+ ErrorAction = 'SilentlyContinue'
+ }
+ if (-not [String]::IsNullOrEmpty($resourceGroupName)) {
+ $innerInputObject['resourceGroupName'] = $resourceGroupName
+ }
+ if (-not [String]::IsNullOrEmpty($ManagementGroupId)) {
+ $innerInputObject['ManagementGroupId'] = $ManagementGroupId
+ }
+
+ [array]$targetResources = Get-DeploymentTargetResourceListInner @innerInputObject
+ if ($targetResources.Count -gt 0) {
+ Write-Verbose ('Found & resolved deployment [{0}]' -f $deploymentNameObject.Name) -Verbose
+ $deploymentNameObject.Resolved = $true
+ $resourcesToRemove += $targetResources
+ }
  }
- [array]$targetResources = Get-DeploymentTargetResourceListInner @innerInputObject
- if ($targetResources) {
+
+ # Break check
+ if ($deploymentNameObjects.Resolved -notcontains $false) {
  break
  }
- Write-Verbose ('No deployment found by name [{0}] in scope [{1}]. Retrying in [{2}] seconds [{3}/{4}]' -f $name, $scope, $searchRetryInterval, $searchRetryCount, $searchRetryLimit) -Verbose
+
+ $remainingDeploymentNames = ($deploymentNameObjects | Where-Object { -not $_.Resolved }).Name
+ Write-Verbose ('No deployment found by name(s) [{0}] in scope [{1}]. Retrying in [{2}] seconds [{3}/{4}]' -f ($remainingDeploymentNames -join ', '), $scope, $searchRetryInterval, $searchRetryCount, $searchRetryLimit) -Verbose
  Start-Sleep $searchRetryInterval
  $searchRetryCount++
  } while ($searchRetryCount -le $searchRetryLimit)
 
- if (-not $targetResources) {
- Write-Warning "No deployment target resources found for [$name]"
+ if (-not $resourcesToRemove) {
+ Write-Warning ('No deployment target resources found for [{0}]' -f ($DeploymentNames -join ', '))
  return @()
  }
 
- return $targetResources
+ return $resourcesToRemove
 }

avm/utilities/pipelines/e2eValidation/resourceRemoval/helper/Remove-Deployment.ps1

@@ -90,20 +90,17 @@ function Remove-Deployment {
 
  # Fetch deployments
  # =================
-
- foreach ($deploymentName in $DeploymentNames) {
- $deploymentsInputObject = @{
- Name = $deploymentName
- Scope = $deploymentScope
- }
- if (-not [String]::IsNullOrEmpty($ResourceGroupName)) {
- $deploymentsInputObject['resourceGroupName'] = $ResourceGroupName
- }
- if (-not [String]::IsNullOrEmpty($ManagementGroupId)) {
- $deploymentsInputObject['ManagementGroupId'] = $ManagementGroupId
- }
- $deployedTargetResources += Get-DeploymentTargetResourceList @deploymentsInputObject
+ $deploymentsInputObject = @{
+ DeploymentNames = $DeploymentNames
+ Scope = $deploymentScope
+ }
+ if (-not [String]::IsNullOrEmpty($ResourceGroupName)) {
+ $deploymentsInputObject['resourceGroupName'] = $ResourceGroupName
+ }
+ if (-not [String]::IsNullOrEmpty($ManagementGroupId)) {
+ $deploymentsInputObject['ManagementGroupId'] = $ManagementGroupId
  }
+ $deployedTargetResources += Get-DeploymentTargetResourceList @deploymentsInputObject
 
  if ($deployedTargetResources.Count -eq 0) {
  throw 'No deployment target resources found.'