Adding Support for Single Template with Multiple Parameter Files (#836)

* BaseLogicUpdate * Update * Update * Update * Update * Update * Update * Update * Update * Update * Update * UpdateStringAndMd * Update * UpdateLogic
Azure · Dec 19, 2023 · a260d0b · a260d0b
1 parent 7c13d2d
commit a260d0b
Show file tree

Hide file tree

Showing 16 changed files with 294 additions and 34 deletions.
diff --git a/docs/wiki/Frequently-Asked-Questions.md b/docs/wiki/Frequently-Asked-Questions.md
@@ -15,6 +15,10 @@ This article answers frequently asked questions relating to AzOps.
     - [**I want to discover all resources in specific resource groups in one specific subscription**](#i-want-to-discover-all-resources-in-specific-resource-groups-in-one-specific-subscription)
     - [**I want to discover a specific resource type in specific resource group in one specific subscription**](#i-want-to-discover-a-specific-resource-type-in-specific-resource-group-in-one-specific-subscription)
     - [**I want to discover and manage several Azure Firewall Policy's and rule collections spread out across several resource groups and subscriptions**](#i-want-to-discover-and-manage-several-azure-firewall-policys-and-rule-collections-spread-out-across-several-resource-groups-and-subscriptions)
+  - [Push scenarios and settings](#push-scenarios-and-settings)
+    - [**I want to have multiple different deployments at scope using the same template file but different parameter files**](#i-want-to-have-multiple-different-deployments-at-scope-using-the-same-template-file-but-different-parameter-files)
+    - [**I have AllowMultipleTemplateParameterFiles set to true and when changes are made to a template no deployment is performed**](#i-have-allowmultipletemplateparameterfiles-set-to-true-and-when-changes-are-made-to-a-template-no-deployment-is-performed)
+    - [**I am getting: Missing defaultValue and no parameter file found, skip deployment**](#i-am-getting-missing-defaultvalue-and-no-parameter-file-found-skip-deployment)
 
 ## Subscriptions or resources not showing up in repository
 
@@ -145,5 +149,64 @@ Yes, ensure the following setting combinations are applied (replace `rgname1`, `
 
 Can AzOps settings be configured to enable this?
 
-Yes, ensure that the variable `AZOPS_CUSTOM_SORT_ORDER` is set to `true` and create a file named `.order` in the same folder as your template files.  
+Yes, ensure that the variable `AZOPS_CUSTOM_SORT_ORDER` is set to `true` and create a file named `.order` in the same folder as your template files.
 Template files listed in the order file will be deployed in the order specified in the file and before any other templates.
+
+## Push scenarios and settings
+
+### **I want to have multiple different deployments at scope using the same template file but different parameter files**
+
+When using custom deployment templates, can I avoid the pattern of duplicating the `.bicep` file for each `parameter` file below?
+```bash
+scope/
+├── template-a.bicep
+├── template-a.bicepparam
+├── template-b.bicep
+├── template-b.bicepparam
+├── template-c.bicep
+└── template-c.parameters.json
+```
+Yes, ensure the following setting combinations are applied (replace `x` with your specific pattern identifier)
+
+```bash
+    "Core.AllowMultipleTemplateParameterFiles": true
+
+    "Core.MultipleTemplateParameterFileSuffix": ".x"
+```
+AzOps module will evaluate each parameter file individually and try to find base template by matching (*regular expression*) according to `MultipleTemplateParameterFileSuffix` pattern identifier.
+```bash
+scope/
+├── template.x1.bicepparam
+├── template.x2.bicepparam
+├── template.x3.parameters.json
+└── template.bicep
+```
+> Note: To avoid having AzOps deploy the base `template.bicep` unintentionally, ensure you have at least one parameter without default value in `template.bicep` and no lingering 1:1 matching parameter file.
+
+### **I have AllowMultipleTemplateParameterFiles set to true and when changes are made to a template no deployment is performed**
+
+When using a custom deployment templates with multiple corresponding parameter files, can I ensure that changes made to the template triggers AzOps to create separate deployments for each corresponding parameter file?
+
+Yes, ensure the following setting `Core.DeployAllMultipleTemplateParameterFiles` is set to `true`.
+
+> Note: By default, AzOps does not try to identify and deploy files that have not changed, by changing this setting AzOps will attempt to resolve matching parameter files for deployment based on deployment template.
+
+### **I am getting: Missing defaultValue and no parameter file found, skip deployment**
+
+To confirm if this applies to you, check the pipeline logs for the following message:
+
+```powershell
+[Resolve-ArmFileAssociation] Template <filepath> with parameter: <missingparam>, missing defaultValue and no parameter file found, skip deployment
+```
+
+What does this mean?
+
+AzOps have detected that parameters used in the template do not have defaultValues, no 1:1 parameter file mapped and that `Core.AllowMultipleTemplateParameterFiles` is set to `true`.
+
+To avoid exiting with error or attempt to deploy the updated base template unintentionally AzOps skips the file and logs it.
+
+The following must be true for this to happen:
+- `Core.AllowMultipleTemplateParameterFiles` is set to `true`
+- A template file is a part of the changeset sent to AzOps
+- Template file contains parameters with no defaultValue
+- Template file does not have 1:1 mapping to parameter file
diff --git a/docs/wiki/Settings.md b/docs/wiki/Settings.md
@@ -35,8 +35,11 @@ The following configuration values can be modified within the `settings.json` fi
 | 23  | State                                | Folder to store AzOpsState artefact, defaults to `root`                                                                                                                               | `"Core.State: "/root"`                                                                          |
 | 24  | SubscriptionsToIncludeResourceGroups | Filter which Subscription IDs should include Resource Groups in pull [Logic Updated in v2.0.0](https://github.com/Azure/AzOps/releases/tag/2.0.0)                                     | `"Core.SubscriptionsToIncludeResourceGroups": ["*"]`                                            |
 | 25  | TemplateParameterFileSuffix          | Default template file suffix. *Not recommended to change*                                                                                                                             | `"Core.TemplateParameterFileSuffix": ".json"`                                                   |
-| 26  | ThrottleLimit                        | Value declaring number of parallel threads. [Read more](https://github.com/azure/azops/wiki/performance-considerations)                                                               | `"Core.ThrottleLimit": 5`                                                                       |
-| 27  | WhatifExcludedChangeTypes            | Exclude specific change types from WhatIf operations                                                                                                                                  | `"Core.WhatifExcludedChangeTypes": ["NoChange","Ignore"]`                                       |
+| 26  | AllowMultipleTemplateParameterFiles          | Control multiple parameter file behaviour. *Not recommended to change*                                                                                                                             | `"Core.AllowMultipleTemplateParameterFiles": false`                                                   |
+| 27  | DeployAllMultipleTemplateParameterFiles          | Control base template deployment behaviour with changes and un-changed multiple corresponding parameter files.                                                                                                                             | `"Core.DeployAllMultipleTemplateParameterFiles": false`                                                   |
+| 28  | MultipleTemplateParameterFileSuffix          | Multiple parameter file suffix identifier. *Example mytemplate.x1.bicepparam*                                                                                                                             | `"Core.MultipleTemplateParameterFileSuffix": ".x"`                                                   |
+| 29  | ThrottleLimit                        | Value declaring number of parallel threads. [Read more](https://github.com/azure/azops/wiki/performance-considerations)                                                               | `"Core.ThrottleLimit": 5`                                                                       |
+| 30  | WhatifExcludedChangeTypes            | Exclude specific change types from WhatIf operations                                                                                                                                  | `"Core.WhatifExcludedChangeTypes": ["NoChange","Ignore"]`                                       |
 
 ## Workflow / Pipeline Settings
 

diff --git a/src/functions/Initialize-AzOpsEnvironment.ps1 b/src/functions/Initialize-AzOpsEnvironment.ps1
@@ -62,6 +62,13 @@
             Stop-PSFFunction -String 'Initialize-AzOpsEnvironment.AzureContext.TooMany' -StringValues $azContextTenants.Count, ($azContextTenants -join ',') -EnableException $true -Cmdlet $PSCmdlet
         }
 
+        # Adjust MultipleTemplateParameterFileSuffix if incorrect MultipleTemplateParameterFileSuffix is set and log warning
+        if (-not $(Get-PSFConfigValue -FullName 'AzOps.Core.MultipleTemplateParameterFileSuffix').StartsWith('.')) {
+            $updateMultipleTemplateParameterFileSuffix = ".$(Get-PSFConfigValue -FullName 'AzOps.Core.MultipleTemplateParameterFileSuffix')"
+            Write-PSFMessage -Level Warning -String 'Initialize-AzOpsEnvironment.MultipleTemplateParameterFileSuffix.Adjustment' -StringValues (Get-PSFConfigValue -FullName 'AzOps.Core.MultipleTemplateParameterFileSuffix'), $updateMultipleTemplateParameterFileSuffix
+            Set-PSFConfig -Module AzOps -Name Core.MultipleTemplateParameterFileSuffix -Value $updateMultipleTemplateParameterFileSuffix
+        }
+
         # Adjust ThrottleLimit from previously default 10 to 5 if system has less than 2 cores
         [int]$cpuCores = if ($IsWindows) { $env:NUMBER_OF_PROCESSORS } else { Invoke-AzOpsNativeCommand -ScriptBlock { nproc --all } -IgnoreExitcode }
         $throttleLimit = (Get-PSFConfig -Module AzOps -Name Core.ThrottleLimit).Value

diff --git a/src/functions/Invoke-AzOpsPush.ps1 b/src/functions/Invoke-AzOpsPush.ps1
@@ -92,25 +92,38 @@
                 #region Directly Associated Template file exists
                 switch ($fileItem.Name) {
                     { $_.EndsWith('.parameters.json') } {
-                        $templatePath = $fileItem.FullName -replace '\.parameters.json', (Get-PSFConfigValue -FullName 'AzOps.Core.TemplateParameterFileSuffix')
-                        $bicepTemplatePath = $fileItem.FullName -replace '.parameters.json', '.bicep'
+                        if ((Get-PSFConfigValue -FullName 'AzOps.Core.AllowMultipleTemplateParameterFiles') -eq $true -and $fileItem.FullName.Split('.')[-3] -match $(Get-PSFConfigValue -FullName 'AzOps.Core.MultipleTemplateParameterFileSuffix').Replace('.','')) {
+                            Write-PSFMessage -Level Verbose @common -String 'Invoke-AzOpsPush.Resolve.MultipleTemplateParameterFile' -StringValues $FilePath
+                            $templatePath = $fileItem.FullName -replace (".$($fileItem.FullName.Split('.')[-3])"), '' -replace '\.parameters.json', '.json'
+                            $bicepTemplatePath = $fileItem.FullName -replace (".$($fileItem.FullName.Split('.')[-3])"), '' -replace '.parameters.json', '.bicep'
+                        }
+                        else {
+                            $templatePath = $fileItem.FullName -replace '\.parameters.json', (Get-PSFConfigValue -FullName 'AzOps.Core.TemplateParameterFileSuffix')
+                            $bicepTemplatePath = $fileItem.FullName -replace '.parameters.json', '.bicep'
+                        }
                         if (Test-Path $templatePath) {
                             Write-PSFMessage -Level Verbose @common -String 'Invoke-AzOpsPush.Resolve.FoundTemplate' -StringValues $FilePath, $templatePath
                             $result.TemplateFilePath = $templatePath
                             return $result
                         }
                         elseif (Test-Path $bicepTemplatePath) {
                             Write-PSFMessage -Level Verbose @common -String 'Invoke-AzOpsPush.Resolve.FoundBicepTemplate' -StringValues $FilePath, $bicepTemplatePath
-                            $transpiledTemplatePaths = ConvertFrom-AzOpsBicepTemplate -BicepTemplatePath $bicepTemplatePath
+                            $transpiledTemplatePaths = ConvertFrom-AzOpsBicepTemplate -BicepTemplatePath $bicepTemplatePath -SkipParam
                             $result.TemplateFilePath = $transpiledTemplatePaths.transpiledTemplatePath
                             return $result
                         }
                     }
                     { $_.EndsWith('.bicepparam') } {
-                        $bicepTemplatePath = $fileItem.FullName -replace '\.bicepparam', '.bicep'
+                        if ((Get-PSFConfigValue -FullName 'AzOps.Core.AllowMultipleTemplateParameterFiles') -eq $true -and $fileItem.FullName.Split('.')[-2] -match $(Get-PSFConfigValue -FullName 'AzOps.Core.MultipleTemplateParameterFileSuffix').Replace('.','')) {
+                            Write-PSFMessage -Level Verbose @common -String 'Invoke-AzOpsPush.Resolve.MultipleTemplateParameterFile' -StringValues $FilePath
+                            $bicepTemplatePath = $fileItem.FullName -replace (".$($fileItem.FullName.Split('.')[-2])"), '' -replace '\.bicepparam', '.bicep'
+                        }
+                        else {
+                            $bicepTemplatePath = $fileItem.FullName -replace '\.bicepparam', '.bicep'
+                        }
                         if (Test-Path $bicepTemplatePath) {
                             Write-PSFMessage -Level Verbose @common -String 'Invoke-AzOpsPush.Resolve.FoundBicepTemplate' -StringValues $FilePath, $bicepTemplatePath
-                            $transpiledTemplatePaths = ConvertFrom-AzOpsBicepTemplate -BicepTemplatePath $bicepTemplatePath
+                            $transpiledTemplatePaths = ConvertFrom-AzOpsBicepTemplate -BicepTemplatePath $bicepTemplatePath -BicepParamTemplatePath $fileItem.FullName
                             $result.TemplateFilePath = $transpiledTemplatePaths.transpiledTemplatePath
                             $result.TemplateParameterFilePath = $transpiledTemplatePaths.transpiledParametersPath
                             return $result
@@ -162,8 +175,41 @@
                 Write-PSFMessage -Level Verbose @common -String 'Invoke-AzOpsPush.Resolve.ParameterFound' -StringValues $FilePath, $parameterPath
                 $result.TemplateParameterFilePath = $parameterPath
             }
+            elseif ((Get-PSFConfigValue -FullName 'AzOps.Core.AllowMultipleTemplateParameterFiles') -eq $true -and (Get-PSFConfigValue -FullName 'AzOps.Core.DeployAllMultipleTemplateParameterFiles') -eq $true) {
+                # Check for multiple associated template parameter files
+                $paramFileList = Get-ChildItem -Path $fileItem.Directory | Where-Object { ($_.Name.Split('.')[-3] -match $(Get-PSFConfigValue -FullName 'AzOps.Core.MultipleTemplateParameterFileSuffix').Replace('.','')) -or ($_.Name.Split('.')[-2] -match $(Get-PSFConfigValue -FullName 'AzOps.Core.MultipleTemplateParameterFileSuffix').Replace('.','')) }
+                if ($paramFileList) {
+                    $multiResult = @()
+                    foreach ($paramFile in $paramFileList) {
+                        # Process possible parameter files for template equivalent
+                        if (($fileItem.FullName.Split('.')[-2] -eq $paramFile.FullName.Split('.')[-3]) -or ($fileItem.FullName.Split('.')[-2] -eq $paramFile.FullName.Split('.')[-4])) {
+                            Write-PSFMessage -Level Verbose @common -String 'Invoke-AzOpsPush.Resolve.MultipleTemplateParameterFile' -StringValues $paramFile.FullName
+                            $multiResult += Resolve-ArmFileAssociation -ScopeObject $scopeObject -FilePath $paramFile -AzOpsMainTemplate $AzOpsMainTemplate
+                        }
+                    }
+                    if ($multiResult) {
+                        # Return completed object
+                        return $multiResult
+                    }
+                    else {
+                        Write-PSFMessage -Level Verbose @common -String 'Invoke-AzOpsPush.Resolve.ParameterNotFound' -StringValues $FilePath, $parameterPath
+                    }
+
+                }
+            }
             else {
                 Write-PSFMessage -Level Verbose @common -String 'Invoke-AzOpsPush.Resolve.ParameterNotFound' -StringValues $FilePath, $parameterPath
+                if ((Get-PSFConfigValue -FullName 'AzOps.Core.AllowMultipleTemplateParameterFiles') -eq $true) {
+                    # Check for template parameters without defaultValue
+                    $defaultValueContent = Get-Content $FilePath
+                    $missingDefaultParam = $defaultValueContent | jq '.parameters | with_entries(select(.value.defaultValue == null))' | ConvertFrom-Json -AsHashtable
+                    if ($missingDefaultParam.Count -ge 1) {
+                        # Skip template deployment when template parameters without defaultValue are found and no parameter file identified
+                        $missingString = foreach ($item in $missingDefaultParam.Keys.GetEnumerator()) {"$item,"}
+                        Write-PSFMessage -Level Verbose -String 'Invoke-AzOpsPush.Resolve.NotFoundParamFileDefaultValue' -StringValues $FilePath, ($missingString | Out-String -NoNewline)
+                        continue
+                    }
+                }
             }
 
             $deploymentName = $fileItem.BaseName -replace '\.json$' -replace ' ', '_'
@@ -350,8 +396,8 @@
         #Sort 'deletionList' based on 'deletionListPriority'
         $deletionList = $deletionList | Sort-Object -Property {$deletionListPriority.IndexOf($_.ScopeObject.Resource)}
 
-        #If addModifySet exists and no deploymentList has been generated at the same time as the StatePath root has additional directories, exit with terminating error
-        if (($addModifySet -and -not $deploymentList) -and (Get-ChildItem -Path $StatePath -Directory)) {
+        #If addModifySet exists and no deploymentList has been generated at the same time as the StatePath root has additional directories and AllowMultipleTemplateParameterFiles is default false, exit with terminating error
+        if (($addModifySet -and -not $deploymentList) -and (Get-ChildItem -Path $StatePath -Directory) -and ((Get-PSFConfigValue -FullName 'AzOps.Core.AllowMultipleTemplateParameterFiles') -eq $false)) {
             Write-PSFMessage -Level Critical @common -String 'Invoke-AzOpsPush.DeploymentList.NotFound'
             throw
         }