Broken .NET Aspire Azure CI/CD pipeline: error unmarshalling Bicep template parameters

[Dayonel]

• Make sure you've installed the latest version using instructions

Output from azd version
Run azd version and copy and paste the output here:

azd version 1.18.0 (commit 2c3bc5a)

Describe the bug

I encountered this issue after upgrading my .NET Aspire project to 9.3.1 in the ci/cd pipeline (Azure DevOps):

ERROR: initializing provisioning manager: resolving bicep parameters file: error unmarshalling Bicep template parameters: invalid character 'a' after object key:value pair

Unfortunately I tried many things but can't pass this issue.

Last working pipeline used AZD_INITIAL_ENVIRONMENT_CONFIG that now is deprecated.
Locally azd provision and azd deploy works.

To Reproduce
Upgrade an Aspire project from 9.1.0 to 9.3.0.
Create a CI/CD pipeline using this command azd pipeline config --provider azdo
Have this main.parameters.json:

{
    "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentParameters.json#",
    "contentVersion": "1.0.0.0",
    "parameters": {
      "principalId": {
        "value": "${AZURE_PRINCIPAL_ID}"
      },
      "Authorities_AzureAd": {
        "value": "${AZURE_AUTHORITIES_AZURE_AD}"
      },
      "Authorities_EntraExternalId": {
        "value": "${AZURE_AUTHORITIES_ENTRA_EXTERNAL_ID}"
      },
      "AzureAd_ClientId": {
        "value": "${AZURE_AZURE_AD_CLIENT_ID}"
      },
      "AzureAd_Instance": {
        "value": "${AZURE_AZURE_AD_INSTANCE}"
      },
      "AzureAd_RedirectUri": {
        "value": "${AZURE_AZURE_AD_REDIRECT_URI}"
      },
      "AzureAd_TenantId": {
        "value": "${AZURE_AZURE_AD_TENANT_ID}"
      },
      "AzureAd_TokenValidationParameters_ValidAudiences": {
        "value": "${AZURE_AZURE_AD_TOKEN_VALIDATION_PARAMETERS_VALID_AUDIENCES}"
      },
      "AzureAd_TokenValidationParameters_ValidIssuers": {
        "value": "${AZURE_AZURE_AD_TOKEN_VALIDATION_PARAMETERS_VALID_ISSUERS}"
      },
      "AzureAd_TokenValidationParameters_ValidateAudience": {
        "value": "${AZURE_AZURE_AD_TOKEN_VALIDATION_PARAMETERS_VALIDATE_AUDIENCE}"
      },
      "AzureAd_TokenValidationParameters_ValidateIssuer": {
        "value": "${AZURE_AZURE_AD_TOKEN_VALIDATION_PARAMETERS_VALIDATE_ISSUER}"
      },
      "CRMIntegration_ApplicationId": {
        "value": "${AZURE_CRMINTEGRATION_APPLICATION_ID}"
      },
      "CRMIntegration_ApplicationSecret": {
        "value": "${AZURE_CRMINTEGRATION_APPLICATION_SECRET}"
      },
      "CRMIntegration_Tenant": {
        "value": "${AZURE_CRMINTEGRATION_TENANT}"
      },
      "EntraExternalId_ClientId": {
        "value": "${AZURE_ENTRA_EXTERNAL_ID_CLIENT_ID}"
      },
      "EntraExternalId_Instance": {
        "value": "${AZURE_ENTRA_EXTERNAL_ID_INSTANCE}"
      },
      "EntraExternalId_RedirectUri": {
        "value": "${AZURE_ENTRA_EXTERNAL_ID_REDIRECT_URI}"
      },
      "EntraExternalId_TenantId": {
        "value": "${AZURE_ENTRA_EXTERNAL_ID_TENANT_ID}"
      },
      "EntraExternalId_TokenValidationParameters_ValidAudiences": {
        "value": "${AZURE_ENTRA_EXTERNAL_ID_TOKEN_VALIDATION_PARAMETERS_VALID_AUDIENCES}"
      },
      "EntraExternalId_TokenValidationParameters_ValidIssuers": {
        "value": "${AZURE_ENTRA_EXTERNAL_ID_TOKEN_VALIDATION_PARAMETERS_VALID_ISSUERS}"
      },
      "EntraExternalId_TokenValidationParameters_ValidateAudience": {
        "value": "${AZURE_ENTRA_EXTERNAL_ID_TOKEN_VALIDATION_PARAMETERS_VALIDATE_AUDIENCE}"
      },
      "EntraExternalId_TokenValidationParameters_ValidateIssuer": {
        "value": "${AZURE_ENTRA_EXTERNAL_ID_TOKEN_VALIDATION_PARAMETERS_VALIDATE_ISSUER}"
      },
      "GRAPHIntegration_ApplicationId": {
        "value": "${AZURE_GRAPHINTEGRATION_APPLICATION_ID}"
      },
      "GRAPHIntegration_ApplicationSecret": {
        "value": "${AZURE_GRAPHINTEGRATION_APPLICATION_SECRET}"
      },
      "GRAPHIntegration_TenantId": {
        "value": "${AZURE_GRAPHINTEGRATION_TENANT_ID}"
      },
      "exchange_postgres_password": {
        "value": "${AZURE_EXCHANGE_POSTGRES_PASSWORD}"
      },
      "exchange_postgres_username": {
        "value": "${AZURE_EXCHANGE_POSTGRES_USERNAME}"
      },
      "orleans_cluster_id": {
        "value": "${AZURE_ORLEANS_CLUSTER_ID}"
      },
      "orleans_service_id": {
        "value": "${AZURE_ORLEANS_SERVICE_ID}"
      },
      "stap_postgres_password": {
        "value": "${AZURE_STAP_POSTGRES_PASSWORD}"
      },
      "stap_postgres_username": {
        "value": "${AZURE_STAP_POSTGRES_USERNAME}"
      },
      "environmentName": {
        "value": "${AZURE_ENV_NAME}"
      },
      "location": {
        "value": "${AZURE_LOCATION}"
      }
    }
  }

Have this main.bicep file:

targetScope = 'subscription'

@minLength(1)
@maxLength(64)
@description('Name of the environment that can be used as part of naming resource convention, the name of the resource group for your application will use this name, prefixed with rg-')
param environmentName string

@minLength(1)
@description('The location used for all deployed resources')
param location string

@description('Id of the user or app to assign application roles')
param principalId string = ''

param Authorities_AzureAd string
param Authorities_EntraExternalId string
param AzureAd_ClientId string
param AzureAd_Instance string
param AzureAd_RedirectUri string
param AzureAd_TenantId string
param AzureAd_TokenValidationParameters_ValidAudiences string
param AzureAd_TokenValidationParameters_ValidIssuers string
param AzureAd_TokenValidationParameters_ValidateAudience string
param AzureAd_TokenValidationParameters_ValidateIssuer string
param CRMIntegration_ApplicationId string
@secure()
param CRMIntegration_ApplicationSecret string
param CRMIntegration_Tenant string
param EntraExternalId_ClientId string
param EntraExternalId_Instance string
param EntraExternalId_RedirectUri string
param EntraExternalId_TenantId string
param EntraExternalId_TokenValidationParameters_ValidAudiences string
param EntraExternalId_TokenValidationParameters_ValidIssuers string
param EntraExternalId_TokenValidationParameters_ValidateAudience string
param EntraExternalId_TokenValidationParameters_ValidateIssuer string
param GRAPHIntegration_ApplicationId string
@secure()
param GRAPHIntegration_ApplicationSecret string
param GRAPHIntegration_TenantId string
@metadata({azd: {
  type: 'generate'
  config: {length:22}
  }
})
@secure()
param exchange_postgres_password string
@metadata({azd: {
  type: 'generate'
  config: {length:10,noNumeric:true,noSpecial:true}
  }
})
param exchange_postgres_username string
@metadata({azd: {
  type: 'generate'
  config: {length:25,noUpper:true,noSpecial:true}
  }
})
param orleans_cluster_id string
@metadata({azd: {
  type: 'generate'
  config: {length:25,noUpper:true,noSpecial:true}
  }
})
param orleans_service_id string
@metadata({azd: {
  type: 'generate'
  config: {length:22}
  }
})
@secure()
param stap_postgres_password string
@metadata({azd: {
  type: 'generate'
  config: {length:10,noNumeric:true,noSpecial:true}
  }
})
param stap_postgres_username string

var tags = {
  'azd-env-name': environmentName
}

resource rg 'Microsoft.Resources/resourceGroups@2022-09-01' = {
  name: 'rg-${environmentName}'
  location: location
  tags: tags
}
module resources 'resources.bicep' = {
  scope: rg
  name: 'resources'
  params: {
    location: location
    tags: tags
    principalId: principalId
  }
}

module exchange_postgres 'exchange-postgres/exchange-postgres.module.bicep' = {
  name: 'exchange-postgres'
  scope: rg
  params: {
    administratorLogin: exchange_postgres_username
    administratorLoginPassword: exchange_postgres_password
    keyVaultName: resources.outputs.SERVICE_BINDING_KVD9BB39FC_NAME
    location: location
  }
}
module redis 'redis/redis.module.bicep' = {
  name: 'redis'
  scope: rg
  params: {
    keyVaultName: resources.outputs.SERVICE_BINDING_KVB6088994_NAME
    location: location
  }
}
module stap_postgres 'stap-postgres/stap-postgres.module.bicep' = {
  name: 'stap-postgres'
  scope: rg
  params: {
    administratorLogin: stap_postgres_username
    administratorLoginPassword: stap_postgres_password
    keyVaultName: resources.outputs.SERVICE_BINDING_KVD959CDED_NAME
    location: location
  }
}

output MANAGED_IDENTITY_CLIENT_ID string = resources.outputs.MANAGED_IDENTITY_CLIENT_ID
output MANAGED_IDENTITY_NAME string = resources.outputs.MANAGED_IDENTITY_NAME
output AZURE_LOG_ANALYTICS_WORKSPACE_NAME string = resources.outputs.AZURE_LOG_ANALYTICS_WORKSPACE_NAME
output AZURE_CONTAINER_REGISTRY_ENDPOINT string = resources.outputs.AZURE_CONTAINER_REGISTRY_ENDPOINT
output AZURE_CONTAINER_REGISTRY_MANAGED_IDENTITY_ID string = resources.outputs.AZURE_CONTAINER_REGISTRY_MANAGED_IDENTITY_ID
output AZURE_CONTAINER_REGISTRY_NAME string = resources.outputs.AZURE_CONTAINER_REGISTRY_NAME
output AZURE_CONTAINER_APPS_ENVIRONMENT_NAME string = resources.outputs.AZURE_CONTAINER_APPS_ENVIRONMENT_NAME
output AZURE_CONTAINER_APPS_ENVIRONMENT_ID string = resources.outputs.AZURE_CONTAINER_APPS_ENVIRONMENT_ID
output AZURE_CONTAINER_APPS_ENVIRONMENT_DEFAULT_DOMAIN string = resources.outputs.AZURE_CONTAINER_APPS_ENVIRONMENT_DEFAULT_DOMAIN
output SERVICE_BINDING_KVB6088994_ENDPOINT string = resources.outputs.SERVICE_BINDING_KVB6088994_ENDPOINT
output SERVICE_BINDING_KVB6088994_NAME string = resources.outputs.SERVICE_BINDING_KVB6088994_NAME
output SERVICE_BINDING_KVD959CDED_ENDPOINT string = resources.outputs.SERVICE_BINDING_KVD959CDED_ENDPOINT
output SERVICE_BINDING_KVD959CDED_NAME string = resources.outputs.SERVICE_BINDING_KVD959CDED_NAME
output SERVICE_BINDING_KVD9BB39FC_ENDPOINT string = resources.outputs.SERVICE_BINDING_KVD9BB39FC_ENDPOINT
output SERVICE_BINDING_KVD9BB39FC_NAME string = resources.outputs.SERVICE_BINDING_KVD9BB39FC_NAME
output STATIC_PUBLIC_IP string = resources.outputs.publicIpAddress

Have this azure-dev.yml file in folder .azdo/pipelines:

# Run when commits are pushed to development
trigger:
  - development

pool:
  vmImage: ubuntu-latest

steps:
  # install dotnet
  - task: UseDotNet@2
    displayName: 'Use .NET SDK 9'
    inputs:
      version: 9.x

  # install azd
  - task: setup-azd@1
    displayName: Install azd
  
  # azd delegate auth to az to use service connection with AzureCLI@2
  - pwsh: |
      azd config set auth.useAzCliAuth "true"
    displayName: Configure AZD to Use AZ CLI Authentication.

  - task: AzureCLI@2
    displayName: Provision Infrastructure
    inputs:
      azureSubscription: azconnection
      scriptType: bash
      scriptLocation: inlineScript
      keepAzSessionActive: true
      workingDirectory: 'Project.AppHost'
      inlineScript: |
        azd provision --no-prompt
    env:
      AZURE_SUBSCRIPTION_ID: $(AZURE_SUBSCRIPTION_ID)
      AZURE_ENV_NAME: $(AZURE_ENV_NAME)
      AZURE_LOCATION: $(AZURE_LOCATION)
      # Deprecated: AZD_INITIAL_ENVIRONMENT_CONFIG: $(AZD_INITIAL_ENVIRONMENT_CONFIG)
      # https://github.com/Azure/azure-dev/pull/5143
      # https://learn.microsoft.com/en-us/dotnet/aspire/whats-new/dotnet-aspire-9.3#-azd-major-improvements-to-cicd-for-aspire-apps

  # Copy files to ArtifactStagingDirectory
  - task: CopyFiles@2
    displayName: Copy files to staging directory
    inputs:
      Contents: '**/*'  # All files in the directory
      TargetFolder: '$(Build.ArtifactStagingDirectory)'

  # List the files in the staging directory
  - task: Bash@3
    displayName: List files in staging directory
    inputs:
      targetType: 'inline'
      script: |
        ls -R $(Build.ArtifactStagingDirectory)

    # Publish the artifact
  - task: PublishPipelineArtifact@1
    displayName: Publish artifact
    inputs:
      targetPath: '$(Build.ArtifactStagingDirectory)'
      artifactName: 'app'  # Name of the artifact
      publishLocation: 'pipeline'

Expected behavior
Application should provision in Azure.

Environment
Information on your environment:
* Language name and version: US English
* IDE and version : Visual Studio Version 17.14.7

Additional context
Now many variables appeared in the CI pipeline, which is a good thing, and an improvement over AZD_INITIAL_ENVIRONMENT_CONFIG.

I tried logging env variables in azure-dev.yml with this step:

  - pwsh: |
      Write-Host "Printing all relevant AZURE_ environment variables:"
      Get-ChildItem Env:AZURE_* | ForEach-Object { Write-Host "$($_.Name) = $($_.Value)" }
    displayName: TEMPORARY Print AZURE_* **variables**

All variables are logged, they do exist.

Finally a glimse of how config.json looks like in .azure > env-dev with random GUID(s) as an example:

{
	"infra": {
		"parameters": {
			"AzureAd_ClientId": "74ddfc09-62db-4baa-9ada-778348be647c",
			"AzureAd_TokenValidationParameters_ValidAudiences": "[\"api://e935a748-8b59-4c26-a59c-9bcc83f5ab57\"]",
			"AzureAd_TokenValidationParameters_ValidIssuers": "[\"https://sts.windows.net/d9a98bfa-f335-4237-bb47-4b2e5deba383/\"]",
			"AzureAd_TokenValidationParameters_ValidateAudience": "true",
			"AzureAd_TokenValidationParameters_ValidateIssuer": "true"
		}
	},
	"vault": "472b923f-2a6e-4132-ac06-6f6d36895629"
}

And this is how some of the variables looks like in Azure DevOps pipeline:
AZURE_AZURE_AD_CLIENT_ID
74ddfc09-62db-4baa-9ada-778348be647c

AZURE_AZURE_AD_TOKEN_VALIDATION_PARAMETERS_VALID_AUDIENCES
["api://e935a748-8b59-4c26-a59c-9bcc83f5ab57"]

And last there's this auto-generated file with suspicious configuration that I'm not sure works:
project-portal.tmpl.yaml

- name: services__project__http__0
  value: http://project-frontend.{{ .Env.AZURE_CONTAINER_APPS_ENVIRONMENT_DEFAULT_DOMAIN }}
- name: services__project__https__0
  value: https://project-frontend.{{ .Env.AZURE_CONTAINER_APPS_ENVIRONMENT_DEFAULT_DOMAIN }}

Please if anybody has a solution for this, any help or clue is greatly appreciated!!

[weikanglim]

Hey @Dayonel apologies for the issue you're running into.

I suspect it's related to nested json and escape characters:

AZURE_AZURE_AD_TOKEN_VALIDATION_PARAMETERS_VALID_AUDIENCES
["api://e935a748-8b59-4c26-a59c-9bcc83f5ab57"]

would need to be changed to:

AZURE_AZURE_AD_TOKEN_VALIDATION_PARAMETERS_VALID_AUDIENCES
[\"api://e935a748-8b59-4c26-a59c-9bcc83f5ab57\"]

same for AZURE_AZURE_AD_TOKEN_VALIDATION_PARAMETERS_VALID_ISSUERS

This would help with an immediate workaround.

As for a longer term solution, I'm not sure what we'd do here; cc: @vhvb1989

[vhvb1989]

@Dayonel , instead of using AZD_INITIAL_ENVIRONMENT_CONFIG, you need to map all the ENV VARS in the azure-dev.yaml pipeline definition, just like how you are mapping:

One thing you can do is to temporarily change the name of azure-dev.yaml and run azd pipeline config - azd will ask you if you want to generate azure-dev.yaml and it will list/map all the ENV VARS. Then you can copy and past that and revert the name to your original flow.

[Menghua1]

According to the investigation, when variables in .azure/envName/config.json (such as "[\"api://e935a748-8b59-4c26-a59c-9bcc83f5ab57\"]") are synchronized to the remote Pipeline, the escape characters are lost and the value becomes ["api://e935a748-8b59-4c26-a59c-9bcc83f5ab57"]. As a result, azd incorrectly identifies it as an array type, causing Bicep parameter parsing to fail.

It is recommended that azd automatically escape special characters before synchronizing variables to the remote Pipeline to ensure that the value finally stored in the Pipeline is consistent with the local .azure/envName/config.json. For example:

1. Original value of .azure/envName/config.json:

"AzureAd_TokenValidationParameters_ValidAudiences": "[\"api://e935a748-8b59-4c26-a59c-9bcc83f5ab57\"]"

2. azd adds an escaping step before syncing to the remote Pipeline to escape the variable value as follows:

"AzureAd_TokenValidationParameters_ValidAudiences": "[\\\"api://e935a748-8b59-4c26-a59c-9bcc83f5ab57\\\"]"

3. Finally, the variable value in Pipeline is:

"AzureAd_TokenValidationParameters_ValidAudiences": "[\"api://e935a748-8b59-4c26-a59c-9bcc83f5ab57\"]"

@vhvb1989 What do you think about this approach? If you agree, I will open a PR to add the escaping step and resolve this issue.

[Dayonel]

@weikanglim @vhvb1989 @Menghua1 thanks you so much for the responses.

I can confirm that the problem is solved escaping the characters.
So the solution as you guys mentioned is:

Now config.json becomes this with escapes:

{
	"infra": {
		"parameters": {
			"AzureAd_TokenValidationParameters_ValidAudiences": "[\\\"api://e935a748-8b59-4c26-a59c-9bcc83f5ab57\\\"]",
			"AzureAd_TokenValidationParameters_ValidIssuers": "[\\\"https://sts.windows.net/d9a98bfa-f335-4237-bb47-4b2e5deba383/\\\"]"
		}
	},
	"vault": "472b923f-2a6e-4132-ac06-6f6d36895629"
}

Azure DevOps pipeline variables become this:

AZURE_AZURE_AD_TOKEN_VALIDATION_PARAMETERS_VALID_AUDIENCES
[\"api://e935a748-8b59-4c26-a59c-9bcc83f5ab57\"]

AZURE_AZURE_AD_TOKEN_VALIDATION_PARAMETERS_VALID_ISSUERS
[\"https://sts.windows.net/d9a98bfa-f335-4237-bb47-4b2e5deba383/\"]

I also regenerated azure-dev.yml file, thanks for that, I didn't know it is automatically generated.

Thanks a lot!

[Menghua1]

Hi @vhvb1989,

When running azd pipeline config, the parameter values synced to the remote pipeline lose their escape characters and become inconsistent with the local .azure/envName/config.json. This issue occurs because the escape characters are lost, causing the JSON string to become an array. As a result, Bicep parameter parsing fails.

I recommend that the parameter values synchronized to the remote pipeline should be consistent with those in the local .azure/envName/config.json, including all escape characters, to avoid type parsing errors.

What do you think about this? If you agree, I can open a PR to fix it.

[vhvb1989]

I do agree @Menghua1 . Thank you for looking into this

[Menghua1]

Hi @vhvb1989, I’ve opened a PR #5568 to fix this issue. Could you please review it? Thank you!