Consider adding a bicep resource type

[davidfowl]

Design

AzureBicepResource allows for defining a piece of piece as either a file on disk, a inline string, or an embedded resource. This is the primitive that allows developers to express a piece of bicep and allows them to capture outputs for referencing in other resources. Bicep resources have parameters and outputs. These are defined in the bicep and are used to marshal data into and out of the template after execution.

Hello World:

As a fake example, here's a hello world example:

test.bicep (in the apphost directory)

param name string

output test string = test

using Aspire.Hosting.Azure;

var builder = DistributedApplication.CreateBuilder(args);

builder.AddAzureProvisioning();

var templ = builder.AddBicepTemplate("test", "test.bicep")
                   .AddParameter("test", "hello world");

builder.AddProject<Projects.BicepSample_ApiService>("api")
       .WithEnvironment("bicepValue_test", templ.GetOutput("test"))

builder.Build().Run();

Azure provisioning will submit this deployment, query the outputs and the environment variable bicepValue_test will be set to "hello world".

The manifest for this looks like:

{
  "resources": {
    "test": {
      "type": "azure.bicep.v0",
      "path": "test.bicep",
      "params": {
        "test": "hello world"
      }
    },
    "api": {
      "type": "project.v0",
      "path": "../BicepSample.ApiService/BicepSample.ApiService.csproj",
      "env": {
        "OTEL_DOTNET_EXPERIMENTAL_OTLP_EMIT_EXCEPTION_LOG_ATTRIBUTES": "true",
        "OTEL_DOTNET_EXPERIMENTAL_OTLP_EMIT_EVENT_LOG_ATTRIBUTES": "true",
        "bicepValue_test": "{test.outputs.test}"
      },
      "bindings": {
        "http": {
          "scheme": "http",
          "protocol": "tcp",
          "transport": "http"
        },
        "https": {
          "scheme": "https",
          "protocol": "tcp",
          "transport": "http"
        }
      }
    }
  }
}

Not too surprising. There's a new resource type azure.bicep.v0 that describes the path to the bicep file, the parameters and their values. In the api's section, the bicepValue_test env variable references the bicep outputs directly. This isn't expressed in the manifest but after the execution engine runs, it get grab the outputs and fill in the values appropriately (as @tg-msft described, if we need to do something special here than we need to be able to mark specific outputs as "please put this in a secret".

Connection Strings

The low-level bicep primitive is great and allows users to define custom resources not supported by aspire out of the box. In order to make WithReference work with IResourceWithConnectionString, bicep resources can define a connection string template (much like container resources). For example, here's the service bus namespace resource in the manifest:

{
  "resources": {
    "sb": {
      "type": "azure.bicep.v0",
      "connectionString": "{sb.outputs.serviceBusEndpoint}",
      "path": "aspire.hosting.azure.bicep.servicebus.bicep",
      "params": {
        "serviceBusNamespaceName": "sb",
        "principalId": "",
        "principalType": "",
        "topics": [
          "topic1"
        ],
        "queues": [
          "queue1"
        ]
      }
    }
  }
}

This defines a connection string property that resolves the serviceBusEndpoint coming from the output of the aspire.hosting.azure.bicep.servicebus.bicep file.

Note: Aspire uses connection strings to mean connection information. These may or may not contain secrets.

When the bicep file wants to output a secret for consumption in another resource outside of the bicep universe (where listKeys can be used to avoid this problem), we need a way to instruct azd to get the keys securely outside of this context. We will pass a key vault name to bicep files that request it so that secrets can be written to this keyvault. This will be referrred to in the manifest via "secretOutputs".

e.g.

This is the cosmos bicep and writes the connection string to the host provided keyvault.

https://github.com/dotnet/aspire/blob/b0570889d78d792407caf7667acb2056732e9266/src/Aspire.Hosting.Azure/Bicep/cosmosdb.bicep

This is how it is referenced in the manifest:

aspire/playground/bicep/BicepSample.AppHost/aspire-manifest.json

Lines 122 to 132 in b057088

	"cosmos": {
	"type": "azure.bicep.v0",
	"connectionString": "{cosmos.secretOutputs.connectionString}",
	"path": "aspire.hosting.azure.bicep.cosmosdb.bicep",
	"params": {
	"databaseAccountName": "cosmos",
	"databases": [
	"db3"
	],
	"keyVaultName": ""
	}

The connection string template for a bicep resource can contain a keys function with the resource name. This instructs azd to get the keys and fill it into this part of the connection string, without doing any crazy keyvault shenanigans 😄:

Example redis:

{
  "resources": {
    "redis": {
      "type": "azure.bicep.v0",
      "connectionString": "{redis.secretOutputs.connectionString}",
      "path": "aspire.hosting.azure.bicep.redis.bicep",
      "params": {
        "redisCacheName": "redis"
      }
    }
  }
}

The keys method takes the full resource name (including the outputs of the bicep). This should be enough to resolve the primary key (do we need to default to one?).

Hopefully, as we default to more managed identities, this will be less of a problem.

Ambient context

When these bicep files get run in either azd of AzureProvisioning, we need to supply context about the current user (when running locally there's no user assigned managed identity for e.g.). Bicep resources can opt into this information by declaring well known parameter names. This could instead be identified by parameter metadata attribute (to avoid the naming matching). These also show up in the manifest as empty strings but it would be more useful to flow that metadata via the manifest.

Examples of these:

• principalId
• principalType
• keyVaultName

Original Issue

This is the "break glass" option for resources not defined in aspire directly and not modeled in C#. The idea inspired by the AWS PR is to expose a low level "stringly typed" bicep resource that points to a bicep file and allows you to map the outputs of that bicep file to environment variables in any target resource.

var builder = DistributedApplication.CreateBuilder(args);

var bicep = builder.AddBicep("storage", "storage.bicep");

builder.AddProject<Projects.WebApplication1>("api")
    .WithReference(bicep);

builder.Build().Run();

This would go into the manifest as a bicep.v0 resource with a path to the file:

{
    "resources": {
          "storage": {
                "type": "bicep.v0",
                "path": "infra/storage.bicep"
           }
     }
}

As with any interop system there will be problems:

• Marshalling inputs from C# to bicep
• Marshalling outputs from bicep to C#

cc @ellismg @tg-msft for thoughts

[eerhardt]

What does this do at F5/dotnet run time?

[davidfowl]

If we were modeling this similar to what we do with other resources, it would explode when WithReference executed because the outputs weren't provided. Today if you call AddServiceBus without AzureProvisioning and the connection string isn't specified, then we explode with an error (missing connection string etc etc). I think we would need to do a similar thing here.

[eerhardt]

and the connection string isn't specified, then we explode with an error (missing connection string etc etc).

But you could add the connection string to the AppHost's config (i.e. user secrets), and it would still get passed through to the app. Will that still work here?

[davidfowl]

That's the idea, though it might not just be a single connection string. It requires knowing the shape of the outputs from the bicep.

[normj]

What does this do at F5/dotnet run time?

If there is a change to the provisioning template, then after pushing F5 you have to wait till the resources are allocated. At least in AWS when we are talking about application level resources like S3 buckets, queues and topics provisioning is pretty quick. For the sample app in my PR which has a topic and queue subscribed to the topic if nothing is provisioning then it is about 20 seconds. If there is no change to the provisioning template, which I assume is the majority of the F5 cases, then it is essentially a noop.

[ellismg]

    "resources": {
          "storage": {
                "type": "bicep.v0",
                "path": "infra/storage.bicep"
           }
     }

Will be nice, I think, if we allow a parameters key on this object.

I assume expressions can access and output from the bicep by using the outputs property (e.g. {storage.outputs.someBicepProperty} and that the connectionString property of this maps to a property named connectionString on the resource itself (e.g. it is sugar for .outputs.connectionString)?

I suppose azd can interpret the @secure decorator on the output to decide if it needs to treat the connection string as a secret or not (e.g reference via keyvault or burn into the environment directly)

Does that seem workable and align with what you were thinking, @tg-msft?

[tg-msft]

This is going to look a little different than integrating a CDK with a "generic" Azure deployment template. Using a prototype CDK I can automatically infer all the params/outputs and link them together using resource specific context (i.e., if I know it's a Storage account that you're trying to reference with managed identity, I can make the bicep generate outputs for the endpoint and Entra client id). Doing that for an arbitrary bicep template is going to be more of a challenge when wiring everything up.

We could explore asking folks to specify their params/outputs when declaring a bicep template so we have objects to glue together. Maybe something like:

// Create a Storage account
var storage = builder.AddAzureStorage("storage");

// Create a user assigned managed identity and grant it read access to my Storage account with a custom bicep template
var bicep = builder.AddBicep("appident", "appident.bicep")
    .WithParam("storageName", storage.Resource.Name) // This assumes a CDK so we can glue template outputs in the manifest
var principalId = bicep.WithOutput("principalId");
var clientId = bicep.WithOutput("clientId");

// Give my ApiService container that managed identity and a reference to my Storage account
builder.AddProject<Projects.HelloAspire_ApiService>("api")
    .WithUserAssignedIdentity(principalId, clientId)
    .WithReference(storage);

and turn that into a manifest kind of like:

{
  "resources": {
    "storage": {
      "type": "azure.template.v0",
      "params": { },
      "outputs": {
        "storage_name": {},
        "storage_properties_primaryEndpoints_blob": {}
      },
      "path": "storage.bicep"
    },
    "appident": {
      "type": "azure.template.v0",
      "params": {
        "storageName": "{storage.outputs.storage_name}"
      },
      "outputs": {
        "principalId": {},
        "clientId": {}
      },
      "path": "appident.bicep"
    },
    "api": {
      "type": "project.v0",
      "path": "HelloAspire.ApiService/HelloAspire.ApiService.csproj",
      "env": {
        "OTEL_DOTNET_EXPERIMENTAL_OTLP_EMIT_EXCEPTION_LOG_ATTRIBUTES": "true",
        "OTEL_DOTNET_EXPERIMENTAL_OTLP_EMIT_EVENT_LOG_ATTRIBUTES": "true",
        "AZURE_CLIENT_ID": "{appident.outputs.clientId}",
        "ConnectionStrings__blobs": "Uri={storage.outputs.storage_properties_primaryEndpoints_blob}"
      },
      "metadata": {
        "azd:userAssignedIdentity": "{appident.outputs.principalId}"
      }
    }
  }
}

Failing that, maybe we can rely on conventions like we've talked about for outputs of unreified azd resources like a container's ID. So if the bicep template has a uri output, WithReference could inject that into the manifest as an env var named "ConnectionStrings__{resource.Name}": "Uri={resource.outputs.uri}"?

[davidfowl]

Do outputs need to be defined in the manifest to be referenced? Those could be implicit properties of the "azure.template.v0" schema. The tool that executes the manifest has to evaluate the bicep anyways right?

[tg-msft]

Do outputs need to be defined in the manifest to be referenced?

At the moment I'm generating the outputs as empty objects where I can potentially stash extra metadata in the near future - like whether or not it represents a @secure value. If that's not needed by resource orchestrators like azd then we could leave them as implicit properties.

@ellismg - can you tell whether a deployment output is meant to be secure from the response? https://learn.microsoft.com/en-us/rest/api/resources/deployments/create-or-update?view=rest-resources-2021-04-01&tabs=HTTP#deploymentpropertiesextended makes me think it's just key/value pairs. Bicep decorators only apply to params as well. I'm curious how you'd automatically insert secure values into KV when they're used in manifest binding expressions if we're not generating a manifest entry with hints.

[ellismg]

@ellismg - can you tell whether a deployment output is meant to be secure from the response? https://learn.microsoft.com/en-us/rest/api/resources/deployments/create-or-update?view=rest-resources-2021-04-01&tabs=HTTP#deploymentpropertiesextended makes me think it's just key/value pairs. Bicep decorators only apply to params as well. I'm curious how you'd automatically insert secure values into KV when they're used in manifest binding expressions if we're not generating a manifest entry with hints.

I do not think that you can (in fact I think that secure outputs don't come back in a majority of the cases where you fetch a deployment).

We can however detect this from the bicep/arm template itself, but examining the outputs property of the template and seeing if things are marked as secure.

I expected that if there was not information in the manifest about if the values were secure or not that azd would determine it by inspecting the bicep/arm template that was referenced by the resource (in practice we likely have to do some level of analysis anyway, to understand things like deployment scopes)

[davidfowl]

I have a branch where I started playing with this:

https://github.com/dotnet/aspire/blob/davidfowl/bicep-resource/playground/azuresql/AzureSql.AppHost/Program.cs

[davidfowl]

Open draft PR so we can riff on something more concrete #2056

[davidfowl]

So @ellismg one thing I am realizing is that it's going to be problematic to design this generic resource type if any imperative code needs to run after outputs are produced in order to produce values that are consumable by other resources (like our fake "connectionString" property), we can't represent it in the manifest.

aspire/playground/azuresql/AzureSql.AppHost/Program.cs

Line 26 in fb2bf77

var cs = $"Server=tcp:{hostName}.database.windows.net,1433;Initial Catalog={databaseName};Encrypt=True;TrustServerCertificate=False;Connection Timeout=30;Authentication=\"Active Directory Default\";";

The connection string in this case is that template and azd can handle this. What about this logic:

aspire/src/Aspire.Hosting.Azure.Provisioning/Provisioners/ServiceBusProvisioner.cs

Line 83 in fb2bf77

resource.ServiceBusEndpoint = new Uri(serviceBusNamespace.Data.ServiceBusEndpoint).Host;

All of the logic inside of these template functions can't be expressed in this declarative manifest:

https://github.com/Azure/azure-dev/blob/b601a659b6598580ab6ba5736406723a15576f03/cli/azd/pkg/project/service_target_dotnet_containerapp.go#L378

So, what does a manifest look like for SQL server that needs a connection string?

Here's the hardcoding of a template that is not great 😢, but this logic needs to live somewhere.

aspire/playground/azuresql/AzureSql.AppHost/aspire-manifest.json

Lines 2 to 27 in fb2bf77

	"resources": {
	"rg": {
	"type": "parameter.v0",
	"value": "{rg.inputs.value}",
	"inputs": {
	"value": {
	"type": "string"
	}
	}
	},
	"sql1": {
	"type": "azure.bicep.v0",
	"path": "sql.bicep",
	"inputs": {
	"p0": "{rg.value}"
	}
	},
	"api": {
	"type": "project.v0",
	"path": "../AzureSql.ApiService/AzureSql.ApiService.csproj",
	"env": {
	"OTEL_DOTNET_EXPERIMENTAL_OTLP_EMIT_EXCEPTION_LOG_ATTRIBUTES": "true",
	"OTEL_DOTNET_EXPERIMENTAL_OTLP_EMIT_EVENT_LOG_ATTRIBUTES": "true",
	"ConnectionStrings__db": "Server=tcp:{sql1.outputs.sqlServerName}.database.windows.net,1433;Initial Catalog=db;Encrypt=True;TrustServerCertificate=False;Connection Timeout=30;Authentication=\u0022Active Directory Default\u0022;"
	},
	"bindings": {

[ellismg]

@davidfowl

Yeah - this is tricky. We already had to add some primitives inside azd for when we parse the container-app.tmpl.yaml files to be able to do things like extract the host. Systems like Terraform and Pulumi have nice ways to put some level of computation in here to do this level of conversion, but we don't have such power (unless we want to get really wacky).

A few thoughts on things we might be able to do (spit-balling here before vacation)

We do have a way to put metadata on outputs in a bicep file (a feature I added a while back). You could imagine something like this:

@metdata({ azd: { type: 'template', context: '<some-output-name>' })
output connectionString string = "{{ (urlParse .ServiceBusEndpoint.Data.ServiceBusEndpoint).host }}"

The above would mean that when a thing is stitching these things together, it should treat the value as text/template and then evaluate it using the given output name as the context object. We could then define helpers like urlParse (like heml does: https://helm.sh/docs/chart_template_guide)

You could image different shapes as well:

@metdata({ azd: { type: 'template' })
output connectionString object = {
  template: "{{ (urlParse .ServiceBusEndpoint.Data.ServiceBusEndpoint).host }}"
  context: { ... }
}

Which might be more ergonomic and require fewer outputs, so I sort of like that more.

I proposed go text/templating because it is easy for me and it is "Cloud Native". But I could image instead an expression language that looks closer to what arm has (and maybe over time they can add more of these primitives so they can just evaluate them? They already allow some level of data transformation, perhaps we just need a few more primitives.

The other option is you could always try to build something using deployment scripts which would allow you to inject arbitrary computation via PowerShell scripts during deployment and do whatever sort of munging you wanted. We do have a user assigned identity that we could run all this stuff under and it would in theory have access to all the resources, so if you want to code spit a bunch of PowerShell, maybe that's a path forward since you could do whatever you needed to do and then set an output of the deployment script which your module then exports. Feels like a pretty big and scary hammer, however, so maybe something more limited like the replacement above is a thing to codify first. It is likely less scary operationally during deployment than running arbitrary PowerShell.

[davidfowl]

OK I made lots of progress on this learning way more about bicep than I'd like. I was able to replat the AzureSql resource on top of this primitive:

This is the sql resource type:
https://github.com/dotnet/aspire/blob/98297f577a2a5e540112e51f94996b5419811639/playground/bicep/BicepSample.AppHost/AzureSqlResource.cs

This is the bicep file:
https://github.com/dotnet/aspire/blob/98297f577a2a5e540112e51f94996b5419811639/playground/bicep/BicepSample.AppHost/sql.bicep

This is the manifest:
https://github.com/dotnet/aspire/blob/f064cf03e37baa4124a4d6e3139dc7bc301ce121/playground/bicep/BicepSample.AppHost/aspire-manifest.json

A couple of thoughts:

• Just like containers, bicep resources need to describe their connection string template. This can use any references from anywhere in the manifest.
• The sql.bicep is attached to the parent resource also has support for provisioning databases (child resources in the model)
• Child resources are typed as the same type but reference properties from the parent. These resources shouldn't be provisioned (I'm not sure if we need to indicate that somehow).
• I'm currently passing the array of database names as a parameter to the bicep because I didn't want to do something like T4. We can do another pass around the template to generate the final string at runtime.

Random follow up:

• Need to be able to store resource types in resources. It'll make it easier to embed them in libraries.
• I'm going to attempt to convert more resources to see what the implications are and if there are any blockers.

[davidfowl]

I think we will need some special case logic for insecure connection strings so that they are never flowed via outputs based on what @ellismg said above. Right now, we happen to be using connection strings for cosmos because it works with the emulator and when deployed. Right now, azd emits the resource name from the bicep and does an explicit API call to get the connection string. This way it's never stored in the deployment logs/ouputs.

For cases where we are using managed identity, it's much nicer as we don't need to do any transformation outside of the template.

The other alternative is to define some syntax in the manifest itself that azd understands to do this transform. That would keep the metadata out of the bicep template.

I'll migrate cosmos to this new model so I can try out the approach.

[davidfowl]

OK got cosmosdb working and would love some feedback on this hacky model. Here's the manifest output:

aspire/playground/bicep/BicepSample.AppHost/aspire-manifest.json

Lines 57 to 68 in 8628a30

	"cosmos": {
	"type": "azure.bicep.v0",
	"connectionString": "AccountEndpoint={cosmos.outputs.documentEndpoint};AccountKey={keys({cosmos.outputs.accountName}})",
	"path": "aspire.hosting.azure.bicep.cosmosdb.bicep",
	"params": {
	"databaseAccountName": "cosmos",
	"databases": [
	"db3"
	]
	},
	"azureResourceType": "Microsoft.DocumentDB/databaseAccounts@2023-04-15"
	},

The connection string template calls a 'keys' method to indicate that the orchestrator should grab the keys for this resource type. The azure resource type is specified here.

[davidfowl]

Updated the issue with a more detailed design of what's implemented in the pull request.

[IEvangelist]

I'm curious about a few aspects of this approach.

It's mentioned that the bicep files would be in the app host directory, yet they're usually in an ./infra/ folder at the root, not the end of the world but would that change in the future?

If an output name in a bicep file changes, that requires an app host code change then too - right, I'd assume so?

I was under the impression that the Azure Provisioning bits relied on the resource management SDKs, is that true?

• Call .AddAzureProvisioning: Provisions Azure resources using the resource management SDKs.
• Reference bicep file: Deploy Azure resources to provisioned targets.
• App host: requires output from deployed resources to configure resources.

Is that an accurate way to think about this? So really, this proposal is just a way of providing arguments as inputs in bicep, and getting outputs from it as well? Might there be an opportunity to do any of this implicitly, meaning all AzureResource types expose a set of well-known bicep output propoerties?

[jongio]

Issue here for secure output support: Azure/bicep#2163

[davidfowl]

It's mentioned that the bicep files would be in the app host directory, yet they're usually in an ./infra/ folder at the root, not the end of the world but would that change in the future?

The bicep files are relative to the apphost directory, but they can be anywhere on disk.

If an output name in a bicep file changes, that requires an app host code change then too - right, I'd assume so?

The caching will take into account the inputs, and the bicep file content (a checksum is generated based on that).

Call .AddAzureProvisioning: Provisions Azure resources using the resource management SDKs.

They still do, but there's a new BicepProvisoner that uses the resource management SDK to deploy the bicep template (well ARM template).

Is that an accurate way to think about this? So really, this proposal is just a way of providing arguments as inputs in bicep, and getting outputs from it as well?

Yep! That's the primitive being introduced.

Might there be an opportunity to do any of this implicitly, meaning all AzureResource types expose a set of well-known bicep output propoerties?

Future, yea, can easily be built (probably using attributes etc).

[davidfowl]

The initial change is in, we'll iterate and file follow up issues as azd attempts to implement this 😃