Proposal - Module metadata

[majastrz]

Current

Currently, any Bicep file can be consumed as a local module without modification. The identity of the module is the path on the local file system and there is no version.

Proposal

Considerations

• The Bicep Registry ([Story] Bicep Registry #2128) feature will allow modules to be published to public and private registries.
• Modules must declare required metadata before publishing:
  • Module identifier
  • Module version
• Modules should declare optional metadata before publishing:
  • Descriptions
  • Author
  • Other registry implementation specific metadata. (Depends on outcome of [Story] Bicep Registry #2128.)

Module authors will need a way to set the metadata before publishing a module to the registry. We will need to choose a combination of the following options:

Option 1 - Convention

Metadata values could be inferred from Bicep module contents and file metadata.

Pros:

• Just works.

Cons:

• Very limited. Could probably only use the .bicep file name as module identifier. And even then the convention of calling modules main.bicep makes it less than useful.

Option 2 - CLI arguments

Metadata values can be passed via CLI arguments to the bicep pack command:

bicep pack main.bicep --module-id My.Module.Id --version 1.2.3 --description "This is my description that could possibly be a novel." --author "me"

Pros:

• No new syntax
• Good for transient information like versions.

Cons:

• bicep pack command line will become overwhelming to use correctly.
• Why is a declarative language using an imperative construct to specify its own package metadata?

Option 3 - Top-level object declaration

package 'My.Module.Id' = {
  description: 'This is my description that could possibly be a novel.'
  author: 'me'
}

Pros:

• Groups all the metadata in one place.
• Easy to provide relevant completions.

Cons:

• New top-level declaration

Option 4 - Top-level declaration with decorators

@description('This is my description that could possibly be a novel.')
@author('me')
package 'My.Module.Id'

Pros:

• Groups all the metadata in one place.

Cons:

• New top-level declaration
• Decorator completions are hard to filter because we don't know if the user is starting a new declaration with decorators or adding decorators to the following declaration.

Option 5 - File-level decorators

I just made up the syntax for file-level decorators. If we are interested in exploring this option, we can come up with more options to refine it:

#package('My.Module.Id')
#author('me')
#description('This is my description that could possibly be a novel.')

Pros:

• None?

Cons:

• Completely new concept
• Does not group metadata in one place unless enforced by a linter rule.

Combination Option A

Metadata Name	Description
module identifier	Defaults to the file name minus the .bicep extension unless specified in the package declaration.
module version	Defaults to 0.0 unless specified in the package declaration. Can be overridden via the --version CLI argument in bicep pack.
description	Optionally set in the package declaration.
author	Optionally set in the package declaration.
etc.	TBD

Considerations:

• bicep pack works on any module without requiring a package declaration.
• package is intended to improve package metadata quality.
• Version must be easy to override in CI. We should not make the mistake NPM did with package.json

Combination Option B

The package declaration is mandatory in this option.

Metadata Name	Description
module identifier	Specified in the required package declaration.
module version	Defaults to 0.0 unless specified in the package declaration. Can be overridden via the --version CLI argument in bicep pack.
description	Optionally set in the package declaration.
author	Optionally set in the package declaration.
etc.	TBD

Considerations:

• bicep pack works on any module that has a package declaration. Packing a module without a package declaration is an error.
• package declaration represents the intent to publish the module to a registry.
• Version must be easy to override in CI. We should not make the mistake NPM did with package.json

Open Questions

1. If we decide to support multi-module packages, where do we put the metadata that is specific to the package itself (package description, package ID, version, etc.)

[miqm]

I wrote this observation in other issue but I will repeat it here - I'd like to have ability to pack multiple independent biceps into single package. Something like a dll having multiple classes to use, not just a single one. You might have a series of small modules and creating feed for every one of them might be way too much.

[majastrz]

Makes sense. If we adopt that, it will also have an effect on the scopes of various metadata. For example, we'd need a package description that is separate from module description.

[majastrz]

The common naming convention for entry point module is main.bicep. As a result, the default module name in Combination A would be main, which is not unique in any way. For that reason, we are leaning towards making the package declaration mandatory for any module that will be published to the registry. It would not be required for local modules.

[miqm]

I think making package declaration inside bicep file is not a good idea. Various systems might require different properties. Some of them (ie. git, zip over https) might require none, unless we decide that a nupkg format is required for module packs.

another case might be that if we pack many files into package, and then when we reference a package - if we do not provide specific file name the main.bicep will be used (some kind of entry point).

open question - if main (or the file with package declaration) uses it’s local files as modules - can we call that files when pointing to a remote? Eg we have a remote package that sets up server farm and app service . Setting up Farm and web app are nested modules called from main. - can we call just a farm module or we need to call only the top one?

[majastrz]

I think making package declaration inside bicep file is not a good idea. Various systems might require different properties. Some of them (ie. git, zip over https) might require none, unless we decide that a nupkg format is required for module packs.

I think we will most likely go with the URI-like format for the module type (#3186), which will have a scheme-like prefix (pkg, nuget, oci, templateSpec, or something like that). As long as the type of the body/value of the package declaration can be determined from from the scheme name, different properties will work fine. Just like different modules or resource types expect different properties in resource or module declarations.

another case might be that if we pack many files into package, and then when we reference a package - if we do not provide specific file name the main.bicep will be used (some kind of entry point).

Regardless if decide on supporting multi-module packages or not, we will need a way to differentiate between the "entry point" (aka main.bicep) module and others. I expect it to based on some metadata or naming convention created by the bicep pack command inside the package. Should be a solvable problem, but I need to do some prototyping before I know more...

open question - if main (or the file with package declaration) uses it’s local files as modules - can we call that files when pointing to a remote? Eg we have a remote package that sets up server farm and app service . Setting up Farm and web app are nested modules called from main. - can we call just a farm module or we need to call only the top one?

Good question. I think it should be an intentional opt-in action to expose a module in a package. "Private" or "local" modules should remain hidden and inaccessible. If we expose them, the back-compat surface gets larger (#3251) and it makes it harder to change any of the internal implementation details in a package.

[BernieWhite]

When emitting an ARM template Bicep already sets a root level metadata property, current including the Bicep version as a generator. Also since Bicep is an abstraction, make sense there would be some correlation here.

{
    "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
    "contentVersion": "1.0.0.0",
    "metadata": {
        "name": "Storage Account",
        "description": "Create or update a Storage Account."
    },
    "parameters": {
        "storageAccountName": {
            "type": "string",
            "metadata": {
                "description": "The name of the Storage Account."
            }
        },
        "tags": {
            "type": "object",
            "metadata": {
                "description": "Tags to apply to the resource.",
                "example": {
                    "service": "<service_name>",
                    "env": "prod"
                }
            }
        }
    },
    "resources": [
    ]
}

Something inline with option 4 seems most logical to me.

Azure/PSDocs.Azure#106

[rouke-broersma]

I wrote this observation in other issue but I will repeat it here - I'd like to have ability to pack multiple independent biceps into single package. Something like a dll having multiple classes to use, not just a single one. You might have a series of small modules and creating feed for every one of them might be way too much.

Multi-module does not seem necessary or useful to me. It creates the need to deal with this construct when it is just as simple to publish your other modules under a different name. There should IMO only be one entrypoint allowed to a bicep module with optionally specifying the name of the entrypoint. Since OCI was chosen as the distribution format for modules this also keeps bicep modules in line with other OCI artifacts such as container images and helm charts.

I think creating the infrastructure to deal with multi-module bicep packages is way more costly than publishing multiple bicep modules especially since when you reference those packages in your bicep you will need to specify the full module name anyway.

[brooke-hamilton]

Metadata at the module level would help with auto-generating documentation, like what PSDocs.Azure does for ARM templates. That tool reads the metadata at the template level, but there's no way to generate the ARM metadata using a Bicep decorator.

Bicep decorators that generate the ARM metadata like this would be helpful:

{
    "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
    "contentVersion": "1.0.0.0",
    "metadata": {
        "name": "Storage Account",
        "description": "Create or update a Storage Account."
    },
...
}

Even if the ARM metadata is not emitted, it would be helpful if the Bicep parser would recognize some module level metadata decorators so that we could develop a doc generation tool that reads the Bicep template rather than the compiled ARM template.

For now we can define our own metadata conventions as comments, like // DESCRIPTION: <my custom description>, but the drawback to that is having each tool define its own required comment-based tags.

[alex-frankel]

This is resolved with the brm tool: https://www.nuget.org/packages/Azure.Bicep.RegistryModuleTool/

[johndowns]

@alex-frankel Is there any plan to document the brm tool?

[alex-frankel]

It is lightly documented in the contribution guide, but we should do a better job of documenting. cc @mumian / @stephaniezyen as FYI.

The only tricky part is using brm requires .net, which makes the tool more trouble than it is worth for most people. I'd want to promote it more if/when it is available is a standalone binary or shipped with the bicep CLI.