Looping & Conditionals syntax spec

[anthony-c-martin]

Proposal - Looping & Conditionals

Goals

• First class support for conditionally deploying resources or sets of resources
• Simple to understand syntax without losing any of the power
• Avoid introducing multiple ways to do the same thing - aim to unify resource-level and property-level looping constructs

Resource-Level Looping: Spec

Example of a regular resource with no modifiers:

resource <provider> <type> <identifier>: {
  ...
}

Conditional modifier

resource <provider> <type> <identifier> when <conditional_expression>: {
  ...
}

Example

resource azrm 'Microsoft.Network/networkInterfaces@2019-10-01' myNic when (deployNic == true): {
  ...
}

The type assigned to the declared <identifier> should be <resource type> | null. This should allow for compile time verification that the user is safely accessing the value with appropriate null checks.

Looping modifier

// element_identifier gives access to the array element or object key
resource <provider> <type> <identifier> for <element_identifier> in <array_expression>: {
  ...
}

// optional index_identifier to get access to a loop index
resource <provider> <type> <identifier> for <element_identifier>, <index_identifier> in <array_expression>: {
  ...
}

Examples

// using the range(..) function to create a list of integers
resource azrm 'Microsoft.Network/networkInterfaces@2019-10-01' myNics for i in range(5): {
  name: 'mynic-${i}'
  ...
}

// using index plus element access
resource azrm 'Microsoft.Network/networkInterfaces@2019-10-01' myNics for name, i in nicNames: {
  name: name,
  properties: {
    index: i
  }
}

The type assigned to the declared <identifier> should be an array of the resource being declared.

<loop_identifier> will be assigned access to the item in the current iteration of the loop.

Conditional and Looping modifiers

The MVP will not support both modifiers being combined. If a user wants to combine, they always have the option to use looping with <condition> ? <array_expression> : range(0) to access the same functionality.

Notes/Caveats

• Looping is supported on arrays as well as objects. For arrays, the identifier allow iteration over elements. For objects, the identifier allows iteration over keys. We may want to consider allowing access to the index in both cases, as it's quite commonly used in ARM.
• Line length could potentially get quite long, and the mixing of multiple keywords and identifiers may make it hard to understand which identifiers are doing what (e.g. which identifier is the resource).
• Loop 'modifiers' such as ARM's batch count, and copy mode are not included in this spec, and some generic form of annotation will need to be added for this - being careful to avoid making it 'ARM-specific'.
• The expression being passed to either the condition or loop statements cannot reference any resource directly or indirectly. This is so the full deployment graph can be built before any resources are deployed.

Potential future improvements not covered in this spec

• Range expression (syntactic sugar for the range function): n...m
• Combining both conditionals and looping on a resource

Property-Level Looping: Spec

Looping

Looping uses a very similar syntax to resource-level looping, and behaves like a map function, where the value generated is written inline, and has access to the item being iterated over. This syntax generates an array of items.

This syntax is valid in any place expecting an expression.

{
  <property>: for <identifier> in <array_expression> <value_expression>,
  ...
}

Example

{
  myLoop: for i in range(2) {
    name: i
  },
  myLoop2: for name in names '${prefix}-${name}',
}

Conditionals

Conditionals use a ternary syntax. This syntax is valid in any place expecting an expression.

{
  <property>: <conditional_expression> ? <val_a> : <val_b>
}

Notes/Caveats

• It's difficult to unify with the resource-level conditional statement, so I've gone for a standard ternary syntax. We may want to think about making a syntax for a ? b : null quick to write as either setting some value or null is a common use case.
• Again, optional access to an index would probably be quite useful.
• The lack of visual separator between <array_expression> <value_expression> makes the looping syntax a little hard to parse.

[bmoore-msft]

Love it... some thoughts in pseudo-random order...

• I like when, can I use “when” on a property? As long as we have the equivalent of json(null) I don’t think we need it but consistency is always good – even though we can still do a ternary syntax

• I think we might need an “else” to go along with when? Thinking about the new/existing pattern we need an elegant way to support that – in context would be best… given that we now have symbolic references, we either need a way to “else” the resource definition or a way to declare a [symbolic] reference to an existing resource somewhere in the template. We need the latter for sure and this scenario doesn't have context (so an else construct doesn't help the latter case).

• FWIW, I thnk the identifier should follow the resource keyword, not the type… e.g.

resource subnets Microsoft.Network/virtualNetworks/subnets {}

Aside:

resource subnets /subnets {}

Should work when /subnets is not ambiguous???

• Is there a syntax with “copies” instead of “for” that would work? We are actually creating copies of a resource… "for" may make sense “above” the resource body, but I personally don’t like that style of looping in a declarative lang:

resource subnets Microsoft.Network/virutalNetworks/subnets copies subnetPrefixes.length as s {}

I don't love it, but wondering if it has legs

• Line length – we should allow line breaks, even if it means I need a continuation char, eg.

resource subnets[] Microsoft.Network/virtualNetworks/subnets
         for i in subnetPrefixes 
         when newOrExistingVnet == true {
}

• +1 on accessing object arrays through the key – this would be a regression if we don’t allow (and not sure how the pattern would work then)

• I’m not 100% certain that batch/mode are ARM specific, could exist in other clouds/scenarios – in any case I think we could suffice with a “batchSize” modifier that would enable both, e.g.

for i in subnetPrefixes batchSize 1

• property level looping – I don’t understand the 2 different examples (and when I’d use one vs. the other) - are myLoop and myLoop2 identifiers or property names?

• re: <array_expression> <value_expression> - In practice this would have to be an object or array – that syntax would separate, (I think) , e.g.

for i in dataDiskSizes [{
	name: lun + i
	lun: i
	sizeInGb: dataDiskSizes[i]
}]

Nit - It would be good to put a “completed” example in the “spec” so we can see the ideas come together, I kept trying that mental exercise as I was parsing this (I'm not great engineer, but I'm a good reverse engineer).

[alex-frankel]

Overall, I'm not in love with the syntax, but I do think it will work. Personally, I feel separating the condition and/or looping statement from the resource declaration helps with readability even though it might be a bit more verbose to author. The syntax feels like I'm writing a SQL query and less like authoring a programming language.

Separating the syntax allows us to cover the "multi-resource" case and the single resource case with one syntax. If we ever do want to support a multi-resource syntax, then we will have two different ways to author conditions and loops.

• +1 on else-like functionality. I think it's critical we nail the new/existing pattern.
• I don't see an example with batchSize or mode - do we need to have the covered?

@bmoore-msft:

• I think myLoop returns an array of objects and myLoop2 returns an array of strings, but it is a little confusing.
• multi-line strings will work without continuation chars. We're pretty firm on not having meaningful whitespace
• I'm not following your /subnets aside

[majastrz]

The syntax for loops and conditionals builds on a plain resource declaration and I feel like we haven't truly nailed that down yet. Should we look at that first?

@alex-frankel and @bmoore-msft

• else gets tricky with when if we allow users to declare a resource of a completely different type when we reference the resource by name. To make that work, the user would have to basically duplicate the condition. We avoid the problem altogether by either not supporting it or requiring the when and else resource types to be of the same type.

@alex-frankel

• I don't love the syntax, either. Something is off with readability or verbosity, but it's hard to put a finger on it :(
• I agree that separating condition from the loop will improve readability - in imperative languages, they are usually on separate lines.

@bmoore-msft

• I agree that for is too imperative, but I'm not sure we're making copies here. It's more like we're transforming an input list into a list of resources. In C#, a similar manipulation uses a Select function with a lambda. In JS, it's called map I believe. On the other hand, In Anders' gist, he was using a for in a declarative way and that worked pretty well too. Maybe it's ok?
• I noticed the inconsistency between the ternary ? operator and when on resource too. I guess one way to justify is because the resource keyword is declaring a side effect of resource deployment where as ? just returns a value.
• Not sure I follow the subnets example, either. What do you mean?
• Definitely agree that we need to handle some of the copy loop settings somehow. @anthony-c-martin had a proposal in Resource Annotations #1.
• Anywhere there's whitespace between keywords or identifiers, a new line can be inserted. We can let the user format as they see fit. Although I imagine at some point, we'll have to build an auto-formatter with some canonical formatting.
• In the example with the following snippet, I read the following as creating an array of arrays of objects rather than an array of objects.

for i in dataDiskSizes [{
	name: lun + i
	lun: i
	sizeInGb: dataDiskSizes[i]
}]

[majastrz]

Also, should we consider a more imperative-looking but more familiar syntax for loops and conditions? Something like this, for example:

if(condition) {
  for (i, j) in range(10) {
    resource ..............
  }
}

[alex-frankel]

this is my preference, but I think the challenge with this syntax is how you reference the resource outside of the scope in which it is declared.

At one point we were considering an as syntax:

if (condition) as myCondition {
  for i in range(10) as loop {
    resource myResource ...
  }
}

so to reference would you do the following?

myCondition.loop[0].myResource

it gets weird

[majastrz]

Good pt, I remember now.

[anthony-c-martin]

Ignoring the syntax, at a high level I feel like these are the viable options for resource-level looping/conditionals:

1. Inline with the resource declaration (this proposal)
   Pros: feels the most readable in the simple case.
   Cons: could become unreadable if care isn't taken when authoring. extensibility will be difficult to tack on (batching, serial copy mode).

resource myResource <provider> <type> when (deployMyResource == true): {
  ...
}

2. Above the resource declaration (e.g. something like C# Attributes)
   Pros: avoids long lines. extending (e.g. with batching, serial copy mode, whatever) is simple.
   Cons: could feel unnatural that the condition is separate from the resource.

@if deployMyResource == true
resource myResource <provider> <type>: {
  ...
}

3. Inside the resource body
   Pros: everything is together, avoids longer lines.
   Cons: the mixing of 'config' and 'metadata'/'control flow logic' feels potentially confusing.

resource myResource <provider> <type>: {
  @if: deployMyResource == true
  ...
}

4. 'imperative-style' but with weird scoping behavior
   Pros: more familiar at first to an imperative programmer. multiple resources can be declared in the same block.
   Cons: scoping behavior is unfamiliar and could cause significant confusion. the type system would either have to be robust enough to handle declaration of different resources in different branches with the same identifier, or simply forbid it.

if (deployMyResource == true) {
  resource myResource <provider> <type>: {
    ...
  }
}
// myResource can be accessed on the outer scope here

[anthony-c-martin]

Amended spec to:

1. Remove combining of 'when' and 'for' on a resource.
2. Add note about future range expression syntax.

[anthony-c-martin]

Putting this on hold for now as it's not going to be part of milestone 0.

[majastrz]

@lwang2016 just added #42 which covers his looping syntax proposal. Linking it here so everyone can see.

[lwang2016]

Addressed comments from #42 and updated proposal with #45

[majastrz]

Out of @shenglol's (#44) and @lwang2016's (#42 and #45), we have settled on the former for the following reasons:

• Consistency between property loops and resource loops
• The chosen syntax does not allow multiple resources in the loop body but it is a rare situation in templates and we are not going to support it initially.
• for x in y or foreach x in y syntax is familiar to users of many programming languages including PowerShell. Concepts learned elsewhere can be reapplied here.

resource databases: 'Microsoft.DocumentDB/databaseAccounts/sqlDatabases@2020-03-01' = [for databaseName in databaseNames: {
    name: "${accountName}/{databaseName}",
    kind: "GlobalDocumentDB",
    dependsOn: dbAccount,
    properties: {
        id: databaseName,
        // The same syntax can be reused for array properties.
        someArrayProperty: [for item in collection: {
            foo: item.foo,
            bar: item.bar,
        }]
    },
}]

for item in collection: syntax can also be replaced with for (item, i) in collection: to access the index within the loop body. For full examples, see #44.

[bmoore-msft]

How would I access the counter in a loop with:
for database in databaseNames: {}
?

[majastrz]

for (database, i) in databaseNames (last line of my comment above)

[majastrz]

Discussed adding a filtering capability to the looping construct we selected previously. We settled on using the where keyword for this purpose (similar to Where-Object in PowerShell or Where in C#) Here's an example that also demonstrates referencing identifiers from the parent loop in the inner loop:

resource listOfStuff '...' = [for thing in stuff where (thing.enabled) {
  name: thing.name
  properties: {
    listOfOtherStuff: [for otherThing in thing.otherStuff where (thing.enabled && otherThing.enabled) {

    }]
  }
}]

@alex-frankel brought up a point that our resource declarations are getting long. We may need to allow the user to format them however they see fit without enforcing a specific format.

Also discussed the condition syntax. There are limitations in the IL that prevent us from compiling a complex if/else construct when combined with a reference() function, so we will keep things simple to start with. The initial public release will use the when syntax without support for else orelse when capabilities. We will add more complex constructs in the future as the language evolves and as we improve the capabilities of the IL. The more advanced capabilities of @lwang2016's proposal or @shenglol's switch proposal will be reconsidered then.

We have also decided that a conditional resource symbol whose condition is false will have an undefined value. We discussed what happens when a property of a such a resource is declared in another resource. It is clear that we will generate a dependsOn in the JSON automatically, but we will not automatically propagate the condition to all resources. The side effect of this that we may not be able to catch all type errors at compile-time and some will get deferred to runtime.

For example, given resource foo 'Microsoft.Example/examples@2020-06-01' when (condition) = {}, foo.name is of type Microsoft.Example/examples@2020-06-01 | undefined without knowing the value of condition. If foo.name is assigned to a string property, then we cannot really generate an error or a warning because we don't know the type in practice unless we introduce some operator to override the warning easily (similar to how C# nullability uses the ! operator).

[majastrz]

@alex-frankel, @marcre, @shenglol please add more comments if I missed anything from the meeting.

[majastrz]

Created #61 to update the language spec in the repo.

[majastrz]

Closing this as #61 was merged.