design: each expression #2

ericsciple · 2018-10-03T19:20:38Z

No description provided.

ericsciple · 2018-10-03T19:23:57Z

@mmitche @chcosta check this out. We've iterated on this quite a bit on our side. We think this will meet the scenario where you need to bootstrap each job in the graph with pre steps (setup microbuild) and post steps (publish telemetry).

ericsciple · 2018-10-03T19:38:37Z

@chrisrpatterson

mmitche · 2018-10-04T15:38:11Z

design/each-expression.md

+    pool: ${{ job.pool }}
+    workspace: ${{ job.workspace }}
+    steps: ${{ job.steps }}
+```


Does this mean this could be written as such (simpler, no need to explicit list out everything)

parameters: jobs: [] jobs: - job: CredScan # Cred scan first pool: MyCredScanPool steps: - task: MyCredScanTask@1 ${{ each job in parameters.jobs }} - ${{ each pair in job }}: # Insert all properties other than "steps" ${{ if ne(pair.key, 'dependsOn') }}: ${{ pair.key }}: ${{ pair.value }} ${{if job.dependsOn}} dependsOn: - ${{ CredScan }} - ${{ job.dependsOn }} ${{ if not(job.dependsOn) }}: dependsOn: - CredScan

design/each-expression.md

vtbassmatt · 2018-10-15T18:16:37Z

Could we do this in $[ ] expressions, too?

variables:
  NODE_VERS: ['6.x', '8.x', '10.x']
  TS_VERS: ['2.8', '2.9', '3.0']
strategy:
  matrix:
    $[ each nodeVer in variables.NODE_VERS ]:
      $[ each tsVer in variables.TS_VERS ]:
        $[ concat(nodeVer, '_', tsVer) ]:
          NODE_VERSION: $[ nodeVer ]
          TS_VERSION: $[ tsVer ]
steps:
- script: echo Node $(NODE_VERSION) against TypeScript $(TS_VERSION)

It's not the prettiest way to create a matrix, for sure. I predict people will try to do it, though, and the same thing is even grosser if you're forced into a template for it :)

chrispat · 2018-10-16T01:51:42Z

@vtbassmatt I would much rather have a specific function for doing a cross product a variable array and apply some exclusions. Seems cleaner.

4c74356b41 · 2018-10-16T05:51:24Z

my comments got deleted, eh?

chcosta · 2018-10-16T15:34:43Z

@4c74356b41, looks like your comments were added as a part of mmitche's review, which was resolved but is still available.

4c74356b41 · 2018-10-16T15:41:30Z

can you provide a link, i cant seem to find those. thanks. @chcosta

chcosta · 2018-10-16T15:41:52Z

#2 (comment)

4c74356b41 · 2018-10-16T15:47:57Z

ok, why is it resolved than? I dont mean to say I'm the most important guy out here, but comments are there for a reason. I work with these yaml definitions and they are pretty horrible for anything remotely clever. why cant I use concats? or other common string\object manipulation functions?

chcosta · 2018-10-16T15:57:54Z

mmitche's review was resolved (likely via a GitHub push). If you have feedback, you should do your own review or start another comment thread so that the topics don't get lost in the mix.

mikeharder · 2018-10-18T22:27:09Z

@vtbassmatt , @chrisrpatterson: Is there a way to create a cross product matrix today? If not, should a separate issue be filed for this?

vtbassmatt · 2018-10-19T13:24:18Z

@mikeharder there isn't a command for cross-product matrix. Feel free to file it; it's on our minds but not highly prioritized right now.

chcosta · 2018-10-19T15:29:51Z

What is meant by "cross-product matrix"? @vtbassmatt , is that the same request as https://dnceng.visualstudio.com/7ea9116e-9fac-403d-b258-b31fcf1bb293/_workitems/edit/59 or a different scenario?

mikeharder · 2018-10-19T16:34:40Z

@chcosta: It looks like https://dnceng.visualstudio.com/7ea9116e-9fac-403d-b258-b31fcf1bb293/_workitems/edit/59 is the same as #4. The ability to use a variable (in this case from a matrix) to set vmImage.

By "cross-product matrix", I mean the scenario where you want to use the matrix strategy on a cross-product of multiple dimensions. For example, say you want to run tests on 2 OS (Linux, Windows) and 2 versions of Python (3.6, 3.7). Currently, you'd need to manually create a matrix with all the combinations:

strategy:
  matrix:
    Linux_Python36:
      VM_IMAGE: 'ubuntu-16.04'
      PYTHON_VERSION: '3.6'
    Linux_Python37:
      VM_IMAGE: 'ubuntu-16.04'
      PYTHON_VERSION: '3.7'
    Windows_Python36:
      VM_IMAGE: 'vs2017-win2016'
      PYTHON_VERSION: '3.6'
    Windows_Python37:
      VM_IMAGE: 'vs2017-win2016'
      PYTHON_VERSION: '3.7'

While this isn't too bad with a small number of dimensions and a small number of variables in each dimension, it can quickly become unmaintainable as the dimensions and variables grow. It would be nice if there was a way to express this more succinctly, something like:

strategy:
  cross-product:
    PYTHON_VERSION: [ '3.6', '3.7' ]
    VM_IMAGE: [ `ubuntu-16.04`, `vs2017-win2016` ]

Tracking issue: #20

vtbassmatt · 2018-10-19T18:21:45Z

We've shied away from cross-product for two reasons:

Now you have to introduce an exception syntax. I don't think I've ever seen a higher-dimension matrix that didn't have at least one exception.
Especially when n=3 or higher, it becomes painfully easy to run dozens/hundreds of jobs unintentionally. I know this sounds crazy since so many other CI systems do it, but we've seen it happen many times with our designer-based builds (which had the cross-product concept).

Notwithstanding the above, I'm open to the concept. Let's start a separate issue about it and have the conversation there. This PR's about each.

mikeharder · 2018-10-19T18:42:21Z

@vtbassmatt: Separate issue for cross-product: #20

andrewsuiter · 2019-01-07T22:17:36Z

@ericsciple, if my solution has multiple projects, I would like to loop through each project and publish build artifacts

task: PublishBuildArtifacts@1
${{ each job in parameters.project }}
displayName: 'Publish Artifact: drop'
inputs:
PathtoPublish: '$(Build.ArtifactStagingDirectory)/${{parameters.project}}.zip'
How would I loop through each project and set the .zip file for each project?
I would like to zip up project 1 and project 2.

Thanks,

ericsciple · 2019-01-08T01:19:42Z

Can you add two tasks?

steps:
- ${{ each job in parameters.project }}:
  - task: PublishBuildArtifacts@1
    displayName: 'Publish ${{ parameters.project }}
    inputs:
      PathtoPublish: '$(Build.ArtifactStagingDirectory)/${{parameters.project}}.zip'

ericsciple · 2019-01-08T01:20:05Z

...@andrewsuiter

andrewsuiter · 2019-01-08T17:05:39Z

@ericsciple I could always do that but there may be a situation where there are multiple projects in one solution, some may have 5, some may have 3 projects. It would make it easier if I could loop through each project that is defined as an array in the parameter.

parameters;
project: 'name: test1 project: project1, name: test2 project: project2, name: test3 project: project3'

Thanks

UPDATE: How do I pass the project parameters?

parameters;
project: [ 'TEST1', 'TEST2']?

andrewsuiter · 2019-01-09T15:38:22Z

@ericsciple any update as to how to pass the project parameter? I am using the each each expression but getting an array can not be converted to a string error.

ericsciple · 2019-01-09T15:58:20Z

@andrewsuiter sorry, my example was wrong.

Here is a better example:

# my-template.yml
parameters:
steps:
- ${{ each project in parameters.project }}:
  - task: PublishBuildArtifacts@1
    displayName: 'Publish ${{ project }}
    inputs:
      PathtoPublish: '$(Build.ArtifactStagingDirectory)/${{ project }}.zip'

# ci.yml
steps:
- template: my-template.yml
  parameters:
  - test1
  - test2

That way, one task is added for each item in the array parameters.project. Does that work? If not, can you paste the cotents of both files (the template, and the caller of the template)

andrewsuiter · 2019-01-09T15:59:41Z

ok great, I will give it a shot, thanks for the help!

andrewsuiter · 2019-01-09T17:17:58Z

@ericsciple That works perfect, exactly what I was looking for, much appreciated!

idubnori · 2019-01-16T15:32:27Z

@ericsciple Is there any way to use nested within parameters?
The example below to be error...

# my-template.yml
parameters:
steps:
- ${{ each pf in parameters.pf }}:
  - task: PublishBuildArtifacts@1
    displayName: 'Publish ${{ pf.rid }} ${{ pf.output }}'
    inputs:
      PathtoPublish: '$(Build.ArtifactStagingDirectory)/${{ pf.output }}.zip'

# ci.yml
steps:
- template: my-template.yml
  parameters:
  - windows:
      rid: 'win-x64'
      output: 'Foo.exe'
  - macos:
      rid: 'osx-x64'
      output: 'foo'

ravican09 · 2020-12-22T01:48:20Z

Hi Team
Sub: Variables in YAML pipeline are not allowing to define array
Error : A sequence was not expected,
Please advise how to define variables in YAML pipelines to define array.

Code:

variables:

name: MyVariableName
value: ['Value1','Value2']

I want to use/pass MyVariableName to template.yml where i want to iterate array

parameters:
MyVariableName: [] #defining array here to expect multiple values.

steps:

${{ each hostname in parameters.MyVariableName}}:
logic

harre096 · 2022-05-24T22:22:48Z

@andrewsuiter sorry, my example was wrong.

Here is a better example:
# my-template.yml
parameters:
steps:
- ${{ each project in parameters.project }}:
  - task: PublishBuildArtifacts@1
    displayName: 'Publish ${{ project }}
    inputs:
      PathtoPublish: '$(Build.ArtifactStagingDirectory)/${{ project }}.zip'
# ci.yml
steps:
- template: my-template.yml
  parameters:
  - test1
  - test2
That way, one task is added for each item in the array parameters.project. Does that work? If not, can you paste the cotents of both files (the template, and the caller of the template)

@ericsciple - FYI the quote is missing in the displayName of your example:
displayName: 'Publish ${{ project }}'

mmitche reviewed Oct 4, 2018

View reviewed changes

design/each-expression.md Show resolved Hide resolved

mmitche approved these changes Oct 4, 2018

View reviewed changes

chcosta approved these changes Oct 4, 2018

View reviewed changes

vtbassmatt mentioned this pull request Oct 5, 2018

Variable number of step in templates microsoft/azure-pipelines-agent#1701

Closed

kaylangan mentioned this pull request Oct 15, 2018

Set up CI with Azure Pipelines TypeStrong/ts-loader#856

Closed

design: each expression

7005acc

ericsciple force-pushed the users/ersciple/m142each branch from f52fb06 to 7005acc Compare October 25, 2018 16:28

ericsciple merged commit 8865554 into master Oct 25, 2018

ericsciple deleted the users/ersciple/m142each branch October 25, 2018 16:29

mitchdenny mentioned this pull request Jan 30, 2019

Proposals/pipeline caching #113

Merged

JustinGrote mentioned this pull request May 28, 2019

Variable/List from file? #219

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

design: each expression #2

design: each expression #2

ericsciple commented Oct 3, 2018

ericsciple commented Oct 3, 2018

ericsciple commented Oct 3, 2018

mmitche Oct 4, 2018 •

edited

Loading

ericsciple Oct 4, 2018

vtbassmatt commented Oct 15, 2018

chrispat commented Oct 16, 2018

4c74356b41 commented Oct 16, 2018

chcosta commented Oct 16, 2018

4c74356b41 commented Oct 16, 2018

chcosta commented Oct 16, 2018

4c74356b41 commented Oct 16, 2018

chcosta commented Oct 16, 2018

mikeharder commented Oct 18, 2018

vtbassmatt commented Oct 19, 2018

chcosta commented Oct 19, 2018

mikeharder commented Oct 19, 2018 •

edited

Loading

vtbassmatt commented Oct 19, 2018

mikeharder commented Oct 19, 2018

andrewsuiter commented Jan 7, 2019

ericsciple commented Jan 8, 2019

ericsciple commented Jan 8, 2019

andrewsuiter commented Jan 8, 2019 •

edited

Loading

andrewsuiter commented Jan 9, 2019

ericsciple commented Jan 9, 2019

andrewsuiter commented Jan 9, 2019

andrewsuiter commented Jan 9, 2019

idubnori commented Jan 16, 2019

ravican09 commented Dec 22, 2020 •

edited

Loading

logic

harre096 commented May 24, 2022 •

edited

Loading

design: each expression #2

design: each expression #2

Conversation

ericsciple commented Oct 3, 2018

ericsciple commented Oct 3, 2018

ericsciple commented Oct 3, 2018

mmitche Oct 4, 2018 • edited Loading

Choose a reason for hiding this comment

ericsciple Oct 4, 2018

Choose a reason for hiding this comment

vtbassmatt commented Oct 15, 2018

chrispat commented Oct 16, 2018

4c74356b41 commented Oct 16, 2018

chcosta commented Oct 16, 2018

4c74356b41 commented Oct 16, 2018

chcosta commented Oct 16, 2018

4c74356b41 commented Oct 16, 2018

chcosta commented Oct 16, 2018

mikeharder commented Oct 18, 2018

vtbassmatt commented Oct 19, 2018

chcosta commented Oct 19, 2018

mikeharder commented Oct 19, 2018 • edited Loading

vtbassmatt commented Oct 19, 2018

mikeharder commented Oct 19, 2018

andrewsuiter commented Jan 7, 2019

ericsciple commented Jan 8, 2019

ericsciple commented Jan 8, 2019

andrewsuiter commented Jan 8, 2019 • edited Loading

andrewsuiter commented Jan 9, 2019

ericsciple commented Jan 9, 2019

andrewsuiter commented Jan 9, 2019

andrewsuiter commented Jan 9, 2019

idubnori commented Jan 16, 2019

ravican09 commented Dec 22, 2020 • edited Loading

logic

harre096 commented May 24, 2022 • edited Loading

mmitche Oct 4, 2018 •

edited

Loading

mikeharder commented Oct 19, 2018 •

edited

Loading

andrewsuiter commented Jan 8, 2019 •

edited

Loading

ravican09 commented Dec 22, 2020 •

edited

Loading

harre096 commented May 24, 2022 •

edited

Loading