Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add job matrix generation scripts #18835

Merged
merged 1 commit into from
Jan 28, 2021
Merged

Conversation

benbp
Copy link
Member

@benbp benbp commented Jan 27, 2021

This adds scripts, docs and samples supporting dynamic, cross-product matrix generation for azure pipeline jobs.
It aims to replicate the cross-product matrix functionality in github actions,
but also adds some additional features like sparse matrix generation, cross-product includes and excludes, parameter grouping and matrix filters.

These changes are live for the .NET tests, so this PR is just a port of the code that already exists in the azure-sdk-for-net repository. The intent is to standardize it across the language repos, and then migrate all the common code into azure-sdk-tools. I will make a follow-on PR with the actual configuration changes to use this functionality in the pipelines, because I want to keep the changes separate to make reviewing/reverting easier.

This functionality is made possible by the ability for the azure pipelines yaml to take a dynamic variable as an input
for a job matrix definition
(see the code sample at the bottom of the linked section).

See the README.md file for more details on the config file syntax and usage, as well as implementation details.

The tests (test-matrix-functions.tests.ps1) contain a lot of detail on expected data structures at various processing stages. The -test-matrix-functions.ps1` file could perhaps be split up or use some more organization, so let me know if it's hard to navigate.

Example:

{
  "displayNames": {
    "true": "TestFromSource"
  },
  "matrix": {
    "Agent": {
      "ubuntu-18.04": { "OSVmImage": "MMSUbuntu18.04", "Pool": "azsdk-pool-mms-ubuntu-1804-general" },
      "windows-2019": { "OSVmImage": "MMS2019", "Pool": "azsdk-pool-mms-win-2019-general" },
      "macOS-10.15": { "OSVmImage": "macOS-10.15", "Pool": "Azure Pipelines" }
    },
    "JavaTestVersion": [ "1.8", "1.11" ],
    "AZURE_TEST_HTTP_CLIENTS": [ "okhttp", "netty" ]
  },
  "include": [
    {
      "Agent": {
          "ubuntu-18.04": { "OSVmImage": "MMSUbuntu18.04", "Pool": "azsdk-pool-mms-ubuntu-1804-general" }
      },
      "JavaTestVersion": "1.11",
      "AZURE_TEST_HTTP_CLIENTS": "netty",
      "TestFromSource": true
    }
  ]
}

Sparse matrix job generation in a pipeline: https://dev.azure.com/azure-sdk/internal/_build/results?buildId=705622&view=results

image

Related discussion: microsoft/azure-pipelines-yaml#20

@benbp benbp self-assigned this Jan 27, 2021
@benbp benbp added the Central-EngSys This issue is owned by the Engineering System team. label Jan 27, 2021
@benbp benbp merged commit 86ce538 into Azure:master Jan 28, 2021
@benbp benbp deleted the benbp/job-matrix branch January 28, 2021 00:13
azure-sdk pushed a commit to azure-sdk/azure-sdk-for-java that referenced this pull request May 23, 2022
Update Microsoft.MixedReality swagger to fix quality issues in s360 (Azure#18835)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Central-EngSys This issue is owned by the Engineering System team.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants