Add cross product matrix tooling

[benbp]

This adds scripts, docs and samples supporting dynamic, cross-product matrix generation for azure pipeline jobs.
It aims to replicate the cross-product matrix functionality in github actions,
but also adds some additional features like sparse matrix generation, cross-product includes and excludes, and (TODO) programmable matrix filters.

This functionality is made possible by the ability for the azure pipelines yaml to take a dynamic variable as an input
for a job matrix definition (see the code sample at the bottom of the linked section).

See the README.md file for more details on the config file syntax and usage, as well as implementation details.

The tests (functions.tests.ps1) contain a lot of detail on expected data structures at various processing stages. The functions.ps1 file could perhaps be split up or use some more organization, so let me know if it's hard to navigate.

Example:

{
  "displayNames": {
    "/p:UseProjectReferenceToAzureClients=true": "UseProjectRef"
  },
  "matrix": {
    "operatingSystem": [
      "windows-2019",
      "ubuntu-18.04",
      "macOS-10.15"
    ],
    "framework": [
      "net461",
      "netcoreapp2.1",
      "net50"
    ]
  },
  "include": [
    {
      "operatingSystem": "windows-2019",
      "framework": "net461",
      "additionalTestArguments": "/p:UseProjectReferenceToAzureClients=true"
    }
  ],
  "exclude": [
    {
      "operatingSystem": "windows-2019",
      "framework": "net461"
    }
  ]
}

Sparse matrix job generation in a pipeline: https://dev.azure.com/azure-sdk/playground/_build/results?buildId=643038&view=results

Related discussion: microsoft/azure-pipelines-yaml#20

[azure-sdk]

The following pipelines have been queued for testing:
java - template
js - template
net - template
python - template
You can sign off on the approval gate to test the release stage of each pipeline.
See eng/common workflow

[weshaggard]

I know this is scoped to a folder but I'd still call this something more specific then functions.

[weshaggard]

We are likely going to want to have a way to set the output variable name.

[weshaggard]

For usage we should use the task directly for better debugging. See https://dev.azure.com/azure-sdk/internal/_wiki/wikis/internal.wiki/2/Azure-DevOps-Pipeline-Guidance?anchor=devops-tasks-vs-pwsh-script-shortcut

[weshaggard]

I suspect this should actually be a shared yml template for folks to include as well.

[weshaggard]

Would it be after processing exclusions? I would expect the higher level filter to potentially exclude this include as well.

[benbp]

I based the behavior off the github actions behavior. I think the reasoning is that you could do a broad exclude, but then need to add something back. Once we set the behavior we'll want to stick to it, so maybe we should brainstorm some scenarios. We could also support exclude/include priority but that feels like an overcomplication (which is ironic to say given what this PR is...).

[weshaggard]

I was thinking about the scenario of I want to run only one entry from the matrix for manual testing. So I was expecting a constraint that runs after the full matrix processing is done.

[weshaggard]

I tend to put write functions like function CreateDisplayName([string]$parameter, [hashtable]$displayNames) and put the { on a new line for larger scopes. While what you have is correct I've never liked using the extra verbose param for functions.

[weshaggard]

instead of having 2 entry point scripts should we just make it a parameter?

[benbp]

I'll do that. This whole entry point is going to go away as well once I add support (which all/sparse will be implemented through).

[benbp]

Closing in favor of landing in a non-shared repo first: Azure/azure-sdk-for-net#17417