-
Notifications
You must be signed in to change notification settings - Fork 527
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
[ci] Use AZDO built-in parallelization strategy. (#7804)
[Previously](#6360), we split our `MSBuildIntegration` unit tests to run across multiple CI test agents. While a huge improvement, there are a few downsides to the approach we went with at the time: - The number of agents is hardcoded by copy/pasting steps in the CI YAML. - The tests must be manually load-balanced across agents with `[Category ("Node-X")]` in code which is cumbersome and hard to keep updated. As we are at the point where our tests have expanded well past the 1 hour target we are faced with needing to increase our parallelization. First, we can remove the YAML duplication by using AZDO's built-in `parallel` strategy, allowing us to control the number of agents to use with a single number: ```yaml - job: strategy: parallel: 4 ``` This leaves us the issue of automatically splitting our tests among an unknown number of test agents. A clever solution is [provided in the AZDO docs](https://learn.microsoft.com/en-us/azure/devops/pipelines/test/parallel-testing-any-test-runner?view=azure-devops): - Use `dotnet test --list-tests` to find all the tests - Use a script to calculate which tests the agent should run, based on `$(System.JobPositionInPhase)` and `$(System.TotalJobsInPhase)` - Pass those test names into `dotnet test` as a filter Unfortunately there are issues with the provided approach: - `dotnet test --list-tests` is not compatible with `--filter` so you have to run *all* tests in the test assembly which is not desirable for us. (dotnet/sdk#8643) - Passing test names (including test parameters) on the command line hits limitations with escaping certain characters and argument limits. So the approach is good, but we have to do all the work ourselves. Enter [dotnet-test-slicer](https://github.com/jpobst/dotnet-test-slicer). This dotnet global tool: - Uses the `NUnit.Engine` NuGet package to find all tests in a test assembly using the desired filter query. - Slices the test list for the current test agent. - Outputs the desired tests into an NUnit-specific `.runsettings` file that can be passed to `dotnet test --settings foo.runsettings`, bypassing command line arguments. Voila! Now we can automatically scale our test agents up or down as needed by changing a single variable in our YAML file. Limitation: The tests are sliced in a round robin fashion. This can produce uneven results if test durations are uneven. A future optimization would be to store approximate test durations so tests can be load balanced more intelligently.
- Loading branch information
Showing
17 changed files
with
191 additions
and
127 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
15 changes: 15 additions & 0 deletions
15
build-tools/automation/yaml-templates/install-dotnet-test-slicer.yaml
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,15 @@ | ||
| parameters: | ||
| version: '0.1.0-alpha1' | ||
| condition: succeeded() | ||
| continueOnError: true | ||
|
|
||
| steps: | ||
| - powershell: dotnet tool uninstall dotnet-test-slicer -g | ||
| displayName: uninstall dotnet-test-slicer | ||
| ignoreLASTEXITCODE: true | ||
| condition: ${{ parameters.condition }} | ||
|
|
||
| - script: dotnet tool update dotnet-test-slicer --version ${{ parameters.version }} --add-source https://api.nuget.org/v3/index.json -g | ||
| displayName: install dotnet-test-slicer ${{ parameters.version }} | ||
| condition: ${{ parameters.condition }} | ||
| continueOnError: ${{ parameters.continueOnError }} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
152 changes: 152 additions & 0 deletions
152
build-tools/automation/yaml-templates/stage-msbuild-emulator-tests.yaml
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,152 @@ | ||
| # Runs MSBuild tests against a device running on macOS | ||
|
|
||
| parameters: | ||
| job_name: 'mac_dotnetdevice_tests' | ||
| agent_count: 6 | ||
| nunit_categories: $(DotNetNUnitCategories) | ||
| target_framework: $(DotNetStableTargetFramework) | ||
| provisionatorChannel: latest | ||
| stageCondition: succeeded() | ||
| jobCondition: succeeded() | ||
|
|
||
| stages: | ||
| - stage: msbuilddevice_tests | ||
| displayName: MSBuild Emulator Tests | ||
| dependsOn: mac_build | ||
| condition: ${{ parameters.stageCondition }} | ||
| jobs: | ||
| - job: ${{ parameters.job_name }} | ||
| strategy: | ||
| parallel: ${{ parameters.agent_count }} | ||
| displayName: "macOS > Tests > MSBuild+Emulator" | ||
| condition: ${{ parameters.jobCondition }} | ||
| pool: | ||
| vmImage: $(HostedMacImage) | ||
| timeoutInMinutes: 90 | ||
| cancelTimeoutInMinutes: 5 | ||
| workspace: | ||
| clean: all | ||
| steps: | ||
| - template: setup-test-environment.yaml | ||
| parameters: | ||
| provisionClassic: false | ||
| provisionatorChannel: ${{ parameters.provisionatorChannel }} | ||
| installTestSlicer: true | ||
|
|
||
| - template: run-xaprepare.yaml | ||
| parameters: | ||
| displayName: install emulator | ||
| arguments: --s=EmulatorTestDependencies --android-sdk-platforms="19,21,26,32,33" | ||
|
|
||
| - task: DownloadPipelineArtifact@2 | ||
| inputs: | ||
| artifactName: $(TestAssembliesArtifactName) | ||
| downloadPath: $(System.DefaultWorkingDirectory)/bin/Test$(XA.Build.Configuration) | ||
|
|
||
| - pwsh: | | ||
| dotnet-test-slicer ` | ||
| --test-assembly="$(System.DefaultWorkingDirectory)/bin/Test$(XA.Build.Configuration)/MSBuildDeviceIntegration/${{ parameters.target_framework }}/MSBuildDeviceIntegration.dll" ` | ||
| --test-filter="cat != TimeZoneInfo & cat != Localization ${{ parameters.nunit_categories }}" ` | ||
| --slice-number=$(System.JobPositionInPhase) ` | ||
| --total-slices=$(System.TotalJobsInPhase) ` | ||
| --outfile="$(System.DefaultWorkingDirectory)/bin/Test$(XA.Build.Configuration)/MSBuildDeviceIntegration/${{ parameters.target_framework }}/MSBuildDeviceIntegration.runsettings" | ||
| displayName: Slice unit tests | ||
| - task: MSBuild@1 | ||
| displayName: start emulator | ||
| inputs: | ||
| solution: tests/Mono.Android-Tests/Mono.Android-Tests.csproj | ||
| configuration: $(XA.Build.Configuration) | ||
| msbuildArguments: /t:AcquireAndroidTarget /bl:$(System.DefaultWorkingDirectory)/bin/Test$(XA.Build.Configuration)/start-emulator.binlog | ||
|
|
||
| - template: run-nunit-tests.yaml | ||
| parameters: | ||
| useDotNet: true | ||
| testRunTitle: MSBuildDeviceIntegration On Device - macOS-$(System.JobPositionInPhase) | ||
| testAssembly: $(System.DefaultWorkingDirectory)/bin/Test$(XA.Build.Configuration)/MSBuildDeviceIntegration/${{ parameters.target_framework }}/MSBuildDeviceIntegration.dll | ||
| dotNetTestExtraArgs: --settings "$(System.DefaultWorkingDirectory)/bin/Test$(XA.Build.Configuration)/MSBuildDeviceIntegration/${{ parameters.target_framework }}/MSBuildDeviceIntegration.runsettings" | ||
| testResultsFile: TestResult-MSBuildDeviceIntegration-${{ parameters.job_name }}-$(System.JobPositionInPhase)-$(XA.Build.Configuration).xml | ||
|
|
||
| - task: MSBuild@1 | ||
| displayName: shut down emulator | ||
| inputs: | ||
| solution: tests/Mono.Android-Tests/Mono.Android-Tests.csproj | ||
| configuration: $(XA.Build.Configuration) | ||
| msbuildArguments: >- | ||
| /t:AcquireAndroidTarget,ReleaseAndroidTarget | ||
| /bl:$(System.DefaultWorkingDirectory)/bin/Test$(XA.Build.Configuration)/shutdown-emulator.binlog | ||
| condition: always() | ||
|
|
||
| - template: upload-results.yaml | ||
| parameters: | ||
| artifactName: Test Results - MSBuild With Emulator - macOS-$(System.JobPositionInPhase) | ||
|
|
||
| - template: fail-on-issue.yaml | ||
|
|
||
| - job: wear_tests | ||
| displayName: macOS > Tests > WearOS | ||
| timeoutInMinutes: 180 | ||
| cancelTimeoutInMinutes: 2 | ||
| strategy: | ||
| matrix: | ||
| Android30-x86: | ||
| avdApiLevel: 30 | ||
| avdAbi: x86 | ||
| avdType: android-wear | ||
| deviceName: wear_square | ||
| pool: | ||
| vmImage: $(HostedMacImage) | ||
| workspace: | ||
| clean: all | ||
| steps: | ||
| - template: setup-test-environment.yaml | ||
| parameters: | ||
| configuration: $(XA.Build.Configuration) | ||
|
|
||
| - template: run-xaprepare.yaml | ||
| parameters: | ||
| displayName: install required brew tools and prepare java.interop | ||
| arguments: --s=Required --auto-provision=yes --auto-provision-uses-sudo=yes | ||
|
|
||
| - template: run-xaprepare.yaml | ||
| parameters: | ||
| displayName: install emulator | ||
| arguments: --s=EmulatorTestDependencies | ||
|
|
||
| - script: echo "##vso[task.setvariable variable=Java8SdkDirectory]$JAVA_HOME_8_X64" | ||
| displayName: set Java8SdkDirectory | ||
|
|
||
| - task: DownloadPipelineArtifact@2 | ||
| inputs: | ||
| artifactName: $(TestAssembliesArtifactName) | ||
| downloadPath: $(System.DefaultWorkingDirectory)/bin/Test$(XA.Build.Configuration) | ||
|
|
||
| - task: MSBuild@1 | ||
| displayName: install and launch emulator | ||
| inputs: | ||
| solution: tests/Mono.Android-Tests/Mono.Android-Tests.csproj | ||
| configuration: $(XA.Build.Configuration) | ||
| msbuildArguments: /t:InstallAvdImage;AcquireAndroidTarget /p:TestDeviceName=$(deviceName) /p:TestAvdApiLevel=$(avdApiLevel) /p:TestAvdAbi=$(avdAbi) /p:TestAvdType=$(avdType) /bl:$(System.DefaultWorkingDirectory)/bin/Test$(XA.Build.Configuration)/install-emulator-$(avdApiLevel).binlog | ||
|
|
||
| - template: run-nunit-tests.yaml | ||
| parameters: | ||
| testRunTitle: WearOS On Device - macOS | ||
| testAssembly: $(System.DefaultWorkingDirectory)/bin/Test$(XA.Build.Configuration)/MSBuildDeviceIntegration/$(DotNetStableTargetFramework)/MSBuildDeviceIntegration.dll | ||
| dotNetTestExtraArgs: --filter "TestCategory = WearOS" | ||
| testResultsFile: TestResult-WearOS--$(XA.Build.Configuration).xml | ||
|
|
||
| - task: MSBuild@1 | ||
| displayName: shut down emulator | ||
| inputs: | ||
| solution: tests/Mono.Android-Tests/Mono.Android-Tests.csproj | ||
| configuration: $(XA.Build.Configuration) | ||
| msbuildArguments: /t:AcquireAndroidTarget,ReleaseAndroidTarget /p:TestDeviceName=$(deviceName) /p:TestAvdApiLevel=$(avdApiLevel) /p:TestAvdAbi=$(avdAbi) /p:TestAvdType=$(avdType) /bl:$(System.DefaultWorkingDirectory)/bin/Test$(XA.Build.Configuration)/shutdown-emulator.binlog | ||
| condition: always() | ||
|
|
||
| - template: upload-results.yaml | ||
| parameters: | ||
| configuration: $(XA.Build.Configuration) | ||
| artifactName: Test Results - Emulator $(avdApiLevel)-$(avdAbi)-$(avdType) - macOS | ||
|
|
||
| - template: fail-on-issue.yaml | ||
|
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.