[ci] Split MSBuildDeviceIntegration tests across multiple build agents. #6360
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
jpobst
force-pushed
the
msbuild-integration-nodes
branch
from
051ce31
to
2c81831
Compare
pjcollins
approved these changes
Oct 4, 2021
jpobst
added a commit
that referenced
this pull request
Feb 27, 2023
[Previously](#6360), we split our `MSBuildIntegration` unit tests to run across multiple CI test agents. While a huge improvement, there are a few downsides to the approach we went with at the time: - The number of agents is hardcoded by copy/pasting steps in the CI YAML. - The tests must be manually load-balanced across agents with `[Category ("Node-X")]` in code which is cumbersome and hard to keep updated. As we are at the point where our tests have expanded well past the 1 hour target we are faced with needing to increase our parallelization. First, we can remove the YAML duplication by using AZDO's built-in `parallel` strategy, allowing us to control the number of agents to use with a single number: ```yaml - job: strategy: parallel: 4 ``` This leaves us the issue of automatically splitting our tests among an unknown number of test agents. A clever solution is [provided in the AZDO docs](https://learn.microsoft.com/en-us/azure/devops/pipelines/test/parallel-testing-any-test-runner?view=azure-devops): - Use `dotnet test --list-tests` to find all the tests - Use a script to calculate which tests the agent should run, based on `$(System.JobPositionInPhase)` and `$(System.TotalJobsInPhase)` - Pass those test names into `dotnet test` as a filter Unfortunately there are issues with the provided approach: - `dotnet test --list-tests` is not compatible with `--filter` so you have to run *all* tests in the test assembly which is not desirable for us. (dotnet/sdk#8643) - Passing test names (including test parameters) on the command line hits limitations with escaping certain characters and argument limits. So the approach is good, but we have to do all the work ourselves. Enter [dotnet-test-slicer](https://github.com/jpobst/dotnet-test-slicer). This dotnet global tool: - Uses the `NUnit.Engine` NuGet package to find all tests in a test assembly using the desired filter query. - Slices the test list for the current test agent. - Outputs the desired tests into an NUnit-specific `.runsettings` file that can be passed to `dotnet test --settings foo.runsettings`, bypassing command line arguments. Voila! Now we can automatically scale our test agents up or down as needed by changing a single variable in our YAML file. Limitation: The tests are sliced in a round robin fashion. This can produce uneven results if test durations are uneven. A future optimization would be to store approximate test durations so tests can be load balanced more intelligently.
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Much like the
MSBuildtests used to, theMSBuild with Emulatortests can take multiple hours.Like we did for
MSBuildtests, the solution is to parallelize them across multiple build agents. This PR splits each test suite across 3 nodes using the same[Category ("Node-X")]hack, bringing the total wall-time of running each suite under an hour.