Attempt to make functional tests more reliable #5874
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,20 +1,44 @@ | ||
| <RunSettings> | ||
| <!-- https://learn.microsoft.com/visualstudio/test/configure-unit-tests-by-using-a-dot-runsettings-file#runconfiguration-element --> | ||
| <RunConfiguration> | ||
| <!-- | ||
| This setting controls the level of parallelism on process-level. Use 0 to enable the maximum process-level parallelism. | ||
|
|
||
| This setting determines the maximum number of test DLLs, or other test containers that can run in parallel. Each DLL runs in its own testhost process, and is isolated on the process level from the tests in other test DLLs. This setting doesn't force tests in each test DLL to run in parallel. Controlling the parallel execution within a DLL (on the thread-level) is up to the test framework such as MSTest, XUnit or NUnit. | ||
|
|
||
| The default value is 1, meaning that only one testhost runs at the same time. A special value 0 allows as many testhosts as you have logical processors (for example, 6, for a computer with 6 physical cores without multi-threading, or 12, for a computer with six physical cores with multi-threading). | ||
|
|
||
| The number of distinct DLLs in the run determines the actual number of testhosts started. | ||
| --> | ||
| <MaxCpuCount>4</MaxCpuCount> | ||
|
|
||
| <!-- Specify a Boolean value, which defines the exit code when no tests are discovered. If the value is true and no tests are discovered, a nonzero exit code is returned. Otherwise, zero is returned. --> | ||
| <TreatNoTestsAsError>true</TreatNoTestsAsError> | ||
|
|
||
| <EnvironmentVariables> | ||
| <!-- Changes the default timeout in VSTest for shutdown to 1000ms instead of 100ms which can cause intermittent failures in CI. --> | ||
| <VSTEST_TESTHOST_SHUTDOWN_TIMEOUT>1000</VSTEST_TESTHOST_SHUTDOWN_TIMEOUT> | ||
| </EnvironmentVariables> | ||
| </RunConfiguration> | ||
|
|
||
| <!-- https://xunit.net/docs/runsettings --> | ||
| <xUnit> | ||
| <!-- Set this value to true to include diagnostic information during test discovery and execution. Each runner has a unique method of presenting diagnostic messages. --> | ||
| <DiagnosticMessages>true</DiagnosticMessages> | ||
|
|
||
| <!-- Set this value to enable long-running (hung) test detection. When the runner is idle waiting for tests to finished, it will report that fact once the timeout has passed. Use a value of 0 to disable the feature, or a positive integer value to enable the feature (time in seconds). --> | ||
| <LongRunningTestSeconds>120</LongRunningTestSeconds> | ||
|
|
||
| <!-- Set this to override the maximum number of threads to be used when parallelizing tests within this assembly. Use a value of 0 or "default" to indicate that you would like the default behavior; use a value of -1 or "unlimited" to indicate that you do not wish to limit the number of threads used for parallelization. --> | ||
| <MaxParallelThreads>4</MaxParallelThreads> | ||
|
|
||
| <!-- Set this to true if this assembly is willing to participate in parallelization with other assemblies. Test runners can use this information to automatically enable parallelization across assemblies if all the assemblies agree to it. --> | ||
| <ParallelizeAssembly>false</ParallelizeAssembly> | ||
|
|
||
| <!-- Set this to true if the assembly is willing to run tests inside this assembly in parallel against each other. Tests in the same test collection will be run sequentially against each other, but tests in different test collections will be run in parallel against each other. Set this to false to disable all parallelization within this test assembly. --> | ||
| <ParallelizeTestCollections>false</ParallelizeTestCollections> | ||
|
There was a problem hiding this comment. I compared the CI run for this PR to a different CI run, because I would have expected that setting ParallelizeTestCollections and ParallelizeAssembly both to false would make the tests run very slowly. But somehow this PR's CI run was margionally faster then the dev branch run. It's totally not intuitive to me why, but I didn't see this mentioned in the PR description or anywhere else, so I wanted to point out that apparently the CI is not going to take a long longer with this PR merged. There was a problem hiding this comment. I struggle to understand it as well. Its still running tests assemblies in parallel with each other, but I guess not running tests themselves in parallel? A machine with only 2 cores would only be able to run two tests at the same time anyway so I think the adjustments don't affect the maximum parallelization we could already achieve on CI agents. It really just lowers the amount of parallelization that's queued up waiting for CPU. |
||
|
|
||
| <!-- Set this to true to use shadow copying when running tests in separate App Domains; set to false to run tests without shadow copying. When running tests without App Domains, this value is ignored. --> | ||
| <ShadowCopy>false</ShadowCopy> | ||
| </xUnit> | ||
| </RunSettings> | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -78,7 +78,7 @@ jobs: | |
| condition: succeeded() | ||
| inputs: | ||
| command: test | ||
|
There was a problem hiding this comment.
Have you talked to the team who owns There was a problem hiding this comment. I see a few issues in the vstest repo: Nothing specific to the problem I was seeing. Supposedly the blame stuff also doesn't supply enough command-line arguments to capture other exceptions than Although now there's an env var we can set to include more arguments to proc dump. The main point in all of this is it seems like adding |
||
| arguments: --no-restore --restore:false --no-build --framework $(TestTargetFramework) --binarylogger:"$(BinlogDirectory)02-RunTests.binlog" | ||
| testRunTitle: ${{ parameters.osName }} $(TestType) Tests ($(TestTargetFramework)) | ||
|
|
||
| - task: PublishPipelineArtifact@1 | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think this setting nullifies the
MaxcpuCount (4)property because as per the comment it looks likeMaxcpuCount specifies the number of distinct DLLs in the run determines the actual number of testhosts started.. Given that the multiple assemblies are not running parallel then my understanding is that only one test host process is started. I guess we should change to true so that 4 DLLs can run in parallel.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Since
MaxCpuCountis set to4(here), there will be 4vstest.console.exeprocesses spawned. I think it then launches each test assembly in those processes in parallel. Then withMaxParallelThreadsset to4, each test assembly runs with 4 threads (if there are enough CPUs).This is really confusing since there are multiple test runners. You can run tests under
xunit.runner.exe,vstest.console.exe, etc. I think theParallelizeAssemblytells xunit specifically that it should run tests from multiple assemblies at the same time when running under certain test runners. But since we're running the tests undervstest.console.exe, it seems to me that we should not be using xunit's parallelization. I spent a lot of time read https://xunit.net/docs/runsettings but only experimenting with these numbers did I find a good balance between speed and reliability. Do you know more about how all of this works?There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It looks like a similar issue has been discussed here. I think we are good because it appears that xUnit parallelization is ignored by the
dotnet testcommand. One interesting thing I learned from this link is that executingdotnet testwithout specifying assemblies runs test assemblies in parallel, whereasdotnet test assembly1.dll assembly2.dlldoes not run test assemblies in parallel. I am not sure if this is accurate in .NET 8, but at least that was the case earlier because the linked comment explained the different behaviors with an example.