Skip to content

Commit

Permalink
Make default test pattern narrower (#16772)
Browse files Browse the repository at this point in the history
* Make default test pattern more narrow

VSTest2 task specifies a very wide dll matching pattern which
includes many unexpected dlls. This includes all TestPlatform dlls,
test framework dlls, and often
dlls from local nuget package cache.

Instead replace it with pattern that matches dlls from bin folder only,
and matches the common test dll naming pattern such as:

Product.Tests.dll
Product.Test.dll
ProductUnitTests.dll
Product.Unit.Tests.dll

But does not match *TestPlatform*.dll or MSTest.TestFramework.dll.

This leads to more correct test runs, and faster test runs.

Related microsoft/vstest#3939

* Update versions

* Update Tasks/VsTestV2/task.loc.json

* Fix stars in loc pattern

---------

Co-authored-by: triptijain2112 <92331194+triptijain2112@users.noreply.github.com>
Co-authored-by: Vinayak <110326599+vinayakmsft@users.noreply.github.com>
  • Loading branch information
3 people authored Jun 27, 2023
1 parent b82b1f7 commit 12b9a6e
Show file tree
Hide file tree
Showing 3 changed files with 33 additions and 7 deletions.
32 changes: 29 additions & 3 deletions Tasks/VsTestV2/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,9 +13,35 @@ Use the following options to select tests and control how the tests are run
* Multiple paths can be specified, one on each line.
* Uses the minimatch patterns. Learn more about minimatch [here](https://aka.ms/minimatchexamples)

For example:
To run tests from any test assembly that has 'test' in the assembly name, `**\*test*.dll`.
To exclude tests in any folder called `obj`, `!**\obj\**`.
Example 1:
Most commonly your test projects follow a naming pattern such as `Product.Tests.dll`, `ProductTests.dll`, `Product.Test.dll`, `Product.UnitTests.dll` or similar. These dlls reside in your `bin` directory. To include all such test dlls use this pattern:

```
**\bin\**\*test.dll
**\bin\**\*tests.dll
```

Example 2:
When it is impossible to determine a naming convention for the tested dlls a wide include pattern can be used (notice the * before .dll). Such pattern can be followed by exclude patterns (starting with `!`) that excludes additional dlls.
This pattern includes all dlls that have `test` in their name, and excludes all dlls from intermediate build `obj` directory:

```
**\*test*.dll
!**\obj\**
```

This pattern is likely to include more dlls than you expect as many other dll names include *test* in their name, such as `MSTest.TestFramework.dll`, `Microsoft.VisualStudio.TestPlatform.ObjectModel.dll` etc. Please review your test log to see which dlls are included, and add appropriate excludes, such as:

```
**\*test*.dll
!**\obj\**
!**\*.resources.dll
!**\*TestAdapter.dll
!**\*Microsoft.*TestPlatform*.dll
!**\*testhost*.dll
!**\testcentric.engine.metadata.dll
```


- **Search Folder:** Use this to specify the folder to search for the test files. Defaults to `$(System.DefaultWorkingDirectory)`

Expand Down
4 changes: 2 additions & 2 deletions Tasks/VsTestV2/task.json
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@
"author": "Microsoft Corporation",
"version": {
"Major": 2,
"Minor": 220,
"Minor": 224,
"Patch": 0
},
"demands": [
Expand Down Expand Up @@ -69,7 +69,7 @@
"name": "testAssemblyVer2",
"type": "multiLine",
"label": "Test files",
"defaultValue": "**\\*test*.dll\n!**\\*TestAdapter.dll\n!**\\obj\\**",
"defaultValue": "**\\bin\\**\\*test.dll\n**\\bin\\**\\*tests.dll",
"required": true,
"helpMarkDown": "Run tests from the specified files.<br>Ordered tests and webtests can be run by specifying the .orderedtest and .webtest files respectively. To run .webtest, Visual Studio 2017 Update 4 or higher is needed. <br><br>The file paths are relative to the search folder. Supports multiple lines of minimatch patterns. [More information](https://aka.ms/minimatchexamples)",
"groupName": "testSelection",
Expand Down
4 changes: 2 additions & 2 deletions Tasks/VsTestV2/task.loc.json
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@
"author": "Microsoft Corporation",
"version": {
"Major": 2,
"Minor": 220,
"Minor": 224,
"Patch": 0
},
"demands": [
Expand Down Expand Up @@ -69,7 +69,7 @@
"name": "testAssemblyVer2",
"type": "multiLine",
"label": "ms-resource:loc.input.label.testAssemblyVer2",
"defaultValue": "**\\*test*.dll\n!**\\*TestAdapter.dll\n!**\\obj\\**",
"defaultValue": "**\\bin\\**\\*test.dll\n**\\bin\\**\\*tests.dll",
"required": true,
"helpMarkDown": "ms-resource:loc.input.help.testAssemblyVer2",
"groupName": "testSelection",
Expand Down

0 comments on commit 12b9a6e

Please sign in to comment.