Take TestCaseFilter from runsettings #2356
Conversation
|
@nohwnd : Just confirming that this works for discovery as well as execution? I'm hoping this just works in Test Explorer to show only the tests specified in this filter after this change... |
|
@AbhitejJohn it doesn't, I have another fix, now I am figuring out why xunit ignores the filter. |
|
We would need update the documentation to include this feature, I'm sure a good number of people would like to make use of it. |
nohwnd
changed the title
|
Marking as WIP till I figure out the remaining requirements, was to fast to post this as a PR. |
|
@vritant24 it now works during discovery (but only for MSTest, unload the xunit project). The simplest way to see this is, to use the vstest console and with list tests: "C:\Program Files (x86)\Microsoft Visual Studio\2019\IntPreview\Common7\IDE\Extensions\TestPlatform\vstest.console.exe" /Platform:x86 /ListTests /TestCaseFilter:TestCategory=include C:\Users\jajares\source\repos\UnitTestProject29\XUnitTestProject1\bin\Debug\netcoreapp3.1\XUnitTestProject1.dll C:\Users\jajares\source\repos\UnitTestProject29\UnitTestProject29\bin\Debug\netcoreapp3.1\UnitTestProject29.dll I can also see in VS that it reports that it discovered 1 test. Running the tests also runs only the one that passes filter. Selecting just the excluded test and running it will write errors into test output, and incorrectly report it as passed in the status bar and output but in TE it stays as the blue outdated icon.
|
|
/azp run |
|
Azure Pipelines successfully started running 1 pipeline(s). |
|
@ViktorHofer Currently we are figuring out how to filter also while the source based test discovery runs. It looks like we would have to adopt all the filtering logic from all the supported adapters (so xunit, mstest, nunit). /cc @vritant24 |
|
@nohwnd thanks for the update. I just thought about this further and I believe that we should not ignore a passed in filter (if set by the CLI) when a default filter is provided by the runsettings file. That would effectively mean that consumers won't be able to use the filter argument in dotnet test scenarios when a default one is set by the repository. Can we instead concatenate the filters together? |
|
@ViktorHofer what you describe is not how it works. The filter that you provide on command line will win. This is a decision that is (or should be) used in all cases, so the user gets the flexibility of providing a custom value, while falling back to the default if not providing any. I am not 100% sure now what will happen if you do this: But I would expect that the options are applied like this: |
Understood. So you don't think concatenating the filters makes sense? |
|
@ViktorHofer in some cases yes, but it might not be what the user wants and it would be different than how the other settings behave, so I would rather keep it as is. |
|
This might be problematic. Our plan in dotnet/runtime was to use this feature to disable tests with the trait "failing" by default in the .runsettings file. What you are saying would mean that the user would always need to append Trait!=failing in the filter as the default one is overridden. |
|
That is true, we will have a meeting on Tuesday about this, so I will discuss it there. |
|
I don't think there is any harm in concatenating filters specified in different sources (CLI vs runsettings). Thanks for following up on that. |
src/vstest.console/Processors/TestCaseFilterArgumentProcessor.cs
Outdated
Show resolved
Hide resolved
In VS with Source Based discovery disabled.
<!-- file UnitTestProject29\UnitTestProject29.csproj -->
<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<TargetFramework>netcoreapp3.1</TargetFramework>
<IsPackable>false</IsPackable>
<RunSettingsFilePath>$(MSBuildProjectDirectory)\..\runsettings.runsettings</RunSettingsFilePath>
</PropertyGroup>
<ItemGroup>
<PackageReference Include="Microsoft.NET.Test.Sdk" Version="16.5.0" />
<PackageReference Include="MSTest.TestAdapter" Version="2.1.0" />
<PackageReference Include="MSTest.TestFramework" Version="2.1.0" />
<PackageReference Include="coverlet.collector" Version="1.2.0" />
</ItemGroup>
</Project>
# file runsettings.runsettings
<RunSettings>
<RunConfiguration>
<TestCaseFilter>TestCategory=include</TestCaseFilter>
</RunConfiguration>
</RunSettings>// file UnitTestProject29\UnitTest1.cs
using Microsoft.VisualStudio.TestTools.UnitTesting;
namespace UnitTestProject29
{
[TestClass]
public class UnitTest1
{
[TestMethod]
[TestCategory("include")]
public void IncludedMsTest()
{
}
[TestMethod]
[TestCategory("exclude")]
public void ExcludedMsTest()
{
}
[TestMethod]
[TestCategory("include")]
public void IncludedMsTest2()
{
}
[TestMethod]
[TestCategory("include")]
public void IncludedMsTest3()
{
}
[TestMethod]
[TestCategory("exclude")]
public void ExcludedMsTest2()
{
}
[TestMethod]
[TestCategory("exclude")]
public void ExcludedMsTest3()
{
}
}
}We agreed that this option should remain undocumented, so we are able to redesign this if we start fixing the source based discovery, for example moving it to adapters. |
It fails because of pin to newer dotnet, it's nice to have will fix later.
test/vstest.console.UnitTests/Processors/RunSettingsArgumentProcessorTests.cs
Show resolved
Hide resolved
nohwnd
changed the title
|
Will this work for c++ tests? [i.e GoogleTest, CppUnitTests (MS) ] |
It should if the adapter implements filtering during discovery (if not there already). Also notice that it comes with a limitation that it does not work for source based discovery. |
and what's that? |
|
In Visual Studio you get tests discovered even before you build your project, you can enable (default), or disable it by the checkbox shown above. This solution only works when it is disabled, which is not the default setting. |
|
as far as I know that's only an option for C# and .NET |
|
@L4ZZA then probably give it a try when it's released and report back (VS 16.6 preview 3 should have it if all goes well), I don't know much about C++ tests yet so I can't reliably tell you, sorry 🙂 |
|
Should this still work for .NET 5? I can't get it working. |
|
Jep that should work. Make sure to disable real time test discovery in VS:
|
|
I rely on this feature (or something similar) in the Pruner CLI (https://github.com/pruner/cli). I would be sad if the feature was removed. I think it should be possible for commandline tools or VS Code plugins to still be able to customize the filter this way through. It's also a perfect fix for this issue: #2625 |
Description
Take default filter from RunSettings.
Related issue
Fixes #2273