Automatically detect runsettings specified in a build property

[vritant24]

Description

The Testwindow in Visual Studio detects a runsettings file specified in the build property RunSettingsFilePath for a project (c#, f#, and c++), but vstest currently doesn't mimic this functionality.

Steps to reproduce

1. add the following to the .csproj of a test project

<PropertyGroup>
    <RunSettingsFilePath>[path to a .runsettings file]</RunSettingsFilePath>
</PropertyGroup>

2. run the tests in the project using the cli

Expected behavior

run settings defined in the specified file should be used

Actual behavior

They are currently not applied

[jimmylewis]

How are you running the tests on the CLI? Seems like an important factor in how this ought to be supported.

[singhsarab]

@vritant24 How will this property be available to the test platform? Unlike the IDE, the test platform does not know csproj for the test binary.

/cc: @ManishJayaswal

[ManishJayaswal]

@singhsarab @vritant24 - what is the current mechanism to pass runsettings to command line test runner?

[singhsarab]

@ManishJayaswal It takes /settings argument

[ManishJayaswal]

@singhsarab - how does things work in "dotnet test" command from CLI? Does it not look at the project files? From this issue, it seems like the expectations is that the command line runners should know what is in the project file.

I am trying to understand if there is precedence for command line runners to actually read the project file and if not then what is the correct approach here.

If we do not want to couple test runners with projct files, then it seems like the best approach would be to put such information in runsettings file and then refer to the runsettings file is the project file ( for VS to read which is already implemented now) and pass in the run settings file as a parameter to command line runners ( which apparently is already supported, as per your comment above).

[singhsarab]

@ManishJayaswal The dotnet test verb does take proj file (or sln) as input. We'd have to add some code to make this sure this property gets passed to vstest side. I think for dotnet test scenario, this seems feasible.

My worry is more around dotnet test with dll or vstest.console. vstest,console runner does not read the project file.

User scenario: User has a multiple projects and he/she is able to run the tests in IDE because IDE knows where to find the runsettings for each project based on this property in the proj file.
And now the same user, sets up a CI (or simply runs via vstest.console runner), user will probably need to configure for each project separately as it won't work out of the box.

Instead, if we had said, that auto-detection would just work if a runsettings is placed next to the source, we could probably make it work for most scenarios and would have given users a seamless experience.
(That said, this would still require some design change on the platform side.)

/cc: @SadagopanRajaram

[ManishJayaswal]

tagging @AbhitejJohn to get this thoughts on this as well.

[AbhitejJohn]

runsettings next to the source has its own set of issues apart from performance concerns. What happens if all the output go to one deployment folder? One runsettings would be overwritten by the other if we followed a naming scheme. If we don't have a naming scheme, we'd need to have some form of mapping. What is the general guideline of setting up a CI where there are different categories of tests? Are response files a possible way of creating a mapping?

[jimmylewis]

Would it help if auto-discovery worked similar to app.config? If executing tests from Foo.dll and if Foo.dll.runsettings exists, then use it? Could a build target be added that would take $(RunSettingsFilePath) and copy it to $(OutputPath)$(AssemblyName).runsettings?

[AbhitejJohn]

@jimmylewis : That sounds like a possible way of fixing this. What I was inclining to earlier is either re-use an existing format for providing runsettings in these workflows or create something new like what you have. I wonder if there is a common place to put that build target so it works for existing project files as well. One other thing we'd need to think about with an app.config kind of provision is source batching - currently the Test Explorer raises a single request for sources with similar settings(including runsettings content) to the platform. We'd want to preserve the same by moving this level of batching into the platform perhaps so as to keep perf numbers intact.

[ViktorHofer]

I believe that we should initially just enable the auto-detection feature (a .runsettings file is searched in the test assembly's folder) to unblock folks and afterwards hook into GetCopyToOutputDirectoryItems to add the $(RunSettingsFilePath) to the AllItemsFullPathWithTargetPath item.

Initially when I heard of the feature in VS I thought that vstest is passing that information down to VS Test Explorer and was suprised that this isn't the case.

[singhsarab]

@vritant24
With this #2272 going in, is there something that we still think is pending?

[vritant24]

@singhsarab No. This should be completely addressed by the #2272

[singhsarab]

Thanks for confirmation.
Just note, that the fix was possible only for the dotnet test for solution.
For vstest.console and dotnet test for dll, in both these scenarios the runner does not have the knowledge of the project file hence cannot figure out the runsettings file.

Closing this issue now. Let me know if someone has any concerns

[ViktorHofer]

Closing this issue now. Let me know if someone has any concerns

Closing this issue is fine as it just covers the msbuild property but we should have another tracking issue to enable:

1. the auto-detection feature (a .runsettings file is searched in the test assembly's folder)
2. hook into GetCopyToOutputDirectoryItems to add the $(RunSettingsFilePath) to the AllItemsFullPathWithTargetPath item.

[CodeChief]

The RunSettingsFilePath project setting in the current VS release 16.4.6 only works with the "$(SolutionDir)" variable not "$(ProjectDir)" as "documented" (in the code). Secondly it is NOT "documented" from a user (developer) perspective. The "Tests" output window suggests that we should read more at https://aka.ms/vs-runsettings links to the only runsettings documentation, nothing new) but this has nothing written about auto detection features or project settings.

Please fix support for ALL standard VS variables and remember to update the public documentation so new users know about it. There should also be some plan to expose this setting in the VS GUI after all it is "Visual" Studio. It's a useful feature and we should not have to find it by mistake.

[vritant24]

@CodeChief thanks for your response.

The link to the documentation is https://aka.ms/vs-runsettings (https://docs.microsoft.com/en-us/visualstudio/test/configure-unit-tests-by-using-a-dot-runsettings-file?branch=release-16.4&view=vs-2019#visual-studio-2019-version-164-and-later specifically) and walks through how to set up auto detection with either with a solution wide “.runsettings” file or one that is specified in the build properties of a project. Please let me know if anything there is unclear and can be improved..

Instead of using $(ProjectDir), we recommend using $(MSBuildProjectDirectory) (docs). I'll update the documentation to point users to the recommended list of properties.

There is a task in the backlog to expose runsettings better in the VS UI, but it is currently lower in priority.

[CodeChief]

@CodeChief thanks for your response.

The link to the documentation is https://aka.ms/vs-runsettings (https://docs.microsoft.com/en-us/visualstudio/test/configure-unit-tests-by-using-a-dot-runsettings-file?branch=release-16.4&view=vs-2019#visual-studio-2019-version-164-and-later specifically) and walks through how to set up auto detection with either with a solution wide “.runsettings” file or one that is specified in the build properties of a project. Please let me know if anything there is unclear and can be improved..

Instead of using $(ProjectDir), we recommend using $(MSBuildProjectDirectory) (docs). I'll update the documentation to point users to the recommended list of properties.

There is a task in the backlog to expose runsettings better in the VS UI, but it is currently lower in priority.

Thanks for the quick response. I can confirm it works now with MSBuild properties and the documentation is visible. Great!

Just a tip, if it does appear in the VS GUI I think users would expect type-in any "normal" VS variable, or if the direction is away from VS towards more cross-platform friendly (e.g. VS Code and solution-less/folder) build environments, then we should be intuitively instructed via information/warning messages during GUI builds. I guess that is the general direction, which if true then a project upgrade tool or link to docs with mapping/manual upgrade instructions would also be necessary (taking careful note of the difference in trailing backslash consistency which could break many builds).

This is all good stuff, now the "new developer/workstation opens solution which works on another PC and claims he can't run the unit tests" problem has been solved. :-)