NUnitTestAdapter 2.1.1 not working with Visual Studio 2017 15.8.0

[gtrubach]

Hello!

VS 2017 recently got an update to 15.8.0 version. After I updated my VS to the latest version and updated test adapter to the version 2.1.1 (NuGet) VS is able to discover tests but cannot run it:

[8/17/2018 9:27:00 AM Informational] ------ Run test started ------
[8/17/2018 9:27:00 AM Informational] ========== Run test finished: 0 run (0:00:00.3633409) ==========

To check this problem I downloaded this repo and opened NUnitTestDemo solution. I installed the latest adapter 2.1.1 through NuGet and see that tests are not able to run.

Also I found out that if I copy adapter libraries (from NUnitTestAdapter.2.1.1\tools folder) to bin/Debug folder test become able to run.

Can you help with this problem?

[OsirisTerje]

I'll have a look

[OsirisTerje]

Are you using VSIX or nuget adapters?
Can you add the adapter nuget package to all test projects? [Update: No, this doesnt help, see below.]
That should make it work.
Optionally check if turning off the Active solution options below helps:

This can be related to what I discuss here: http://hermit.no/vsix-based-test-adapters-to-be-deprecated-in-visual-studio-for-c-and-visual-basic-to-speed-up-testing/

[Update]
I checked and see that the 2.X adapter doesn't copy its dll to the output, as the 3.X adapter do. We might need to fix that. But turning off the options above should make things work back to normal. The secondary option there is meant to make this work anyway, but it might not work as intended. This the behaviour you see is a breaking change in 15.8, but the second option above should keep it non-breaking......

[OsirisTerje]

I can repro this issue, and it is solved by turning off the Active Solution checks above.
With the Active Solution boxes checked:

With the Active Solution boxes unchecked:

No restart of VS is needed when changing options.

@gtrubach Can you confirm this works for you?

We will make a change to adapter to make it fully compatible, and raise the issue with MS since the second option should have made this work anyway.

[gtrubach]

@OsirisTerje Thanks for your work! I will check if this option works for me on Monday.

[Shyam-Gupta]

@OsirisTerje The second option: 'For compatibility, fallback to global VSIX adapter for MSTest v1 and NUnit test projects that don’t reference a test adapter', comes into effect only for VSIX test adapters. Since in this case, nuget adapter is being used, this option will not help.

@gtrubach I have verified that as a workaround unchecking both the options does work.
However, in my opinion, a better solution would be to move to latest NUnit3TestAdapter which does not have this problem and also offers a bunch of perf improvements.

[gtrubach]

@OsirisTerje Solution with unchecked Active Solution boxes works. Thanks!

@Shyam-Gupta I'm working on project that uses NUnit 2 so I have to use NUnitTestAdapter for them. There is no possibility to move to NUnit 3 now.

[kestasjas]

Unchecked Active Solutions works with NUAdapter-2.0.0 & NUnit-2.6.4.
Not working with NUAdapter-2.1.1 & NUnit-3.10.1

[OsirisTerje]

@kestasjas NUAdapter2 only works with NUnit2. NUnitAdapter3 only works with NUnit3.

[kestasjas]

Clear! Somehow was not able to find NUAdapter-3. ?!

[bgavrilMS]

I have the same problem, but I am using VS 15.9.0 Preview 5 and the workaround does not seem to work. I captured the diagnostic logs
log.txt

The project used is a minimal test with Nunit2:
https://github.com/bgavrilMS/NUnitTestProject1/blob/e804748ca9f7aae96b2b64f603fbc0e0e0adf47f/NUnitTestProject1/UnitTest1.cs

Please let me know if you prefer that I log a new issue.

By the way, to give you an idea of the impact - Microsoft AppCenter requires NUnit 2.6.4 tests, there is no support for NUnit3. So at the moment we are doing some hacky things - we use NUnit3 to develop the tests locally, then switch to NUnit2 for AppCenter to be able to run them.

[OsirisTerje]

@bgavrilMS What kind of appcenter app are you working on? It doesn't make sense that a Microsoft product should use NUnit 2.

We'll anyway need to fix this issue.

[bgavrilMS]

@OsirisTerje - apologies, I meant Xamarin UITests have a hard dependency on NUnit 2, we just run our Xamarin tests in AppCenter.

See https://docs.microsoft.com/en-us/appcenter/test-cloud/uitest/, it clearly states

" Note - Xamarin.UITest requires NUnit 2.6.3 or 2.6.4 to run tests. Xamarin.UITest is not compatible with NUnit 3.x."

[ghost]

Are you using VSIX or nuget adapters?
Can you add the adapter nuget package to all test projects? [Update: No, this doesnt help, see below.]
That should make it work.
Optionally check if turning off the Active solution options below helps:

This can be related to what I discuss here: http://hermit.no/vsix-based-test-adapters-to-be-deprecated-in-visual-studio-for-c-and-visual-basic-to-speed-up-testing/

[Update]
I checked and see that the 2.X adapter doesn't copy its dll to the output, as the 3.X adapter do. We might need to fix that. But turning off the options above should make things work back to normal. The secondary option there is meant to make this work anyway, but it might not work as intended. This the behaviour you see is a breaking change in 15.8, but the second option above should keep it non-breaking......

Regarding your screenshot, I would recommend to keep "Additionally discover tests from built assemblies..." box checked. Indeed I've experienced some missing tests in my run for second platform (Platform.iOS) after rebuilding solution because this option was unchecked.

[OsirisTerje]

Thanks @sdelval ! Good advice!

[joaorosado]

I was changing my nuget packages to use PackageReferences instead of the package.config files and even the workaround suddenly stopped working.
I guess that with the new nuget system it really needs the adapter files on the output directory since It started working as soon as I manually copied them to the folder.

Trying to workaround it but it's not easy due to having a different version of mono.cecil on my project :/

[chari-sampathkumar-axway]

I use VS 2017 pro version 15.9.2 and nunit3testadapter 3.10.0. I can repro the issue and the work around solves the problem. But the settings changes get's saved into .suo file which goes away when we do "git clean". Is there a way out of it that enables us not to touch .suo files?

[rprouse]

The setting mentions that you can add the setting to the runsettings file, although I'm not sure how that is done. Google? 😄

[chari-sampathkumar-axway]

@rprouse not much info on google.

If anyone watching the issue, if they can let us know how to control that setting via runsettings instead of losing it via .suo, that will be helpful.

[joaorosado]

You probably should open a issue on nunit3-vs-adapter repository instead since they have a diferent code base. You should not be having this problem with it as long as your project has both adapter and nunit framework included via nugets. Answering your question, as far my tests gone the settings like you noticed are stored in the .suo. The path for the runsettings file is also stored there ...so also not an option. Back to this topic. As workaround for this problem in my case was to replicate the fix they have on the nunit3-vs-adapter into one of my test projects csproj (with the assemblies changed to match the ones for this adapter version): https://github.com/nunit/nunit3-vs-adapter/blob/master/nuget/net35/NUnit3TestAdapter.props With that and both adapter and framework comming from nugets everything works fine without changing the settings.

…

On Mon, Dec 17, 2018, 22:16 Chari SampathKumar ***@***.*** wrote: @rprouse <https://github.com/rprouse> not much info on google. If anyone watching the issue, if they can let us know how to control that setting via runsettings instead of losing it via .suo, that will be helpful. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#174 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AFETPJfOEVcZyYEc1em74UFMBqM7wCayks5u6BfXgaJpZM4WBCy9> .

[chari-sampathkumar-axway]

@joaorosado thanks for the reply. I do pull both the adapter (3.10.0 - little behind than the latest) and nunit framework (3.10.1) from nuget and the problem still remains. It might be possible that with the latest versions of both the problem may be solved. I will give it a shot and confirm.

[chaami]

Hi,
It seems that the test adapter is going to work even less with the upcoming 16.0 version (16.0.0 Preview 3.0).
Tested configuration: NUnit 2.6.4 with NUnitTestAdapter 2.1.1
From what I can tell:

• The workaround allows the NUnit VS Adapter to be called.
  It seems equivalent to copying the dlls to the test bin directory as the nunit3-vs-adapter fix does.
• The way the test adapter is called must have been changed: I now get a warning indicating that the given testcase filter (FullyQualifiedName=testName1|testName2) does not match any test.
  Launching the tests with the nunit3-console works fine (with the NUnit2FrameworkDriver).

Is there any lead to fix this issue and/or even get rid of the workaround ?
I have no idea on how things work, but I am ready to give a hand if possible...
Kind regards.

[OsirisTerje]

@chaami I'll have a look, awesome if you can help out.

[chari-sampathkumar-axway]

@chaami @OsirisTerje FYI: With VS 2017, if you use PackageReference, everything seem to be working fine without having to modify any visual studio settings. I'm not sure how it will impact with upcoming version though. Worth a try.

[GideonZhao]

Turning off the Active Solution works for me. VS 2019, NUnit 2 Test Adapter 2.1.1

[chrisogonas]

off the Active Solution

How do you turn off the Active Solution ?

[OsirisTerje]

Please update to the newest adapter version, https://www.nuget.org/packages/NUnitTestAdapter/, or https://github.com/nunit/nunit-vs-adapter/releases/tag/V2.2 , see release notes: https://github.com/nunit/docs/wiki/AdapterV2-Release-Notes