NUnit runner: Handle specific non-zero exit codes

[jrnail23]

Right now, any time the NUnit console tool returns a non-zero exit code, the Cake NUnit runner throws a CakeException with the less-than-helpful error message, NUnit: Process returned an error.

I think this behavior is too broad, as it treats failed tests the same as an unexpected problem.
I'd like the flexibility to decide whether to blow up my build based on test failures, errors, etc.

NUnit does have meaningful exit codes (discussed here), so we should throw more specific exceptions as appropriate.

[jrnail23]

Also, here's the relevant source code for the NUnit runner (v2.6.4):

https://github.com/nunit/nunitv2/blob/master/src/ConsoleRunner/nunit-console/ConsoleUi.cs

According to the source (Line 183), it appears that a return code > 0 is the sum of the count of errors, failures, and non-runnable tests:
returnCode = summary.Errors + summary.Failures + summary.NotRunnable;

[patriksvensson]

I didn't know this. Excellent that you added an issue about this 👍

It might make sense to add a virtual member to Tool<ToolSettings> so that each tool can opt in to handle it's own errors.

[rprouse]

It is the same for NUnit 3. A positive return code is the number of failed tests, a negative return code is an error.

[molehillrocker]

Has anybody made any progress yet regarding this issue? We are currently facing this problem on our build server and since I have only access to the web interface, I have no clue what actually goes wrong.

I tried to hack on this a little bit this evening, but in the end I couldn't test it anymore. 😲

My first idea was to redirect the overall output of NUnit to the console, since the only information we get at the moment is the above mentioned error (no other NUnit output 😒). This seems to be due to the NUnitRunner calling the inherited Tool.Run() method internally with two parameters only, which results in the ProcessSettings object to be null until it is instantiated as fallback in the Tool class, resulting in ProcessSettings.RedirectStandardOutput to bet set to false.

Be aware that I just traced that visually, since I have my developer machine not at hands right now. 😄

[devlead]

@molehillrocker What CI server you using? Most output the buildlog.

You could redirect all output via the bootstrapper to a file like I do here
https://github.com/devlead/cake/blob/f/kudu/kudu.ps1

Also if you set verbosity of cake to diagnostic you will see how it's launched.

[molehillrocker]

We're using TFS 2015. Until friday I'm out of office, so I can't give you the real log output. But IIRC, it was something like:

Starting task Unit-Test...
*** custom log message right before starting NUnit that was inserted for testing :) ***
NUnit: Process returned an error

Previous tasks such as the build task were logged with lots of details, which is why I assumed that setting ProcessSettings.RedirectStandardOutput may hopefully provide some more information, at least the default NUnit output (version, etc.) to see if probably already the invocation fails.

[devlead]

Actually by setting RedirectStandardOutput it won't output anything as it's redirected to an buffer that needs to be enumerated. By default all tools should be outputted to the console.
Do you get any errors if you run it locally?
Do you have a #tool directive for nunit or do you restore via package.config?

[molehillrocker]

When I run it locally, it works fine. It is included via package.config.

[molehillrocker]

Back in the office, this is the log output from our TFS 2015:

<snip>
...
435 2016-01-19T14:58:25.3240665Z Finished executing task: Build-Server
436 2016-01-19T14:58:25.3240665Z ========================================
437 2016-01-19T14:58:25.3240665Z Unit-Test-Server
438 2016-01-19T14:58:25.3240665Z ========================================
439 2016-01-19T14:58:25.3240665Z Executing task: Unit-Test-Server
440 2016-01-19T14:58:25.5896943Z ##[error]An error occured when executing task 'Unit-Test-Server'.
441 2016-01-19T14:58:25.5896943Z ##[error]Error: NUnit: Process returned an error.
442 2016-01-19T14:58:25.5896943Z ----------------------------------------
443 2016-01-19T14:58:25.5896943Z Teardown
444 2016-01-19T14:58:25.5896943Z ----------------------------------------
...
</snip>

I'm trying to dig deeper into this now.

[devlead]

Could you turn on diagnostic logging with Cake (by passing the -verbosity=Diagnostic parameter or -Verbosity Diagnostic if you're using the powershell bootstrapper)? Then you will also get how it's launched (something like Executing: "C:....). Could provide more clues on what is going on.

Also which version of Cake and NUnit are you running?

Launching Cake with Cake.exe -Version will display the version (0.8.0 is the latest) and launching NUnit with .\nunit3-console.exe --version will display it's version (believe latest is Nuget 3.0.1 and assembly version 3.0.5. 813).

Also if you check the NUnit3 aliases you can pass an NUnit3Settings, on this you can specify an ErrorOutputFile and Verbose to get more info about what's going on.

[molehillrocker]

I installed a build of Cake from the develop branch (where the error code is printed) and found out that the process returns error code -2146232576 from the NUnit process. This seems to have something to do with a .NET framework version mismatch on the build server (see here). We use NUnit 2.6.4 at the moment.

In addition, on our build server, only .NET framework 4.6 seems to be installed (if I try to build with any other option than MSBuildToolVersion.NET46 resp. MSBuildToolVersion.Default, even the build fails). Also, according to the docs, NUnit 2.6.4 does not support higher framework versions than 4.0.

So I guess I should install either an older framework version on the build server to support NUnit 2.6.4 or migrate to NUnit 3.0.1.

[molehillrocker]

After migrating our Cake build script to use NUnit3 instead of NUnit and replacing <package id="NUnit.Runners" version="2.6.4" /> by <package id="NUnit.Console" version="3.0.1" /> in tools\packages.config, everything works fine now. Thanks for your help, @devlead! passing a Snus over the wire 😄

[devlead]

@molehillrocker superb 👍

[ChrisMaddock]

I'd be interested in taking a look at this. We wrote some new scripts for the NUnit code samples recently, and it was this limitation which meant we decided to use NUnit via the console, instead of the tool.

It might make sense to add a virtual member to Tool so that each tool can opt in to handle it's own errors.

@patriksvensson - I'm not yet familiar with Cake's internal tool architecture, and it sounds like you have some ideas - mind posting a rough overview of what you'd expect this to look like?

[gep13]

@jrnail23 this should be in the next release of Cake, big thanks to @ChrisMaddock