Run multiple xUnit.net test libraries as one test suite

[ploeh]

When using the 'raw' xUnit.net 2 console runner, it's possible to run multiple test libraries as a single test suite, simply by supplying all the library files on the command line. Here's an example:

$ packages/xunit.runner.console.2.0.0/tools/xunit.console.exe CaseConversion.FSharp/bin/Debug/Ploeh.Samples.CaseConversion.FSharp.dll CaseConversion.CSharp/bin/Debug/Ploeh.Samples.CaseConversion.CSharp.dll

There are multiple advantages of doing this. First of all, the output includes a test execution summary, with totals and grand totals. Here's an example:

=== TEST EXECUTION SUMMARY ===
   Ploeh.Samples.CaseConversion.CSharp  Total:  7, Errors: 0, Failed: 0, Skipped: 0, Time: 0,297s
   Ploeh.Samples.CaseConversion.FSharp  Total:  7, Errors: 0, Failed: 0, Skipped: 0, Time: 0,683s
                                               --          -          -           -        ------
                                  GRAND TOTAL: 14          0          0           0        0,981s (2,081s)

Another advantage is that if you ask the test runner to parallelise the suite, it runs all discovered tests in parallel, as can be seen in this example:

Discovering: Ploeh.Samples.CaseConversion.FSharp
Discovering: Ploeh.Samples.CaseConversion.CSharp
Discovered:  Ploeh.Samples.CaseConversion.CSharp
Discovered:  Ploeh.Samples.CaseConversion.FSharp
Starting:    Ploeh.Samples.CaseConversion.FSharp
Starting:    Ploeh.Samples.CaseConversion.CSharp
Finished:    Ploeh.Samples.CaseConversion.CSharp
Finished:    Ploeh.Samples.CaseConversion.FSharp

Notice how discovery completes for both test libraries before execution starts.

This isn't how FAKE currently works. As far as I can tell from the code, FAKE runs each test library independently.

This means that I never get a single summary, and I can't parallelise all tests.

Is there a way to configure it so that it runs all libraries as a single test suite?

If not, could there be?

I'll be happy to attempt a PR, but before I do that, I wanted to ask if I'd missed something, and if not, if this feature is wanted.

[forki]

If xunit2 allows to natively run tests in parallel then we should use that feature. I assume the xunit2 helper was basically copied from the xunit helper (which is still there for backwards compatibility) and we didn't know about that feature.

Tl;dr pull request would be very welcome

[ploeh]

Would you like to change the behaviour of the xunit2 helper? This would be the simplest change, I think, but could be argued to be a breaking change.

Or would it be better to add a flag to ParallelOption? This wouldn't be a breaking change, but would be harder to discover, and probably also a more complicated implementation.

[forki]

In this case I would argue a breaking change is OK since from what I can understand it's the default in xunit2. Adding a RunSynchronously flag to restore the current behavior would be nice.

[ploeh]

OK, sounds good. I'll take a look at it this weekend, I hope.

[neoeinstein]

This was also something that I was very much looking to do as well. I would like to see the xunit tasks converge on the parameter style of the nunit and mspec tasks, e.g. being able to specify what the output file names should be rather than just providing a true/false value. I've already created a shadow task that I use in some of my local builds instead of the built-in xunit task. I can prepare that as a pull request to see if it matches your expectations.

Caveat being that, again, it's a breaking change.

[ploeh]

@neoeinstein Good points. Perhaps you should go first, then 😄

As I look through XUnit2Helper.buildXUnit2Args, I can see that the output file names are currently implicitly derived from a single assembly name. This isn't going to work with my suggested change, because the xUnit.net 2 console runner expects multiple test assemblies, but only a single output file.

Thus, it may make sense to make your suggested change first, because that would effect that necessary decoupling first.

[moodmosaic]

This is extremely helpful! Thank you! 😄