CollectionAttribute seems to be not respected

[weltkante]

WinForms has some tests that cannot be executed in parallel, the corresponding class has an attribute [Collection("Sequential")] which is supposed to put all test methods of that class in a collection and execute them in sequence (the name is probably irrelevant, as far as I understand its just grouping by collection name, but I didn't research this in detail).

/cc @hughbe @RussKie @AArnott

[AArnott]

Interesting. I didn't think that that attribute was the responsibility of this Library. I thought Xunit itself would take care of it. I'll look into it.

[weltkante]

It might have been broken by one of the overrides WinFormsFact/Theory provide

[AArnott]

I did some quick research in how CollectionAttribute works and don't see any correlation to the test discovery system.

So I wrote a quick test:

[Collection("A")]
public class ConcurrentClass1Tests
{
    [WinFormsFact]
    public void ConcurrentA()
    {
        Thread.Sleep(10000);
    }
}

[Collection("A")]
public class ConcurrentClass2Tests
{
    [WinFormsFact]
    public void ConcurrentB()
    {
        Thread.Sleep(10000);
    }
}

Without the Collection attribute on the test classes, Test Explorer in VS configured to run tests in parallel will run these two tests in parallel. But with the Collection attribute, only one test runs at a time. So I can't repro the problem you're describing.

[weltkante]

Thanks, I'll have another look and see if I can build a repro. One difference in scenario to your snippet: the test runs I was debugging in WinForms were on the same class.

[AArnott]

Thanks. I look forward to your repro.

One difference in scenario to your snippet: the test runs I was debugging in WinForms were on the same class.

That's not how CollectionAttribute works, at least not base don my reading of the docs.
Per my reading, tests within a class never run in parallel anyway (given default behavior anyway). The CollectionAttribute allows you to extend that non-concurrency across multiple classes, as in my example.

[AArnott]

FWIW I just tried moving the two methods into one class and removing the Collection attribute. The VS Test Explorer still ran them one at a time, which is consistent with the xunit doc I link to.

[weltkante]

That's not how CollectionAttribute works, at least not base don my reading of the docs.
Per my reading, tests within a class never run in parallel anyway (given default behavior anyway).

Thanks for looking into it and clearing up how its supposed to work, looks what I was seeing was a red herring.

I was running tests from VS ("Debug test" command on a single test class) and when I did break into the debugger due to failed tests I was seeing all other methods having a thread as well. I could reproduce that but the test tasks already are on RanToCompletion so I assume you report success back to Xunit before you terminate the threads.

Since the test failures looked like they were comming from running tests in parallel I probably was jumping to the wrong conclusions, sorry. Can't reproduce the test failures anymore so I'm not sure what was causing them, but apparently not what I thought it was.