Tests are being silently skipped when using the release CLR

[Anipik]

I tried adding tests to Pathtests.cs and pathtests.netcoreapp.cs. But these tests are not getting run. Even if you change some existing test, this change is also not getting reflected. Although it gives a compile error if there is some typo or missing semicolon

eg https://github.com/Anipik/corefx/tree/temp3
if you try running system.runtime.extensions.test for this branch, it shows that that all tests ran successfully which should not be the case due to this test.
https://github.com/Anipik/corefx/commit/070d55153a5e74dd8cc3eeac2968ab7ba694051b#diff-19631815f66f1f0ad0492da989a7463cR16

if we remove the pathtests.netcoreapp.cs from proj file, everything starts to work fine.
The possible reason behind such a behavior is that there is some limit on number of tests that can be included in one class, and we are exceeding this limit due to which these new changes are not affecting the class behavior.

Changing the class name of pathtest.netcoreapp.cs also resolves the problem.

cc @JeremyKuhne @maryamariyan @danmosemsft

[stephentoub]

due to the limit being exceeded on number of tests

Can you elaborate? What limit?

[Anipik]

@stephentoub I changed the description. I hope it clarifies

[JeremyKuhne]

@stephentoub there appears to be a bug in Xunit that silently skips classes if they have too many test methods. Taking a few methods out of PathTests causes them to run again. Just adding/removing simple [Fact] methods cause the issue to surface/go away. It is possible, however, that something else is contributing to this other than the number of tests- @maryamariyan is trying to reproduce the problem with a simpler test class.

[jkotas]

bug in Xunit that silently skips classes if they have too many test methods

Based on these symptoms, I would put more money on that this is a bug in the runtime/framework...

[jkotas]

This repros with retail build of CoreCLR only. Switching to checked CoreCLR makes the problem go away:

CoreCLR amd64ret: System.Runtime.Extensions.Tests  Total: 3417, Errors: 0, Failed: 0, Skipped: 15, Time: 30.000s
CoreCLR amd64chk: System.Runtime.Extensions.Tests  Total: 3440, Errors: 0, Failed: 11, Skipped: 13, Time: 181.550s