Inconclusive test cases in NUnit theories yield test step failures

[ghost]

Bug description

When using NUnit's TheoryAttribute (https://docs.nunit.org/articles/nunit/writing-tests/attributes/theory.html) together with assumptions, it is normal that some test cases of a theory yield an "inconclusive" test result. This is expected, and inconclusive theory results should be filtered out from normal test results, as the NUnit documentation states:

Since the user does not generally care about inconclusive cases under a theory, they are not normally displayed in the Gui. For situations where they are needed - such as debugging - the context menu for the theory provides an option to display them.

However, it seems that Unity returns exit code 2 when having inconclusive tests, and therefore run_tests.sh marks the run illegally as failed, here:

unity-test-runner/dist/steps/run_tests.sh

Line 96 in cdfccd0

echo "Run succeeded, some tests failed";

Instead, the step should not be marked as failed at this point, since it leads to a workflow failure, but detection of the final result should be deferred to the test evaluation step.

How to reproduce

Set up a test project having at least one NUnit theory test, similar to this:

using NUnit.Framework;

namespace Tests.EditMode {

[TestFixture]
public class ExampleTheory {
    [Theory]
    public void SomeTestTheory([Values(1, 2, 3)] int number) {
        // This will not run the theory for number == 3.
        Assume.That(number != 3);
        // This will assert true, since it only runs for number == 1 and number == 2.
        Assert.Less(number, 3);
    }
}

}

This will create three test cases, two of which run successful, and one which runs without conclusion (so neither success nor failure). While the game.ci test report shows 2 successful tests of a total of 3 tests, and no failure, the test step is marked as failure due to the return code emitted by Unity.

[webbertakken]

Hi @moonshinebot. And thanks a lot for the detailed description and clear explanation!

I have marked this issue as an improvement rather than a bug because we're running unity-editor -runTests as per the Unity Editor specification. It is the unity editor itself that returns an erroneous exit code.

That said, I think it's a good idea to implement something in test-runner that will run a check and filter the results like you suggested. We're open to PRs for anyone who'd like to take this on.