Gitlab CI number of tests is wrong

[hoppities]

Version="2.1.81"

Edit: I updated to the latest version and have the same issue.

Tests:
  image: mcr.microsoft.com/dotnet/core/sdk:3.1
  only:
    - merge_requests
    - branches
  script:
    - dotnet test --test-adapter-path:. --logger:"junit;LogFilePath=../artifacts/{assembly}-result.xml;MethodFormat=Class;FailureBodyFormat=Verbose"
  stage: build
  artifacts:
    when: always
    reports:
      junit:
        - $CI_PROJECT_DIR/artifacts/*-result.xml

This is my test step for GitLab CI, and I have tests set up with MSTest that use parameters.

        [DataTestMethod]
        [DataRow(true)]
        [DataRow(false)]
        public async Task Some_Test_Method(bool result)
        {
            ...
        }

The results, however, don't show the parameters that are passed in, so they get combined into one test by Gitlab.

    <testcase classname="My.Test.Class.Name" name="name.Some_Test_Method" time="0.0147381" />
    <testcase classname="My.Test.Class.Name" name="name.Some_Test_Method" time="0.0118888" />
    <testcase classname="My.Test.Class.Name" name="name.Some_Test_Method" time="0.0003662" />

Expected Result: The name of the testcase will include the parameter values like the GitLab CI Documentation states.

Do I have something misconfigured?

[Siphonophora]

Hi, I can do some testing with MSTest, which I think may behave differently from other frameworks with regard to test data.

Based on the name of the ticket, I gather you are also seeing the issue we have here #15. There is an underlying issue with how gitlab counts tests. If that is part of what you are seeing it wouldn't hurt to upvote the gitlab issue https://gitlab.com/gitlab-org/gitlab/-/issues/31975 they have made some progress, but its been slow going.

[hoppities]

Hi, I can do some testing with MSTest, which I think may behave differently from other frameworks with regard to test data.

Based on the name of the ticket, I gather you are also seeing the issue we have here #15. There is an underlying issue with how gitlab counts tests. If that is part of what you are seeing it wouldn't hurt to upvote the gitlab issue https://gitlab.com/gitlab-org/gitlab/-/issues/31975 they have made some progress, but its been slow going.

Yes, I'm seeing that. I'll go upvote that and ask my coworkers to do the same :)

[Siphonophora]

@hoppities MSTest is, in fact, behaving differently from xunit and nunit in this regard. This impacts all our loggers, so I will fix this in the core logger, under issue spekt/testlogger#20

[Siphonophora]

@hoppities I just published a fix for this to myget https://www.myget.org/feed/spekt/package/nuget/JunitXml.TestLogger/3.0.104 If you have time and can test it, let us know. I did run a test shown below.

v3.0.98

<testsuites>
  <testsuite name="TestProject1.dll" tests="17" skipped="2" failures="0" errors="0" time="0.007450599999999999" timestamp="2021-07-27T20:16:04" hostname="W0239867" id="0" package="TestProject1.dll">
    <properties />
    <testcase classname="TestProject1.UnitTest1" name="UnitTest1.TestMethod1" time="0.0035334" />
    <testcase classname="TestProject1.UnitTest1" name="UnitTest1.Some_Test_Method" time="0.0018167" />
    <testcase classname="TestProject1.UnitTest1" name="UnitTest1.Some_Test_Method" time="0.0000478" />
    <testcase classname="TestProject1.UnitTest1" name="UnitTest1.Some_Test_Method" time="0.0000041" />
    <testcase classname="TestProject1.UnitTest1" name="UnitTest1.Some_Async_Test_Method" time="0.0002927" />
    <testcase classname="TestProject1.UnitTest1" name="UnitTest1.Some_Async_Test_Method" time="0.0002446" />
    <testcase classname="TestProject1.UnitTest1" name="UnitTest1.Some_Async_Test_Method" time="0.0000081" />
    <testcase classname="TestProject1.UnitTest1" name="UnitTest1.IsPrime_ValuesLessThan2_ReturnFalse" time="0.0000852" />
    <testcase classname="TestProject1.UnitTest1" name="UnitTest1.IsPrime_ValuesLessThan2_ReturnFalse" time="0.0000344" />
    <testcase classname="TestProject1.UnitTest1" name="UnitTest1.IsPrime_ValuesLessThan2_ReturnFalse" time="0.0000021" />
    <testcase classname="TestProject1.UnitTest1" name="UnitTest1.IsPrime_ValuesLessThan2_ReturnFalse" time="0.0000012" />
    <testcase classname="TestProject1.UnitTest1" name="UnitTest1.TestAddDynamicDataMethod" time="0.0010952" />
    <testcase classname="TestProject1.UnitTest1" name="UnitTest1.TestAddDynamicDataMethod" time="0.0000311" />
    <testcase classname="TestProject1.UnitTest1" name="UnitTest1.TestAddDynamicDataMethod" time="0.0000033" />
    <testcase classname="TestProject1.UnitTest1" name="UnitTest1.TestAddDynamicDataMethod" time="0.0000047" />
    <testcase classname="TestProject1.UnitTest1" name="UnitTest1.DataDrivenTest" time="0.0001306">
      <skipped />
    </testcase>
    <testcase classname="TestProject1.UnitTest1" name="UnitTest1.DataDrivenTest" time="0.0001154">
      <skipped />
    </testcase>
    <system-out></system-out>
    <system-err></system-err>
  </testsuite>
</testsuites>

MyGet preview version 3.0.104 https://www.myget.org/feed/spekt/package/nuget/JunitXml.TestLogger/3.0.104

<testsuites>
  <testsuite name="TestProject1.dll" tests="17" skipped="2" failures="0" errors="0" time="0.006163199999999999" timestamp="2021-07-27T20:22:11" hostname="W0239867" id="0" package="TestProject1.dll">
    <properties />
    <testcase classname="TestProject1.UnitTest1" name="UnitTest1.TestMethod1" time="0.0022678" />
    <testcase classname="TestProject1.UnitTest1" name="UnitTest1.Some_Test_Method" time="0.0016660" />
    <testcase classname="TestProject1.UnitTest1" name="UnitTest1.Some_Test_Method (True)" time="0.0000527" />
    <testcase classname="TestProject1.UnitTest1" name="UnitTest1.Some_Test_Method (False)" time="0.0000037" />
    <testcase classname="TestProject1.UnitTest1" name="UnitTest1.Some_Async_Test_Method" time="0.0002872" />
    <testcase classname="TestProject1.UnitTest1" name="UnitTest1.Some_Async_Test_Method (True)" time="0.0002428" />
    <testcase classname="TestProject1.UnitTest1" name="UnitTest1.Some_Async_Test_Method (False)" time="0.0000089" />
    <testcase classname="TestProject1.UnitTest1" name="UnitTest1.IsPrime_ValuesLessThan2_ReturnFalse" time="0.0000888" />
    <testcase classname="TestProject1.UnitTest1" name="UnitTest1.IsPrime_ValuesLessThan2_ReturnFalse (-1,a)" time="0.0000360" />
    <testcase classname="TestProject1.UnitTest1" name="UnitTest1.IsPrime_ValuesLessThan2_ReturnFalse (0,a)" time="0.0000027" />
    <testcase classname="TestProject1.UnitTest1" name="UnitTest1.IsPrime_ValuesLessThan2_ReturnFalse (1,acsdf)" time="0.0000012" />
    <testcase classname="TestProject1.UnitTest1" name="UnitTest1.TestAddDynamicDataMethod" time="0.0011878" />
    <testcase classname="TestProject1.UnitTest1" name="UnitTest1.TestAddDynamicDataMethod (1,1,2)" time="0.0000334" />
    <testcase classname="TestProject1.UnitTest1" name="UnitTest1.TestAddDynamicDataMethod (12,30,42)" time="0.0000039" />
    <testcase classname="TestProject1.UnitTest1" name="UnitTest1.TestAddDynamicDataMethod (14,1,15)" time="0.0000014" />
    <testcase classname="TestProject1.UnitTest1" name="UnitTest1.DataDrivenTest" time="0.0001472">
      <skipped />
    </testcase>
    <testcase classname="TestProject1.UnitTest1" name="UnitTest1.DataDrivenTest" time="0.0001317">
      <skipped />
    </testcase>
    <system-out></system-out>
    <system-err></system-err>
  </testsuite>
</testsuites>

[Siphonophora]

@codito Two things:

• Between Fix: Use parsed Method, not the test framework dependent name. #45 and Feat: update core logger, for #42 #46, I'm feeling like we might want acceptance / unit testing across multiple test frameworks. Currently we just use Nunit, and those two pull requests were fixing Xunit and MSTest issues respectively. I just realized this issue on mstest was actually a regression vs the v2 logger, we didn't have any way to catch it. What do you think?
• I did manual acceptance testing on these and believe we can release v3.0.102, unless we can readily add some tests, I think this is ready to go.
• Do you want me to make PRs against the other loggers. All our MSTest users should be having this issue.

[hoppities]

@Siphonophora We'll get this updated to the preview and let you know what happens. Thanks!

Edit: We tested this, and it's now displaying tests correctly. We're seeing the parameterized tests show up now rather than being grouped by the test method name.

[Siphonophora]

@hoppities Thanks for the confirmation. I will keep this open till this is published on NuGet.

[BottlecapDave]

Is there an update on when we might see this released? I've also been hit by this issue.

[Siphonophora]

@hoppities @BottlecapDave This is released in 3.0.110