JUnitReport.xml only includes 50 tests out of 73 #1634

dmytrodanylyk · 2021-02-26T03:38:23Z

👋 Thank you for this amazing tool.

Describe the bug
There are 2 issues that are probably related:

When I run tests via flank with max-test-shards: "5" for some reason it uses only 1 shard, I can't find the reason why.
The JUnitReport.xml which is created only contains 50 tests, while matrix_0/Nexus5-23-en-portrait/test_result_1.xml has 73 tests.

Is there any hard-coded value of the number of tests JUnitReport.xml can have?

To Reproduce

Steps to reproduce the behaviour:

Run tests via flank with following configuration:

gcloud:
  app: '/Users/ddanylyk/.../build/testlab/dummy.apk'
  test: '/Users/ddanylyk/Work/.../debug/android-debug-androidTest.apk'
  use-orchestrator: false
  record-video: true
  auto-google-login: false
  environment-variables:
    coverage: false
    clearPackageData: false
    coverageFile: /sdcard/coverage.ec
  device:
    - model: Nexus5
      version: 23
      locale: en
      orientation: portrait
  test-targets:
     - 'notAnnotation QuarantineTest'
flank:
  max-test-shards: '5'
  full-junit-result: true
  shard-time: '120'
  num-repeat-tests: '1'
  disable-sharding: 'false'
  smart-flank-gcs-path: gs://...testlabRunAndroidTest.xml

Expected behaviour

Flank should use more than 1 shard if max-test-shards: '5'.
JUnitReport.xml should contain all tests, not just 50.

Details (please complete the following information):

Have you tested on the latest Flank snapshot?

No

Post the output of flank --version.

v20.09.3 (tried v21.02.0 same issue)

Additional context

Exanple of matrix_0/Nexus5-23-en-portrait/test_result_1.xml file:

<?xml version='1.0' encoding='UTF-8' ?>
<testsuite name="" tests="73" failures="0" errors="0" skipped="0" time="25.17" timestamp="2021-02-26T03:07:15" hostname="localhost">
  <properties />
  <testcase name="test1" classname="...CustomTagHandlerTest" time="0.306" />
  <testcase name="test2" classname="...CustomTagHandlerTest" time="0.306" />
...71 more entries
</testsuite>

Example of JUnitReport.xml file:

<?xml version='1.0' encoding='UTF-8' ?>
<testsuites>
  <testsuite name="Nexus5-23-en-portrait" tests="50" failures="0" flakes="0" errors="0" skipped="0" time="25.170" timestamp="2021-02-26T03:06:50" hostname="localhost">
    <testcase name="test1" classname="...CustomTagHandlerTest" time="0.489">
      <webLink>https://console.firebase.google.com/project/.../testcases/1</webLink>
    </testcase>
...49 more entries
</testsuites>

The text was updated successfully, but these errors were encountered:

pawelpasterz · 2021-02-26T06:01:08Z

Hey 👋
Thanks for reporting. We will take a look shortly.

Is there any hard-coded value of the number of tests JUnitReport.xml can have?

There is not, JUnitReport should contain all tests

There should be android_shards.json created by the flank (it is done automatically since v20.12.0, with an earlier version you need to add --dump-shards). This file contains shards created by flank (as JSON representation), could you verify if there is something missing or wrong? You can send you obfuscated JSON (add --obfuscate)

Do you have parameterized tests?

Sorry for a bunch of questions, it will help us to track the problem :)

pawelpasterz · 2021-02-26T06:14:55Z

Ah, I think I've found something...
@dmytrodanylyk could you try with full-junit-result: false ?

dmytrodanylyk · 2021-02-26T06:18:09Z

@pawelpasterz the full-junit-result: true is something I tried to see if .xml generated by this flag would contain all tests, but unfortunately it did not. My original YAML flank file didn't have this flag.

I am trying to update to v21.02.0 and check the android_shards.json, I will post results shortly.

Thank you for jumping on this issue so quickly.

pawelpasterz · 2021-02-26T06:24:01Z

NP :)

full-junit-result is used mainly together with num-flaky-tests-attempts: [number]. If set with true Flank will produce FullJUnitTestResult with all runs for specific test case (meaning, it includes failed attempts for flaky tests)

dmytrodanylyk · 2021-02-26T06:56:45Z

Ok, it may take more time than I expected to run flank with --dump-shards because we are using some custom gradle plugin to tun flank in Android.

I noticed a strange coincidence that the smart flank cache hit is 50 / 73, and that's the number of tests that I see in JUnitReport. Can this be an issue 🤔 ?

RunTests

 Smart Flank cache hit: 68% (50 / 73)
  Shard times: 2790s

  Uploading dummy.apk .
  Uploading androidTest.apk .
  73 tests / 1 shard

pawelpasterz · 2021-02-26T07:19:46Z

That's definitely valuable info for us. 50 means flank found 50 tests in XML provided in smart flank path. Meaning flank has acquired times for 50 tests and it will be used in shard calculation. For the rest default value will be used (120s)

Do you have parameterized tests?

dmytrodanylyk · 2021-02-26T07:58:27Z

Do you have parameterized tests?
Nope

dmytrodanylyk · 2021-02-26T12:53:27Z

The android_shards.json looks perfectly normal, there are 73 tests (as expected).

{
  "matrix-0": {
    "app": ".../build/testlab/dummy.apk",
    "test": ".../debug-androidTest.apk",
    "shards": {
      "shard-0": [
        "class a.a.a.a.a.a.A#a",
        "class a.a.a.a.a.a.A#b",
        "class a.a.a.a.a.a.A#c",
        "class a.a.a.a.a.a.A#d",
        "class a.a.a.a.a.a.A#e",
        "class a.a.a.a.a.a.A#f",
        "class a.a.a.a.a.a.A#g",
        "class a.a.a.a.a.a.A#h",
        "class a.a.a.a.a.a.A#i",
        "class a.a.a.a.a.a.A#j",
        "class a.a.a.a.a.a.A#k",
        "class a.a.a.a.a.a.A#l",
        "class a.a.a.a.a.a.B#a",
        "class a.a.a.a.a.a.B#b",
        "class a.a.a.a.a.a.B#c",
        "class a.a.a.a.a.a.B#d",
        "class a.a.a.a.a.a.B#e",
        "class a.a.a.a.a.a.B#f",
        "class a.a.a.a.a.a.B#g",
        "class a.a.a.a.a.a.B#h",
        "class a.a.a.a.a.a.B#i",
        "class a.a.a.a.a.a.B#j",
        "class a.a.a.a.a.a.B#k",
        "class a.a.a.b.a.a.A#a",
        "class a.a.a.b.a.a.A#b",
        "class a.a.a.a.a.a.C#a",
        "class a.a.a.a.a.b.A#a",
        "class a.a.a.a.a.b.A#b",
        "class a.a.a.b.a.a.A#c",
        "class a.a.a.b.a.a.A#d",
        "class a.a.a.b.a.a.A#e",
        "class a.a.a.b.a.a.A#f",
        "class a.a.a.b.a.a.A#g",
        "class a.a.a.a.a.a.D#a",
        "class a.a.a.a.a.a.E#a",
        "class a.a.a.a.a.A#a",
        "class a.a.a.a.a.a.C#b",
        "class a.a.a.a.a.a.C#c",
        "class a.a.a.a.a.A#b",
        "class a.a.a.a.a.A#c",
        "class a.a.a.a.a.A#d",
        "class a.a.a.a.a.A#e",
        "class a.a.a.a.a.A#f",
        "class a.a.a.a.a.A#g",
        "class a.a.a.a.a.A#h",
        "class a.a.a.a.a.A#i",
        "class a.a.a.a.a.A#j",
        "class a.a.a.a.a.a.D#b",
        "class a.a.a.a.a.a.D#c",
        "class a.a.a.a.a.a.D#d",
        "class a.a.a.a.a.a.D#e",
        "class a.a.a.a.a.a.D#f",
        "class a.a.a.a.a.a.D#g",
        "class a.a.a.a.a.a.D#h",
        "class a.a.a.a.a.a.B#l",
        "class a.a.a.a.a.a.B#m",
        "class a.a.a.a.a.a.B#n",
        "class a.a.a.a.a.a.B#o",
        "class a.a.a.a.a.a.B#p",
        "class a.a.a.a.a.a.E#b",
        "class a.a.a.a.a.a.F#a",
        "class a.a.a.a.a.a.F#b",
        "class a.a.a.a.a.a.F#c",
        "class a.a.a.a.a.a.F#d",
        "class a.a.a.a.a.a.F#e",
        "class a.a.a.a.a.a.F#f",
        "class a.a.a.a.a.a.G#a",
        "class a.a.a.a.a.a.G#b",
        "class a.a.a.a.a.a.G#c",
        "class a.a.a.a.a.a.G#d",
        "class a.a.a.a.a.a.C#d",
        "class a.a.a.a.a.a.C#e",
        "class a.a.a.a.a.a.C#f"
      ]
    },
    "junit-ignored": []
  }
}

pawelpasterz · 2021-02-26T13:01:25Z

Thanks a lot!

I see we have 2 issues here (could be related though):

tests are not shared at all
missing testcases in result file

Ok, it may take more time than I expected to run flank with --dump-shards because we are using some custom gradle plugin to tun flank in Android.

Is it fladle? 🤔

dmytrodanylyk · 2021-02-26T13:21:54Z

Is it fladle? 🤔

No, we wrote our own testlab-plugin looong time ago to proxy all the work related to testlab, since we have multiple products which are using testlab.

It really looks like it just took 50 first testcase for some reasons. All testcase in test_result_1.xml and JUnitReport.xml are in perfect order.

It doesn't happen with multiple shards, e.g. if there are 4 matrix each contains N<50 number of tests, than JUnitReport.xml will have all 73 tests.

dmytrodanylyk · 2021-03-01T03:56:06Z

Hey, good news I found what the problem is.

    fun listTestCases(toolResultsStep: ToolResultsStep): ListTestCasesResponse {
        return service
            .projects()
            .histories()
            .executions()
            .steps()
            .testCases()
            .list(
                toolResultsStep.projectId,
                toolResultsStep.historyId,
                toolResultsStep.executionId,
                toolResultsStep.stepId
            ).executeWithRetry()
    }

Basically, this code executes an API request which only returns 50 results.

dmytrodanylyk · 2021-03-01T04:58:55Z

Here is the patch with a fix.

#1634_-_Fixed_issue_when_JUnitReport_xml_only_contained_50_test_results.patch.zip

pawelpasterz · 2021-03-01T06:05:24Z

Hey, good news I found what the problem is.

    fun listTestCases(toolResultsStep: ToolResultsStep): ListTestCasesResponse {
        return service
            .projects()
            .histories()
            .executions()
            .steps()
            .testCases()
            .list(
                toolResultsStep.projectId,
                toolResultsStep.historyId,
                toolResultsStep.executionId,
                toolResultsStep.stepId
            ).executeWithRetry()
    }

Basically, this code executes an API request which only returns 50 results.

Awesome! PRs are always welcome 😸
Anyway, we will use your solution once we will start working on this issue.

One puzzle is done, we need to find out why shading is not working in your case 🤔

dmytrodanylyk · 2021-03-01T06:11:56Z

Awesome! PRs are always welcome 😸

I will try to send PR tomorrow, it has to be from the forked repo right?

One puzzle is done, we need to find out why shading is not working in your case 🤔

I think it just hit the limit, we have over 100 modules and 800 android tests

bootstraponline · 2021-03-01T13:37:19Z

I will try to send PR tomorrow, it has to be from the forked repo right?

Yes, please send a PR from a forked repo. Thanks for debugging and fixing the issue.

…results

Fixes #1634 ## Test Plan > How do we know the code works? Existing tests should not be broken ## Checklist - [x] Documented - [x] Unit tested - [x] Integration tests updated

dmytrodanylyk added the Bug label Feb 26, 2021

bootstraponline added the P0 label Feb 26, 2021

zuziaka added the Tiger 🐯 label Mar 1, 2021

zuziaka added this to the Sprint 3 milestone Mar 1, 2021

dmytrodanylyk pushed a commit to dmytrodanylyk/flank that referenced this issue Mar 1, 2021

Flank#1634 - Fixed issue when JUnitReport.xml only contained 50 test …

70e871e

…results

dmytrodanylyk mentioned this issue Mar 1, 2021

fix: JUnitReport.xml only contained 50 test results #1649

Merged

3 tasks

bootstraponline pushed a commit to dmytrodanylyk/flank that referenced this issue Mar 1, 2021

Flank#1634 - Fixed issue when JUnitReport.xml only contained 50 test …

f69ba8f

…results

dmytrodanylyk added a commit to dmytrodanylyk/flank that referenced this issue Mar 2, 2021

Flank#1634 - Fixed issue when JUnitReport.xml only contained 50 test …

e18b78f

…results

piotradamczyk5 added a commit to dmytrodanylyk/flank that referenced this issue Mar 2, 2021

Merge branch 'master' into Flank#1634-junitreport-includes-only-50-tests

bb8e5a7

mergify bot closed this as completed in #1649 Mar 2, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

JUnitReport.xml only includes 50 tests out of 73 #1634

JUnitReport.xml only includes 50 tests out of 73 #1634

dmytrodanylyk commented Feb 26, 2021

pawelpasterz commented Feb 26, 2021 •

edited

Loading

pawelpasterz commented Feb 26, 2021

dmytrodanylyk commented Feb 26, 2021

pawelpasterz commented Feb 26, 2021

dmytrodanylyk commented Feb 26, 2021

pawelpasterz commented Feb 26, 2021

dmytrodanylyk commented Feb 26, 2021

dmytrodanylyk commented Feb 26, 2021

pawelpasterz commented Feb 26, 2021

dmytrodanylyk commented Feb 26, 2021

dmytrodanylyk commented Mar 1, 2021 •

edited

Loading

dmytrodanylyk commented Mar 1, 2021

pawelpasterz commented Mar 1, 2021

dmytrodanylyk commented Mar 1, 2021 •

edited

Loading

bootstraponline commented Mar 1, 2021

JUnitReport.xml only includes 50 tests out of 73 #1634

JUnitReport.xml only includes 50 tests out of 73 #1634

Comments

dmytrodanylyk commented Feb 26, 2021

pawelpasterz commented Feb 26, 2021 • edited Loading

pawelpasterz commented Feb 26, 2021

dmytrodanylyk commented Feb 26, 2021

pawelpasterz commented Feb 26, 2021

dmytrodanylyk commented Feb 26, 2021

pawelpasterz commented Feb 26, 2021

dmytrodanylyk commented Feb 26, 2021

dmytrodanylyk commented Feb 26, 2021

pawelpasterz commented Feb 26, 2021

dmytrodanylyk commented Feb 26, 2021

dmytrodanylyk commented Mar 1, 2021 • edited Loading

dmytrodanylyk commented Mar 1, 2021

pawelpasterz commented Mar 1, 2021

dmytrodanylyk commented Mar 1, 2021 • edited Loading

bootstraponline commented Mar 1, 2021

pawelpasterz commented Feb 26, 2021 •

edited

Loading

dmytrodanylyk commented Mar 1, 2021 •

edited

Loading

dmytrodanylyk commented Mar 1, 2021 •

edited

Loading