Measure code coverage

[Popolechien]

As part of our deal with the Foundation we're supposed to increase code coverage but in order to do so we should have the means to do it.

We need A HTML based report:

• computed automatically, for example one time a night on mastet
• which include code coverage numbers gathered including unit testing, integration testing and UI testing
• A way to visualize module by module which part of the code is covered

[kelson42]

@julianharty Would you be interested to measure this within the automated tests?

[mhutti1]

In the past we have looked into this a bit. We have a build variant with a coverage library I think for instrumentation tests and unit test coverage can be measured by Android Studio. I think we would need to flesh these out more to meet the requirements of this issue though.

[dr0pdb]

Hi @mhutti1 @kelson42
Got free from GSoC duties at Fossasia and hence back.

If I understood the requirements of the issue correctly, we need a way to measure the code coverage of the project. Most other open source Android projects use https://codecov.io/. It can easily be integrated with Travis. Sample integration example: https://github.com/codecov/example-android

It will display the code coverage for each PR. Let me know if you want to integrate this.

[abdulwd]

@srv-twry Welcome back!
The JVM tests will be run on Travis so, codecov.io can be used but the Espresso tests are run on physical devices hosted by Bitbar. It will be great if you can tell how will you get the coverage of the Espresso tests.

[dr0pdb]

@abdulwd Hi, I wasn't aware of this. I thought we run integration tests on Travis only. I'll do some more research to see if I can find a solution to this.

[kelson42]

@srv-twry Welcome back, I have enhanced the requirements. All-in-one they are quite standards.

[kelson42]

The priority of this ticket has been raised up, we need to get these number for october. If someone wants to volunteering on this, I would be really thankful. Maybe @siddharth2010 this is finally a solution to measure all the good work you have done on testing :)

[julianharty]

A quick update. I have been able to generate a plausible code coverage when running the tests on a particular device (a Nexus 5x running Android 8.0.0).

When I ran the tests on a Samsung Phone (a Galaxy2 SM-G360F running Android 5.0.2) the coverage report was generated but showed 0 lines were covered i.e. coverage of 0.0%

I'd temporarily changed the debug build flavor to enable test coverage as I was struggling to get any coverage report to be generated; (this may be caused by other factors Action: now I've got something working I'll reverse this change and retest)

Several tests failed on the Samsung device. I don't yet know if failing tests are enough to cause the 0 lines covered on this device. Action: I'll investigate or whether the cause is a function of running code coverage on [some] Samsung devices generating a coverage file of 0 bytes e.g. see https://stackoverflow.com/a/37475182/340175

It's also not clear yet which build targets actually cause the tests to run and the coverage report to be generated. There are numerous build targets that include coverage:

./gradlew tasks | grep -i Coverage
assembleCoverage - Assembles all Coverage builds.
compileCustomexampleCoverageSources
compileCustomexampleCoverageUnitTestSources
compileKiwixCoverageSources
compileKiwixCoverageUnitTestSources
installCustomexampleCoverage - Installs the CoverageCustomexample build.
installKiwixCoverage - Installs the CoverageKiwix build.
uninstallCustomexampleCoverage - Uninstalls the CoverageCustomexample build.
uninstallKiwixCoverage - Uninstalls the CoverageKiwix build.
createCustomexampleCoverageCoverageReport - Creates test coverage reports for the customexampleCoverage variant.
createCustomexampleDebugCoverageReport - Creates test coverage reports for the customexampleDebug variant.
createCustomexampleLocal_download_serverCoverageReport - Creates test coverage reports for the customexampleLocal_download_server variant.
createCustomexampleMock_networkCoverageReport - Creates test coverage reports for the customexampleMock_network variant.
createKiwixCoverageCoverageReport - Creates test coverage reports for the kiwixCoverage variant.
createKiwixDebugCoverageReport - Creates test coverage reports for the kiwixDebug variant.
createKiwixLocal_download_serverCoverageReport - Creates test coverage reports for the kiwixLocal_download_server variant.
createKiwixMock_networkCoverageReport - Creates test coverage reports for the kiwixMock_network variant.
lintCustomexampleCoverage - Runs lint on the CustomexampleCoverage build.
lintKiwixCoverage - Runs lint on the KiwixCoverage build.
testCustomexampleCoverageUnitTest - Run unit tests for the customexampleCoverage build.
testKiwixCoverageUnitTest - Run unit tests for the kiwixCoverage build.

The first one that worked for me is:
./gradlew createKiwixDebugCoverageReport --info

Note: this is with testCoverageEnabled = true in the debug section of app/build.gradle otherwise this build target doesn't exist
Tips:

• Code coverage reports are created in sub-directories of ./app/build/outputs/code-coverage/

[julianharty]

Reverting the testCoverageEnabled to false in app/build.gradle firstly changes the build targets in Gradle, secondly and more relevantly it seems that gradle doesn't perform any build actions for build targets (a.k.a. Gradle tasks) with Coverage in their name.

I've experimented by changing the section called coverage to Julian to see how gradle tasks such as createKiwixCoverageCoverageReport get generated (it becomes createKiwixJulianCoverageReport BTW). That didn't help though in terms of getting coverage to be generated. Action: I'll investigate how properties are inherited in gradle build files

[julianharty]

Seems like the Samsung device is falling foul of the 0 length coverage file reported by others:

ls -l ./app/build/outputs/code-coverage/connected/flavors/KIWIX/SM-G360F\ -\ 5.0.2-coverage.ec
-rw-r--r--  1 julianharty  staff  0 17 Sep 15:14 ./app/build/outputs/code-coverage/connected/flavors/KIWIX/SM-G360F - 5.0.2-coverage.ec

[julianharty]

After several test runs where tests failed I have finally managed to experience a clean test run and a coverage report was generated that includes code coverage data.

On the previous test run several tests failed. That time a coverage.ec was also generated with contents but the report wasn't generated. I've yet to discover how to generate a report when there are failing tests _Action: be great to find a way to generate the report as and when we have a coverage.ec with contents.

[julianharty]

Yet more notes, if only for my own failing memory...
When tests fail, the coverage report doesn't seem to be updated.

Here are the details for the report's index.html (last updated over an hour ago, when all the tests passed for the coverage build target)

ls -l app/build//reports/coverage/kiwix/debug/org.kiwix.kiwixmobile/index.html
-rw-r--r--  1 julianharty  staff  27878 17 Sep 15:49 app/build//reports/coverage/kiwix/debug/org.kiwix.kiwixmobile/index.html

However the coverage data is more recent:

ls -l app/build//outputs/code-coverage/connected/flavors/KIWIX/Nexus\ 7\ -\ 5.1.1-coverage.ec 
-rw-r--r--  1 julianharty  staff  14121 17 Sep 16:37 app/build//outputs/code-coverage/connected/flavors/KIWIX/Nexus 7 - 5.1.1-coverage.ec

This behaviour is unwelcome and the effect is the coverage report is misleading. Running clean removes the previous/old coverage report so for now I've combined it with the other build target:
./gradlew clean createKiwixDebugCoverageReport --info

[julianharty]

Here, finally seems to be the way to circumvent various gotchas related to generating code coverage for the tests, at least on devices that provide coverage reports with data in.

This answer is 3 years old https://groups.google.com/forum/#!topic/adt-dev/DUGRuTm6u1g

The gradle plugin right now can only run tests against a single build type, which by default is debug.

You can change that with

android {
   testBuildType "coverage"
}

which was the key to enabling me to use a buildType other than debug to generate the coverage. Gradle otherwise declares the tasks to generate coverage for other buildTypes have no actions. It'd be great if they also provided a clue as to the cause (the above limitation).

Note: I've tested (using Gradle 4.6, 2 Android devices, and various build targets from both the command line using the gradle wrapper and also Android Studio 3.2 Preview) that I can:

• debug and see the local variables when using a debug build target
• generate code coverage and see data in the report
• run the espresso tests for both debug and coverage buildTypes

Limitations:
I can't be sure other foibles won't emerge; and we've yet to decide how to run code coverage in the combined environment of travis-ci and TestDroid.

[julianharty]

For those that follow on, here's one of the best examples of the problem and the way we addressed the issue: https://stackoverflow.com/questions/31720389/coverage-on-android-build

[mhutti1]

@julianharty We really can't have the testBuildType set to coverage. I spent 3 hours trying to work out why it wasn't working and it was giving me completely unrelated errors. We need to change it back to debug. There's probably a way to pro grammatically change it when needed.

[kelson42]

@macgills Might that be that you have done all the work to achieve this?

[macgills]

Yes, the pr I just merged addresses this

[macgills]

Resolved by #1176, closing