Running Code Coverage

[julianharty]

This is related to issue #836 which focuses on enabling the generation of code coverage reports. I've decided to create this issue where I/we can focus on how (and perhaps also where) the code coverage reports can and will be generated as there are distinct topics such as:

• Do we want to generate code coverage on each build, or on a periodic basis?
• Do we care whether it's generated on real devices or would an emulator do?
• How many devices should we run the coverage on? and if we run it on several, what do we do in terms of combining/harmonising the results into a cohesive report?
• Where will the reports appear?
• Will we use a free service or one that requires payment? if payment is needed, who pays and makes sure the payment continues without interruption?

As a starting point: in a discussion with @kelson42 he would prefer code coverage to be generated periodically and the reports hosted externally. He doesn't want them to be generated on each pull request.

There seem to be various logistical challenges in terms of configuring and generating code coverage. Also the code coverage results in one or more files being created on the Android device, sometimes in a private directory/folder. We'll need to adapt our approach so it works despite these restrictions and challenges.

[julianharty]

As a quick update: I've successfully generated code coverage for Kiwix Android on bitbar's cloud. To do this I ended up creating a fresh project, partly as I couldn't see a way to change the configuration for the current project (I'll ask the Bitbar team for their advice on whether we can do this).

The coverage file is generated and placed in the output-files.zip file for that test run. In here there's the coverage.ec file. The next challenge is to generate coverage reports using this file together with other files generated as part of the Gradle build. I'm experimenting with this now.

[julianharty]

I think the various prerequisites are now in place for us to enable code coverage for our tests that run on Bitbar's device cloud. They have updated their systems, API, and gradle plugin bitbar/android-gradle-plugin#28 so that we can now specify we want to use their test framework that enables the code coverage file to be generated when the tests are run. They also fixed some bugs in their scripts e.g. which meant the video recordings weren't generated initially when they generated code coverage (they're not mutually exclusive, it was just a bug introduced accidentally when they implemented the code coverage feature).

I've created and tested a new 'Project', as they're called by Bitbar, which has the same set of devices as we currently use.

To implement the change in our codebase we need to:

• add a new parameter frameworkId set to 748.
• update app/build.gradle to use at least release 2.63.1 of the testdroid dependency: classpath 'com.testdroid:gradle:2.63.1'

We may need to also specify the new Project which we'd do in 2 places:

1. in app/build/gradle to update the Project name from Kiwix to KiwixCoverage, and
2. in testdroid.py python script to poll for Project Id 194794698 (which matches the project ID used to view the project's test results interactively for those who have direct access to their environment i.e. https://cloud.bitbar.com/#testing/projects/194794698 - you need a valid account on their system to see the project details).

These changes might not be absolutely necessary owing to the nature of how Bitbar currently implemented their changes, nonetheless if we do apply these changes the overall understanding will be clearer (we specify we want to run the code coverage test framework on the project that has test coverage enabled when it was created). There are a bunch of details I'm eliding here related to how to create and revise projects in Bitbar's system vs. how we've configured our scripts to run the tests and obtain the results.

Details on the testdroid Gradle plugin are available in their project's README https://github.com/bitbar/testdroid-gradle-plugin

And the various releases of their plugin are available at https://github.com/bitbar/testdroid-gradle-plugin/releases

[julianharty]

Note: my interim code changes are in a local branch julianharty/enable_code_coverage_without_breaking_debugging

[mhutti1]

@julianharty Have you pushed that branch?

[julianharty]

@mhutti1 I hadn't, got distracted last week with dental treatment and dealing with family stuff. I can see you've made some major changes with #1016 which breaks my changes. I've had a go at setting the frameworkId and running the tests - I'm able to run the tests ./gradlew kiwixtestUploadKiwix and can see they ran successfully on bitbar devices. However the coverage files weren't created and I've no idea where I can configure the tests in TestDroidUpload in https://github.com/kiwix/kiwix-android/blob/89f74f4b7c51792c016fe7a0381e18b07c5818da/app/build.gradle Can you give me any pointers please?

[mhutti1]

Yeah so basically our run configuration has now been moved to a separate file where we call the API directly. I have shared the repo with you (its private for now). We can change the framework Id and other config options in https://github.com/mhutti1/kiwix-test-gateway/blob/master/runtest.php#L26 and we should have even more control than before.

With regards to running the coverage variant. We can probably set testBuildType with an enviromental var set by travis but debug by default.

[julianharty]

@mhutti1 Excellent and thank you for sharing the repo. Would you mind changing the frameworkId to 748 and pushing the change to wherever the code runs? then kick off travis-CI (presumably) to run a set of tests on bitbar that use this frameworkId. If the frameworkId is accepted the tests should run, essentially unchanged, the only material difference would be each device's test output would include a zip file (called something like output-files.zip - in that zip file there should then be a coverage.ec file. I'll then like to create a script that downloads this zip file, unpacks the zip file and generates a code coverage report. It'll need to run somewhere where the app has been built (so perhaps in the same docker image used to build the app?) as it needs access to the code and (I think) also the intermediate build files.

[mhutti1]

Sounds good. I will try to get this working.

[julianharty]

@mhutti1 Would it be helpful if I came to London and/or we arrange a pairing session somehow so we can break the back of this work? Just ask, I can make the time to help get this done. Thank you.

[mhutti1]

Hey, yeah I'm very busy this and next week with Uni Exams + GSoC Enquires, I am open to suggestions but can't spend a huge amount of time on this for two weeks :(

[mhutti1]

@julianharty I have made the framework changes and submitted a pr to change the default build #1137 to test this. It works and can be seen in Auto Test 2282 on the Kiwix project on testdroid. As mentioned this breaks debugging so I will try to configure this option on the fly to use debug locally and coverage on travis.

[julianharty]

@mhutti1 Excellent, thank you. I can see the output-files.zip has been generated and contains coverage.ec 👍

From what I understand this file needs to be processed together with the intermediate build files (I don't yet know if the coverage report is generated from the source files, the class files, or a mix) if the coverage report is to be accurate. Anyway, we're at least one step closer :)

BTW: You'd mentioned in another issue #1116 that Google have finally added support for a default build target, perhaps we can find a way to implement support for both?

[macgills]

Resolved by #1176 , closing