Add coverage support for Bazel

[ssuchter]

I have a project which uses Bazel for Java compilation. Within Intellij, I can run the unit tests both normally and in debug mode. But when I attempt to run a unit test with coverage, I get this output:

Testing started at 7:01 PM ...
ERROR: Unrecognized option: --combined_report=lcov
Empty test suite.

I agree that I can't find any such option in Bazel "--combined_report." Am I missing something?

I've tried this with Bazel 0.5.4 and Bazel 0.7.0. I'm running the plugin version 2017.10.09.0.1.

Am I missing something here? I see that running tests with coverage was a new feature of that release, but I don't see how it could possibly work when using an option that Bazel doesn't seem to have.

Thanks, Sean

[brendandouglas]

Thanks for letting us know. Looks like coverage support works quite differently in Bazel.

For now, I'll just disable coverage integration for Bazel, so the non-functional option doesn't appear.

Leaving this open to add coverage support.

[ssuchter]

"Looks like coverage support works quite differently in Bazel" - differently than what? (E.g. what was being expected?)

I noticed that in the CHANGELOG, this was an entry for 2017.10.09.0.1:

• Support 'Run with Coverage' for all Bazel test run configurations.

That seemed pretty explicit, so I wonder what kind of configuration the change that led to that CHANGELOG entry used.

BTW, just guessing from the name of the offending option - it seems like it is unnecessary. When I run this:

bazel coverage //myproject:MyProjectTests

I get a lcov style combined output file:
...
INFO: Elapsed time: 0.210s, Critical Path: 0.00s
... /private/var/tmp/_bazel_ssuchter/7c7fcfbe09869ece8a3b7ee9ffb8cb4a/execroot/main/bazel-out/darwin_x86_64-fastbuild/testlogs/myproject/MyProjectTests/coverage.dat

So perhaps what was attempting to be accomplished with that option is already the default, and the option is unnecessary?

Thanks, Sean

[brendandouglas]

Coverage support (and almost all other features) are developed for Blaze, the internal version of Bazel.

Generally, things just work in Bazel as well, and when they don't, we typically notice during testing. We don't use Bazel ourselves though (although our tests do), so some things do slip through.

In this case, Blaze has an option to provide a single output file combining coverage results for all test targets run during the 'bazel test' invocation. That functionality is apparently not present in Bazel, so we'd need to parse the coverage results on a per-target basis.

It's not difficult to implement, it's just a matter of finding time to do it.

[hsyed]

What is this status on this on ? This would help in verifying the coverage support in the Kotlin rules.

[brendandouglas]

We have no current plans to work on this -- it's something that would be nice to have, but to be honest is unlikely to ever reach the top of our priority queue.

[keithkml]

I notice that the "Run with coverage" button is visible but grayed out. Are you saying this button does not work and never will?

[brendandouglas]

Actually, it looks like bazel now supports exactly the same combined coverage output we use for blaze (bazelbuild/bazel#5246).

I'll check whether it works as expected, and if so, we'll turn on coverage support for bazel.

[brendandouglas]

It's failing, at least for java targets, due to bazelbuild/bazel#6450.

I'll turn it on for now anyway, and look at fixing bazel coverage separately.

[brendandouglas]

I'll close this for now. The button will be enabled in the release after next, though coverage support does appear to be entirely broken on the bazel side (at least with the --combined_report=lcov option).