multiproject reports using coverageAggregate are incomplete

[johnynek]

suppose you two subprojects: core and tests. Now, in tests, there are is some test harness code that makes it easier to run your tests that others might want (in tests/src/main). Then you exercise this harness in tests/src/test. The coverage for core/src/main that you get from the tests in tests/src/tests will be excluded from the report.

The report seems to only aggregate code that is tested IN ITS OWN test directory.

[lloydmeta]

👍 Running into this issue w/ a multi-subproject Play app as well.

In our case, we have integration tests written in the main/root app, where we inject concrete implementations for interface-typed constructor params in our subprojects' controllers. Like @johnynek , we are seeing that the coverage provided by these tests (in our root project) are not counted in the aggregated report.

cc @mauhiz

[kamilkloch]

+1

[bfil]

I'm experiencing this as well, I check regularly hoping for an update on the issue.

I tend to write libraries with the following multi-project structure:

example
example-testkit
example-tests

So providing a separate testkit can be used to write tests more easily for the users of the library, and the actual example-tests would also depend on the example-testkit to write the tests for the library.

This project structure just makes scoverage output 0% coverage on the main project, which makes it not very useful for this use case.

[kamilkloch]

Aggregated coverage seems to be working only after two consecutive runs of coverage test. See the demo of this peculiar behaviour here: https://github.com/kamilkloch/scoverage-multi-module

[bfil]

@kamilkloch you are right, the two consecutive runs do generate a usable report. The first one yields 0% test coverage but the second time it correctly generates the report.

I noticed though that using this approach generates broken links to the file sources in the report, the original issue about this was here: #69
The links are generated fine in the other case, but the coverage is wrong.

[kamilkloch]

@bfil My aggregated links are fine, are you using sbt-coverage 1.1.0?

[bfil]

Oh no that was my fault, I tried on a different project which was still on 1.0.4, on the projects using 1.1.0 the links are fine!

[RichardBradley]

I have the same problem, but running "test" twice doesn't fix it for me.

I have uploaded a sample project to https://github.com/RichardBradley/sbt-scoverage-multiproject-sample

If I run

sbt clean coverage test
sbt coverage test
sbt coverageAggregate

then I don't get any report for the "util" project: "No subproject data to aggregate, skipping reports"

Is there a workaround?

[RichardBradley]

I've found a workaround for my case:

sbt clean
sbt "set coverageEnabled := true" "set coverageEnabled in util := true" test coverageReport
sbt coverageAggregate

I found that running

sbt clean coverage compile

would instrument the main project but not the "util" project, but running

sbt clean coverage "set coverageEnabled in util := true" compile

would instrument the "util" project but not the main project.

EDIT: I'm testing if instrumentation was applied using "javap":

javap -c -classpath target/scala-2.10/classes/ myproject.App ; javap -c -classpath util/target/scala-2.10/classes/ util.Helper$

Something is definitely up with the "coverage" command in multi-project builds.

[RichardBradley]

I have a new version of the "coverage" command which fixes this for me:

commands += Command.command("coverageAllProjs", "enable coverage (all projects)", "") { state: State =>
  val extracted = Project.extract(state)
  val newSettings = extracted.structure.allProjectRefs map { proj =>
    coverageEnabled in proj := true
  }
  extracted.append(newSettings, state)
}

Now for the https://github.com/RichardBradley/sbt-scoverage-multiproject-sample project, if I run:

sbt clean coverageAllProjs test coverageReport coverageAggregate

then I get a working coverage report.

I'll create a PR to update the coverage command.

[sksamuel]

Thanks for your work on this Richard.