Documentation on generating test coverage is outdated

[bvobart]

The TESTING documentation states that test coverage can be generated with a Maven command. However, Elasticsearch currently uses Gradle. Issue #13930 describes the migration from Maven to Gradle and states that the documentation for building and testing was completely updated except for the documentation on JaCoCo reports.

So is it currently possible to create a test coverage report? And if so, how? I'd gladly open a PR with updated documentation once I get it working.

[jasontedor]

So is it currently possible to create a test coverage report? And if so, how?

No, more than a documentation update is needed here, the Jacoco coverage is not even configured into the build.

I'd gladly open a PR with updated documentation once I get it working.

Are you offering to contribute all the work needed here?

[jasontedor]

FYI @elastic/es-core-infra.

[bvobart]

Ahh okay. Sure, I'll be able to contribute. Seeing as I'll have to rummage around in the build configuration, is there any specific place you would like me to place the JaCoCo configuration? Or anything else I should keep in consideration?

[jasontedor]

Thanks a lot!

Take a look at our BuildPlugin in the buildSrc directory where we configure the Java plugin. We would want to configure the Jacoco plugin here too (this way all Java sub-projects pick it up). Also take a look at what is done in BootstrapForTesting to ensure that Jacoco has permissions to write out its output.

[hub-cap]

Feel free to ping me as a reviewer as well @bvobart , ill run it thru the paces locally once you get it pushed up

[bvobart]

@jasontedor @hub-cap I'm having quite some problems getting JaCoCo working (or Cobertura for that matter). Having tried adding JaCoCo in multiple different ways without success, I finally found that RandomizedTestingTask is the culprit of why JaCoCo does not execute.

It turns out that the JaCoCo plugin looks for all tasks with type org.gradle.api.tasks.testing.Test. However, the plugin that enables randomized testing replaces the regular testing tasks with these RandomizedTestingTasks, which don't inherit from the Test type. JaCoCo is therefore not able to recognize these tasks as testing tasks and add the JaCoCo extension (I believe Cobertura works in the same way as well).

While RandomizedTestingTask and Test both extend DefaultTask, it is apparently not possible to just change RandomizedTestingTask to extend Test for two reasons.
The first is easy to solve; a number of methods that RandomizedTestingTask overrides (jvmArgs, include, exclude etc.) are required by Test to return an instance of Test, namely itself (this) (see here) to enable chaining. Nothing too weird here.
The second is more intricate; upon executing ./gradlew test jacocoTestReport the following error occurs:

Click to expand error message

$ ./gradlew test jacocoTestReport --stacktrace
To honour the JVM settings for this build a new JVM will be forked. Please consider using the daemon: https://docs.gradle.org/4.5/userguide/gradle_daemon.html.
Daemon will be stopped at the end of the build stopping after processing

> Configure project :benchmarks 
=======================================
Elasticsearch Build Hamster says Hello!
=======================================
  Gradle Version        : 4.5
  OS Info               : Linux 4.13.0-36-generic (amd64)
  JDK Version           : Oracle Corporation 9.0.4 [Java HotSpot(TM) 64-Bit Server VM 9.0.4+11]
  JAVA_HOME             : /usr/lib/jvm/java-9-oracle
  Random Testing Seed   : 1093B0D85566FF48


FAILURE: Build failed with an exception.

* What went wrong:
@Option 'tests' linked to multiple elements in class 'com.carrotsearch.gradle.junit4.RandomizedTestingTask_Decorated'.

* Try:
Run with --info or --debug option to get more log output. Run with --scan to get full insights.

* Exception is:
org.gradle.api.internal.tasks.options.OptionValidationException: @Option 'tests' linked to multiple elements in class 'com.carrotsearch.gradle.junit4.RandomizedTestingTask_Decorated'.
        at org.gradle.api.internal.tasks.options.OptionReader.loadClassDescriptorInCache(OptionReader.java:60)
        at org.gradle.api.internal.tasks.options.OptionReader.getOptions(OptionReader.java:45)
        at org.gradle.execution.commandline.CommandLineTaskConfigurer.configureTasksNow(CommandLineTaskConfigurer.java:51)
        at org.gradle.execution.commandline.CommandLineTaskConfigurer.configureTasks(CommandLineTaskConfigurer.java:44)
        at org.gradle.execution.commandline.CommandLineTaskParser.parseTasks(CommandLineTaskParser.java:44)
        at org.gradle.execution.TaskNameResolvingBuildConfigurationAction.configure(TaskNameResolvingBuildConfigurationAction.java:44)
        at org.gradle.execution.DefaultBuildConfigurationActionExecuter.configure(DefaultBuildConfigurationActionExecuter.java:48)
        at org.gradle.execution.DefaultBuildConfigurationActionExecuter.access$000(DefaultBuildConfigurationActionExecuter.java:25)
        at org.gradle.execution.DefaultBuildConfigurationActionExecuter$1.proceed(DefaultBuildConfigurationActionExecuter.java:54)
        at org.gradle.execution.DefaultTasksBuildExecutionAction.configure(DefaultTasksBuildExecutionAction.java:44)
        at org.gradle.execution.DefaultBuildConfigurationActionExecuter.configure(DefaultBuildConfigurationActionExecuter.java:48)
        at org.gradle.execution.DefaultBuildConfigurationActionExecuter.access$000(DefaultBuildConfigurationActionExecuter.java:25)
        at org.gradle.execution.DefaultBuildConfigurationActionExecuter$1.proceed(DefaultBuildConfigurationActionExecuter.java:54)
        at org.gradle.execution.ExcludedTaskFilteringBuildConfigurationAction.configure(ExcludedTaskFilteringBuildConfigurationAction.java:47)
        at org.gradle.execution.DefaultBuildConfigurationActionExecuter.configure(DefaultBuildConfigurationActionExecuter.java:48)
        at org.gradle.execution.DefaultBuildConfigurationActionExecuter.select(DefaultBuildConfigurationActionExecuter.java:36)
        at org.gradle.initialization.DefaultGradleLauncher$CalculateTaskGraph.run(DefaultGradleLauncher.java:268)
        at org.gradle.internal.progress.DefaultBuildOperationExecutor$RunnableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:336)
        at org.gradle.internal.progress.DefaultBuildOperationExecutor$RunnableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:328)
        at org.gradle.internal.progress.DefaultBuildOperationExecutor.execute(DefaultBuildOperationExecutor.java:199)
        at org.gradle.internal.progress.DefaultBuildOperationExecutor.run(DefaultBuildOperationExecutor.java:110)
        at org.gradle.initialization.DefaultGradleLauncher.constructTaskGraph(DefaultGradleLauncher.java:175)
        at org.gradle.initialization.DefaultGradleLauncher.doBuildStages(DefaultGradleLauncher.java:130)
        at org.gradle.initialization.DefaultGradleLauncher.executeTasks(DefaultGradleLauncher.java:109)
        at org.gradle.internal.invocation.GradleBuildController$1.call(GradleBuildController.java:78)
        at org.gradle.internal.invocation.GradleBuildController$1.call(GradleBuildController.java:75)
        at org.gradle.internal.work.DefaultWorkerLeaseService.withLocks(DefaultWorkerLeaseService.java:152)
        at org.gradle.internal.invocation.GradleBuildController.doBuild(GradleBuildController.java:100)
        at org.gradle.internal.invocation.GradleBuildController.run(GradleBuildController.java:75)
        at org.gradle.tooling.internal.provider.ExecuteBuildActionRunner.run(ExecuteBuildActionRunner.java:28)
        at org.gradle.launcher.exec.ChainingBuildActionRunner.run(ChainingBuildActionRunner.java:35)
        at org.gradle.tooling.internal.provider.ValidatingBuildActionRunner.run(ValidatingBuildActionRunner.java:32)
        at org.gradle.launcher.exec.RunAsBuildOperationBuildActionRunner$1.run(RunAsBuildOperationBuildActionRunner.java:43)
        at org.gradle.internal.progress.DefaultBuildOperationExecutor$RunnableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:336)
        at org.gradle.internal.progress.DefaultBuildOperationExecutor$RunnableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:328)
        at org.gradle.internal.progress.DefaultBuildOperationExecutor.execute(DefaultBuildOperationExecutor.java:199)
        at org.gradle.internal.progress.DefaultBuildOperationExecutor.run(DefaultBuildOperationExecutor.java:110)
        at org.gradle.launcher.exec.RunAsBuildOperationBuildActionRunner.run(RunAsBuildOperationBuildActionRunner.java:40)
        at org.gradle.tooling.internal.provider.SubscribableBuildActionRunner.run(SubscribableBuildActionRunner.java:51)
        at org.gradle.launcher.exec.InProcessBuildActionExecuter.execute(InProcessBuildActionExecuter.java:49)
        at org.gradle.launcher.exec.InProcessBuildActionExecuter.execute(InProcessBuildActionExecuter.java:32)
        at org.gradle.launcher.exec.BuildTreeScopeBuildActionExecuter.execute(BuildTreeScopeBuildActionExecuter.java:39)
        at org.gradle.launcher.exec.BuildTreeScopeBuildActionExecuter.execute(BuildTreeScopeBuildActionExecuter.java:25)
        at org.gradle.tooling.internal.provider.ContinuousBuildActionExecuter.execute(ContinuousBuildActionExecuter.java:80)
        at org.gradle.tooling.internal.provider.ContinuousBuildActionExecuter.execute(ContinuousBuildActionExecuter.java:53)
        at org.gradle.tooling.internal.provider.ServicesSetupBuildActionExecuter.execute(ServicesSetupBuildActionExecuter.java:57)
        at org.gradle.tooling.internal.provider.ServicesSetupBuildActionExecuter.execute(ServicesSetupBuildActionExecuter.java:32)
        at org.gradle.tooling.internal.provider.GradleThreadBuildActionExecuter.execute(GradleThreadBuildActionExecuter.java:36)
        at org.gradle.tooling.internal.provider.GradleThreadBuildActionExecuter.execute(GradleThreadBuildActionExecuter.java:25)
        at org.gradle.tooling.internal.provider.ParallelismConfigurationBuildActionExecuter.execute(ParallelismConfigurationBuildActionExecuter.java:43)
        at org.gradle.tooling.internal.provider.ParallelismConfigurationBuildActionExecuter.execute(ParallelismConfigurationBuildActionExecuter.java:29)
        at org.gradle.tooling.internal.provider.StartParamsValidatingActionExecuter.execute(StartParamsValidatingActionExecuter.java:64)
        at org.gradle.tooling.internal.provider.StartParamsValidatingActionExecuter.execute(StartParamsValidatingActionExecuter.java:29)
        at org.gradle.tooling.internal.provider.SessionFailureReportingActionExecuter.execute(SessionFailureReportingActionExecuter.java:59)
        at org.gradle.tooling.internal.provider.SessionFailureReportingActionExecuter.execute(SessionFailureReportingActionExecuter.java:44)
        at org.gradle.tooling.internal.provider.SetupLoggingActionExecuter.execute(SetupLoggingActionExecuter.java:45)
        at org.gradle.tooling.internal.provider.SetupLoggingActionExecuter.execute(SetupLoggingActionExecuter.java:30)
        at org.gradle.launcher.daemon.server.exec.ExecuteBuild.doBuild(ExecuteBuild.java:67)
        at org.gradle.launcher.daemon.server.exec.BuildCommandOnly.execute(BuildCommandOnly.java:36)
        at org.gradle.launcher.daemon.server.api.DaemonCommandExecution.proceed(DaemonCommandExecution.java:122)
        at org.gradle.launcher.daemon.server.exec.WatchForDisconnection.execute(WatchForDisconnection.java:37)
        at org.gradle.launcher.daemon.server.api.DaemonCommandExecution.proceed(DaemonCommandExecution.java:122)
        at org.gradle.launcher.daemon.server.exec.ResetDeprecationLogger.execute(ResetDeprecationLogger.java:26)
        at org.gradle.launcher.daemon.server.api.DaemonCommandExecution.proceed(DaemonCommandExecution.java:122)
        at org.gradle.launcher.daemon.server.exec.RequestStopIfSingleUsedDaemon.execute(RequestStopIfSingleUsedDaemon.java:34)
        at org.gradle.launcher.daemon.server.api.DaemonCommandExecution.proceed(DaemonCommandExecution.java:122)
        at org.gradle.launcher.daemon.server.exec.ForwardClientInput$2.call(ForwardClientInput.java:74)
        at org.gradle.launcher.daemon.server.exec.ForwardClientInput$2.call(ForwardClientInput.java:72)
        at org.gradle.util.Swapper.swap(Swapper.java:38)
        at org.gradle.launcher.daemon.server.exec.ForwardClientInput.execute(ForwardClientInput.java:72)
        at org.gradle.launcher.daemon.server.api.DaemonCommandExecution.proceed(DaemonCommandExecution.java:122)
        at org.gradle.launcher.daemon.server.exec.LogAndCheckHealth.execute(LogAndCheckHealth.java:50)
        at org.gradle.launcher.daemon.server.api.DaemonCommandExecution.proceed(DaemonCommandExecution.java:122)
        at org.gradle.launcher.daemon.server.exec.LogToClient.doBuild(LogToClient.java:62)
        at org.gradle.launcher.daemon.server.exec.BuildCommandOnly.execute(BuildCommandOnly.java:36)
        at org.gradle.launcher.daemon.server.api.DaemonCommandExecution.proceed(DaemonCommandExecution.java:122)
        at org.gradle.launcher.daemon.server.exec.EstablishBuildEnvironment.doBuild(EstablishBuildEnvironment.java:82)
        at org.gradle.launcher.daemon.server.exec.BuildCommandOnly.execute(BuildCommandOnly.java:36)
        at org.gradle.launcher.daemon.server.api.DaemonCommandExecution.proceed(DaemonCommandExecution.java:122)
        at org.gradle.launcher.daemon.server.exec.StartBuildOrRespondWithBusy$1.run(StartBuildOrRespondWithBusy.java:50)
        at org.gradle.launcher.daemon.server.DaemonStateCoordinator$1.run(DaemonStateCoordinator.java:295)
        at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:63)
        at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:46)
        at org.gradle.internal.concurrent.ThreadFactoryImpl$ManagedThreadRunnable.run(ThreadFactoryImpl.java:55)


* Get more help at https://help.gradle.org

Deprecated Gradle features were used in this build, making it incompatible with Gradle 5.0.
See https://docs.gradle.org/4.5/userguide/command_line_interface.html#sec:command_line_warnings

BUILD FAILED in 12s

Do you have any idea how we could fix this? Or work around this in some other way? If you want to see the changes I made to the build configuration, see the jacoco branch on my fork here.

[rjernst]

I originally tried to make RandomizedTestingTask extend Test, but there were many issues with it (I don't remember if they were exactly the same as you found here). I suggest calling jacoco in ant directly.

[bvobart]

Hmm okay. Personally, I believe that it should be a goal to have RandomizedTestingTask extend Test to minimize the chance of possible future issues with adding other instrumentation tools (e.g. SonarQube, mutation testing tools, etc.). However, If there is no viable or no better way of doing so, I'll look into calling JaCoCo from Ant directly then. Where would I be able to configure this though?

While it might be slightly off-topic, I do want to ask: how do you as main Elastic developers currently get insight in test coverage?

[rjernst]

I believe that it should be a goal to have RandomizedTestingTask extend Test

We have no control over that really. I don't think gradle has any desire to make Test extendable. As I said before, I did start there when creating RandomizedTestingTask.

If there is no viable or no better way of doing so, I'll look into calling JaCoCo from Ant directly then. Where would I be able to configure this though?

We should have a new task in buildSrc under org.elasticsearch.gradle.precommit which is then setup in PrecommitTasks.

how do you as main Elastic developers currently get insight in test coverage?

Currently, I run these in Intellij for specific test classes when I want to ensure something is covered.

[bvobart]

We have no control over that really. I don't think gradle has any desire to make Test extendable. As I said before, I did start there when creating RandomizedTestingTask.

Actually, Gradle's Test is extendable, as I've already been able to have RandomizedTestingTask extend Test and both compile and run. Strangely enough, the error I described in my previous comment somehow resolved itself over a week's time and turned into a different exception being thrown. I was able to hack in a simple solution omitting the exception (literally just catching it, printing "Hi" and continuing anyway), but since I'm not very experienced with the exact expected functionality of your build code, I have no idea what this may have broken behind the scenes (especially since it printed "Hi" 16 times). Most (not all) of the tests still ran and all succeeded, and JaCoCo folders did get made in each module's build directory, but nothing was put there. This indicates to me that it should be possible, but I think it does require a refactor / rewrite of RandomizedTestingTask by a member with sufficient experience so as to keep the required existing functionalities. Seeing as this is your greatest technical debt testing-wise, it can very well be worth it.

We should have a new task in buildSrc under org.elasticsearch.gradle.precommit which is then setup in PrecommitTasks.

I took a look here and there are several reasons why this either will not work, or why you should not want it there. The most important reason is that JaCoCo requires that the tests are run before it can generate its coverage reports, so you'd have to run the tests during the precommit tasks. Even then, I doubt whether you wouldn't run into the same issue with RandomizedTestingTask.

---

For the mean time, I'll create a PR updating the documentation on generating test coverage to reflect that it is currently only really possible through IntelliJ, linking to this issue.

[jasontedor]

Closed by #29255