Exception running bazel coverage on java tests #4398

robfig · 2018-01-08T14:27:36Z

Description of the problem / feature request:

"bazel coverage" produces an exception internal to bazel

$ bazel clean --expunge
INFO: Starting clean (this may take a while). Consider using --async if the clean takes more than several minutes.

$ bazel coverage --instrumentation_filter=//src/com/corp/profiles //test/com/corp/profiles:unit
...........
INFO: Analysed target //test/com/corp/profiles:unit (171 packages loaded).
INFO: Found 1 test target...
ERROR: /Users/robfig/alpha/src/com/corp/profileservice/validation2/error/BUILD:3:1: Building src/com/corp/profileservice/validation2/error/liberror.jar (3 source files) failed (Exit 1): java failed: error executing command
  (cd /private/var/tmp/_bazel_robfig/1db08896ecff7e5e96a745981c3b77e6/execroot/__main__ && \
  exec env - \
    LC_CTYPE=en_US.UTF-8 \
  external/local_jdk/bin/java -XX:+TieredCompilation '-XX:TieredStopAtLevel=1' -Xbootclasspath/p:external/bazel_tools/third_party/java/jdk/langtools/javac-9-dev-r4023-3.jar -jar external/bazel_tools/tools/jdk/JavaBuilder_deploy.jar @bazel-out/darwin-fastbuild/bin/src/com/corp/profileservice/validation2/error/liberror.jar-2.params)
java.nio.file.DirectoryNotEmptyException: bazel-out/darwin-fastbuild/coverage-metadata/com/corp/profileservice/completion
	at sun.nio.fs.UnixFileSystemProvider.implDelete(UnixFileSystemProvider.java:242)
	at sun.nio.fs.AbstractFileSystemProvider.delete(AbstractFileSystemProvider.java:103)
	at java.nio.file.Files.delete(Files.java:1126)
	at com.google.devtools.build.buildjar.instrumentation.JacocoInstrumentationProcessor$2.postVisitDirectory(JacocoInstrumentationProcessor.java:153)
	at com.google.devtools.build.buildjar.instrumentation.JacocoInstrumentationProcessor$2.postVisitDirectory(JacocoInstrumentationProcessor.java:143)
	at java.nio.file.Files.walkFileTree(Files.java:2688)
	at java.nio.file.Files.walkFileTree(Files.java:2742)
	at com.google.devtools.build.buildjar.instrumentation.JacocoInstrumentationProcessor.recursiveRemove(JacocoInstrumentationProcessor.java:141)
	at com.google.devtools.build.buildjar.instrumentation.JacocoInstrumentationProcessor.processRequest(JacocoInstrumentationProcessor.java:79)
	at com.google.devtools.build.buildjar.SimpleJavaLibraryBuilder.buildJar(SimpleJavaLibraryBuilder.java:140)
	at com.google.devtools.build.buildjar.SimpleJavaLibraryBuilder.run(SimpleJavaLibraryBuilder.java:118)
	at com.google.devtools.build.buildjar.BazelJavaBuilder.processRequest(BazelJavaBuilder.java:105)
	at com.google.devtools.build.buildjar.BazelJavaBuilder.runPersistentWorker(BazelJavaBuilder.java:67)
	at com.google.devtools.build.buildjar.BazelJavaBuilder.main(BazelJavaBuilder.java:45)
Target //test/com/corp/profiles:unit failed to build
INFO: Elapsed time: 111.911s, Critical Path: 39.00s
FAILED: Build did NOT complete successfully

Executed 0 out of 1 test: 1 fails to build.

$ bazel test //test/com/corp/profiles:unit
INFO: Analysed target //test/com/corp/profiles:unit (0 packages loaded).
INFO: Found 1 test target...
Target //test/com/corp/profiles:unit up-to-date:
  bazel-bin/test/com/corp/profiles/unit.jar
  bazel-bin/test/com/corp/profiles/unit
INFO: Elapsed time: 26.749s, Critical Path: 22.90s
INFO: Build completed successfully, 139 total actions
//test/com/corp/profiles:unit                                            PASSED in 5.5s

Executed 1 out of 1 test: 1 test passes.

The directory being complained about does not exist under output_path

$ bazel info
output_path: /private/var/tmp/_bazel_robfig/1db08896ecff7e5e96a745981c3b77e6/execroot/__main__/bazel-out

$ ls /private/var/tmp/_bazel_robfig/1db08896ecff7e5e96a745981c3b77e6/execroot/__main__/bazel-out/darwin-fastbuild/coverage-metadata/com/corp/profileservice/completion
ls: /private/var/tmp/_bazel_robfig/1db08896ecff7e5e96a745981c3b77e6/execroot/__main__/bazel-out/darwin-fastbuild/coverage-metadata/com/corp/profileservice/completion: No such file or directory

Feature requests: what underlying problem are you trying to solve with this feature?

Run java tests with coverage enabled

Bugs: what's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

I don't have an open source reproducer. I was hoping the stack trace would be enough

What operating system are you running Bazel on?

Mac OS High Sierra

What's the output of `bazel info release`?

release 0.9.0

Have you found anything relevant by searching the web?

This appears to be the main discussion about coverage, but it seems to indicate that java coverage is expected to be in working order
#1118

The text was updated successfully, but these errors were encountered:

robfig · 2018-01-11T20:44:00Z

Testing it on another package, I see a different exception

java.lang.NoClassDefFoundError: org/jacoco/agent/rt/internal_b0d6a23/Offline

Full output:

$ bazel clean --expunge
INFO: Starting clean (this may take a while). Consider using --async if the clean takes more than several minutes.

$ bazel coverage --instrumentation_filter=//src/com/corp/util/function //test/com/corp/util/function/...

.
INFO: Analysed 2 targets (20 packages loaded).
INFO: Found 1 target and 1 test target...
FAIL: //test/com/corp/util/function:unit (see /private/var/tmp/_bazel_robfig/1db08896ecff7e5e96a745981c3b77e6/execroot/__main__/bazel-out/darwin-fastbuild/testlogs/test/com/corp/util/function/unit/test.log)
INFO: From Testing //test/com/corp/util/function:unit:
==================== Test output for //test/com/corp/util/function:unit:
+ [[ -z bazel-out/darwin-fastbuild/bin/external/bazel_tools/tools/test/LcovMerger ]]
+ COVERAGE_LEGACY_MODE=
+ [[ -z bazel-out/darwin-fastbuild/bin/test/com/corp/util/function/unit.instrumented_files ]]
+ ROOT=/private/var/tmp/_bazel_robfig/1db08896ecff7e5e96a745981c3b77e6/bazel-sandbox/3966943955256125145/execroot/__main__
+ [[ bazel-out/darwin-fastbuild/bin/test/com/corp/util/function/unit.instrumented_files != /* ]]
+ export COVERAGE_MANIFEST=/private/var/tmp/_bazel_robfig/1db08896ecff7e5e96a745981c3b77e6/bazel-sandbox/3966943955256125145/execroot/__main__/bazel-out/darwin-fastbuild/bin/test/com/corp/util/function/unit.instrumented_files
+ COVERAGE_MANIFEST=/private/var/tmp/_bazel_robfig/1db08896ecff7e5e96a745981c3b77e6/bazel-sandbox/3966943955256125145/execroot/__main__/bazel-out/darwin-fastbuild/bin/test/com/corp/util/function/unit.instrumented_files
+ export COVERAGE_DIR=_coverage/test/com/corp/util/function/unit/test
+ COVERAGE_DIR=_coverage/test/com/corp/util/function/unit/test
+ [[ _coverage/test/com/corp/util/function/unit/test == /private/var/tmp/_bazel_robfig/1db08896ecff7e5e96a745981c3b77e6/bazel-sandbox/3966943955256125145/execroot/__main__* ]]
+ COVERAGE_DIR=/private/var/tmp/_bazel_robfig/1db08896ecff7e5e96a745981c3b77e6/bazel-sandbox/3966943955256125145/execroot/__main__/_coverage/test/com/corp/util/function/unit/test
+ mkdir -p /private/var/tmp/_bazel_robfig/1db08896ecff7e5e96a745981c3b77e6/bazel-sandbox/3966943955256125145/execroot/__main__/_coverage/test/com/corp/util/function/unit/test
+ COVERAGE_OUTPUT_FILE=bazel-out/darwin-fastbuild/testlogs/test/com/corp/util/function/unit/coverage.dat
+ [[ bazel-out/darwin-fastbuild/testlogs/test/com/corp/util/function/unit/coverage.dat == /private/var/tmp/_bazel_robfig/1db08896ecff7e5e96a745981c3b77e6/bazel-sandbox/3966943955256125145/execroot/__main__* ]]
+ COVERAGE_OUTPUT_FILE=/private/var/tmp/_bazel_robfig/1db08896ecff7e5e96a745981c3b77e6/bazel-sandbox/3966943955256125145/execroot/__main__/bazel-out/darwin-fastbuild/testlogs/test/com/corp/util/function/unit/coverage.dat
+ export JAVA_COVERAGE_FILE=/private/var/tmp/_bazel_robfig/1db08896ecff7e5e96a745981c3b77e6/bazel-sandbox/3966943955256125145/execroot/__main__/_coverage/test/com/corp/util/function/unit/test/jvcov.dat
+ JAVA_COVERAGE_FILE=/private/var/tmp/_bazel_robfig/1db08896ecff7e5e96a745981c3b77e6/bazel-sandbox/3966943955256125145/execroot/__main__/_coverage/test/com/corp/util/function/unit/test/jvcov.dat
+ export COVERAGE=1
+ COVERAGE=1
+ export BULK_COVERAGE_RUN=1
+ BULK_COVERAGE_RUN=1
+ [[ ! -n '' ]]
+ for name in '"$LCOV_MERGER"'
+ [[ ! -e bazel-out/darwin-fastbuild/bin/external/bazel_tools/tools/test/LcovMerger ]]
+ [[ -n '' ]]
+ cd /private/var/tmp/_bazel_robfig/1db08896ecff7e5e96a745981c3b77e6/bazel-sandbox/3966943955256125145/execroot/__main__/bazel-out/darwin-fastbuild/bin/test/com/corp/util/function/unit.runfiles/__main__
+ /private/var/tmp/_bazel_robfig/1db08896ecff7e5e96a745981c3b77e6/bazel-sandbox/3966943955256125145/execroot/__main__/bazel-out/darwin-fastbuild/bin/test/com/corp/util/function/unit.runfiles/__main__/test/com/corp/util/function/unit
JUnit4 Test Runner
.E.E.E.E.E.E
Time: 0.07
There were 6 failures:
1) test_mapping_nonEmpty(com.corp.util.function.OptionalBooleanTest$ExplicitTests)
java.lang.NoClassDefFoundError: org/jacoco/agent/rt/internal_b0d6a23/Offline
	at com.corp.util.function.OptionalBoolean.$jacocoInit(OptionalBoolean.java)
	at com.corp.util.function.OptionalBoolean.coerce(OptionalBoolean.java)
	at com.corp.util.function.OptionalBooleanTest$ExplicitTests.test_mapping_nonEmpty(OptionalBooleanTest.java:59)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
	at org.junit.runners.Suite.runChild(Suite.java:127)
	at org.junit.runners.Suite.runChild(Suite.java:26)
	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
	at org.junit.runners.Suite.runChild(Suite.java:127)
	at org.junit.runners.Suite.runChild(Suite.java:26)
	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
	at com.google.testing.junit.runner.internal.junit4.CancellableRequestFactory$CancellableRunner.run(CancellableRequestFactory.java:89)
	at org.junit.runner.JUnitCore.run(JUnitCore.java:160)
	at org.junit.runner.JUnitCore.run(JUnitCore.java:138)
	at com.google.testing.junit.runner.junit4.JUnit4Runner.run(JUnit4Runner.java:112)
	at com.google.testing.junit.runner.BazelTestRunner.runTestsInSuite(BazelTestRunner.java:144)
	at com.google.testing.junit.runner.BazelTestRunner.main(BazelTestRunner.java:82)
Caused by: java.lang.ClassNotFoundException: org.jacoco.agent.rt.internal_b0d6a23.Offline
	at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	... 42 more

(MORE SIMILAR FAILURES HERE)

FAILURES!!!
Tests run: 6,  Failures: 6


BazelTestRunner exiting with a return value of 1
JVM shutdown hooks (if any) will run now.
The JVM will exit once they complete.

-- JVM shutdown starting at 2018-01-11 15:41:08 --

+ TEST_STATUS=1
+ touch /private/var/tmp/_bazel_robfig/1db08896ecff7e5e96a745981c3b77e6/bazel-sandbox/3966943955256125145/execroot/__main__/bazel-out/darwin-fastbuild/testlogs/test/com/corp/util/function/unit/coverage.dat
+ [[ 1 -ne 0 ]]
+ echo --
--
+ echo Coverage runner: Not collecting coverage for failed test.
Coverage runner: Not collecting coverage for failed test.
+ echo The following commands failed with status 1
The following commands failed with status 1
+ echo /private/var/tmp/_bazel_robfig/1db08896ecff7e5e96a745981c3b77e6/bazel-sandbox/3966943955256125145/execroot/__main__/bazel-out/darwin-fastbuild/bin/test/com/corp/util/function/unit.runfiles/__main__/test/com/corp/util/function/unit
/private/var/tmp/_bazel_robfig/1db08896ecff7e5e96a745981c3b77e6/bazel-sandbox/3966943955256125145/execroot/__main__/bazel-out/darwin-fastbuild/bin/test/com/corp/util/function/unit.runfiles/__main__/test/com/corp/util/function/unit
+ exit 1
================================================================================
INFO: Elapsed time: 14.977s, Critical Path: 2.30s
INFO: Build completed, 1 test FAILED, 34 total actions
//test/com/corp/util/function:unit                                       FAILED in 0.6s
  /private/var/tmp/_bazel_robfig/1db08896ecff7e5e96a745981c3b77e6/execroot/__main__/bazel-out/darwin-fastbuild/testlogs/test/com/corp/util/function/unit/test.log

Executed 1 out of 1 test: 1 fails locally.

$ bazel test --instrumentation_filter=//src/com/corp/util/function //test/com/corp/util/function/...
INFO: Analysed 2 targets (0 packages loaded).
INFO: Found 1 target and 1 test target...
INFO: Elapsed time: 1.166s, Critical Path: 0.85s
INFO: Build completed successfully, 9 total actions
//test/com/corp/util/function:unit                                       PASSED in 0.5s

Executed 1 out of 1 test: 1 test passes.

robfig · 2018-01-17T15:18:06Z

The first exception does not occur if I pass "--jobs 1", so presumably this is a bug in the interaction with concurrent workers. If you have any guidance or links as to how to debug / develop a fix for it I'd be happy to try to put together a pull request. Looking at the second one..

robfig · 2018-01-24T04:04:49Z

Any feedback on this?

robfig · 2018-02-02T23:22:08Z

I tried with bazel 0.10.0 and these errors still occur in the same way

robfig · 2018-02-05T16:16:50Z

Running with --experimental_java_coverage solves the NoClassDefFoundError but not the java.nio.file.DirectoryNotEmptyException. At least this means I can run tests with --jobs 1 and have them succeed. Wonderful.

robfig · 2018-02-05T16:19:44Z

(Looking at it again, I figured out that the NoClassDefFoundError was caused by the java_test depending on a java_binary target (accidentally) instead of java_library. It began working after correcting that.)

iirina · 2018-02-07T10:22:17Z

Are there any more coverage errors you get now with --experimental_java_coverage?

robfig · 2018-02-07T12:29:25Z

Yes, the concurrency-related error still occurs as before when I run coverage on the ~25 test suite targets in our util directory. It does not occur with --jobs=1. Is that sufficient to diagnose?

iirina · 2018-02-08T10:35:14Z

Can you tell me more about your setup? How are your tests/libraries structured?

robfig · 2018-02-08T13:05:33Z

I have about 30 test packages under util, with a representative illustration of one in
https://github.com/robfig/bazel_coverage_repro

and I'm running them like

$ bazel coverage --experimental_java_coverage
   --instrumentation_filter=//src/com/corp/util
   test/com/corp/util/...

which passes (as expected) because it's only one test. I believe if I created 30 (or at least a number of) different packages it would reproduce.

Here is a latest example of failure, where one failed to build, and the rest were skipped:

ERROR: /Users/robfig/alpha/src/com/corp/util/lock/BUILD:3:1: Building src/com/corp/util/lock/liblock.jar (6 source files) failed (Exit 1): java failed: error executing command
  (cd /private/var/tmp/_bazel_robfig/1db08896ecff7e5e96a745981c3b77e6/execroot/__main__ && \
  exec env - \
    LC_CTYPE=en_US.UTF-8 \
  external/local_jdk/bin/java -Xbootclasspath/p:external/bazel_tools/third_party/java/jdk/langtools/javac-9-dev-r4023-3.jar -jar external/bazel_tools/tools/jdk/JavaBuilder_deploy.jar @bazel-out/darwin-fastbuild/bin/src/com/corp/util/lock/liblock.jar-2.params)
java.nio.file.NoSuchFileException: bazel-out/darwin-fastbuild/coverage-metadata/com/corp/util/time/Timestamps.class.uninstrumented
	at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
	at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
	at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
	at sun.nio.fs.UnixFileSystemProvider.implDelete(UnixFileSystemProvider.java:244)
	at sun.nio.fs.AbstractFileSystemProvider.delete(AbstractFileSystemProvider.java:103)
	at java.nio.file.Files.delete(Files.java:1126)
	at com.google.devtools.build.buildjar.instrumentation.JacocoInstrumentationProcessor$2.visitFile(JacocoInstrumentationProcessor.java:147)
	at com.google.devtools.build.buildjar.instrumentation.JacocoInstrumentationProcessor$2.visitFile(JacocoInstrumentationProcessor.java:143)
	at java.nio.file.Files.walkFileTree(Files.java:2670)
	at java.nio.file.Files.walkFileTree(Files.java:2742)
	at com.google.devtools.build.buildjar.instrumentation.JacocoInstrumentationProcessor.recursiveRemove(JacocoInstrumentationProcessor.java:141)
	at com.google.devtools.build.buildjar.instrumentation.JacocoInstrumentationProcessor.processRequest(JacocoInstrumentationProcessor.java:79)
	at com.google.devtools.build.buildjar.SimpleJavaLibraryBuilder.buildJar(SimpleJavaLibraryBuilder.java:157)
	at com.google.devtools.build.buildjar.SimpleJavaLibraryBuilder.run(SimpleJavaLibraryBuilder.java:134)
	at com.google.devtools.build.buildjar.BazelJavaBuilder.processRequest(BazelJavaBuilder.java:105)
	at com.google.devtools.build.buildjar.BazelJavaBuilder.runPersistentWorker(BazelJavaBuilder.java:67)
	at com.google.devtools.build.buildjar.BazelJavaBuilder.main(BazelJavaBuilder.java:45)
INFO: Elapsed time: 77.067s, Critical Path: 46.09s
FAILED: Build did NOT complete successfully
//test/com/corp/util/function:unit                              (cached) PASSED in 4.5s
  /private/var/tmp/_bazel_robfig/1db08896ecff7e5e96a745981c3b77e6/execroot/__main__/bazel-out/darwin-fastbuild/testlogs/test/com/corp/util/function/unit/coverage.dat
//test/com/corp/util/addressutil:addressutil                          NO STATUS
//test/com/corp/util/appspartners:integration                         NO STATUS
//test/com/corp/util/base:unit                                        NO STATUS
//test/com/corp/util/categories:unit                                  NO STATUS
//test/com/corp/util/crypto:unit                                      NO STATUS
//test/com/corp/util/csv:unit                                         NO STATUS
//test/com/corp/util/elasticsearch:unit                               NO STATUS
//test/com/corp/util/flags:unit                                       NO STATUS
//test/com/corp/util/fuse:unit                                        NO STATUS
//test/com/corp/util/geo/country:unit                                 NO STATUS
//test/com/corp/util/geo/geocoding:unit                               NO STATUS
//test/com/corp/util/geo/geocoding/providers:unit                     NO STATUS
//test/com/corp/util/geo/ip:unit                                      NO STATUS
//test/com/corp/util/hash:unit                                        NO STATUS
//test/com/corp/util/http:unit                                        NO STATUS
//test/com/corp/util/i18n:i18n                                        NO STATUS
//test/com/corp/util/objectcache:unit                                 NO STATUS
//test/com/corp/util/protobuf:unit                                    NO STATUS
//test/com/corp/util/retry:unit                                       NO STATUS
//test/com/corp/util/serialization/formenc:unit                       NO STATUS
//test/com/corp/util/serialization/js:unit                            NO STATUS
//test/com/corp/util/serialization/xml:unit                           NO STATUS
//test/com/corp/util/sql:unit                                         NO STATUS
//test/com/corp/util/subscriptionfeatures:integration_bazel           NO STATUS
//test/com/corp/util/text:unit                                        NO STATUS
//test/com/corp/util/time:unit                                        NO STATUS
//test/com/corp/util/uaparser:uaparser_unit                           NO STATUS
//test/com/corp/util/youtube:unit                                     NO STATUS

Executed 0 out of 30 tests: 1 test passes, 1 fails to build and 28 were skipped.

Based on this error:

It was building src/com/corp/util/lock
As part of that, it was deleting the bazel-out/darwin-fastbuild/coverage-metadata directory
com/corp/util/time/Timestamps.uninstrumented was under that directory, and it was deleted by a different worker between the time the file system walker listed the directory and the time it tried to delete that file.
Note that com.corp.util.time IS NOT a dependency of com.corp.util.lock, so it's clearly incorrect for the worker doing coverage of util.lock to delete files used by the util.time worker.

This is supported by the fact that running the coverage using --jobs=1 causes it to work.

Does that seem plausible? Or is it something in our setup / bazel.rc that is causing the test suites to get mixed up?

iirina · 2018-02-08T13:20:53Z

Yes, it's totally plausible. I'll try to reproduce. Thanks for the info!

iirina · 2018-02-08T15:03:19Z

I managed to get a couple of errors that seem related with the bazel java tests. Both commands are successful if used with --jobs=1.

bazel coverage src/test/java/com/google/devtools/build/lib:java-rules-tests

ERROR: /Users/elenairina/bazel/src/main/java/com/google/devtools/build/lib/vfs/BUILD:34:1: Building src/main/java/com/google/devtools/build/lib/vfs/libvfs.jar (26 source files) and running annotation processors (AutoCodecProcessor, OptionProcessor) failed (Exit 1)
java.nio.file.NoSuchFileException: bazel-out/darwin-fastbuild/bin/src/main/java/com/google/devtools/build/lib/vfs/_javac/vfs/libvfs_classes/com/google/devtools/build/lib/vfs/Root$PathRoot.class -> bazel-out/darwin-fastbuild/coverage-metadata/com/google/devtools/build/lib/vfs/Root$PathRoot.class
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
	at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
	at sun.nio.fs.UnixCopyFile.move(UnixCopyFile.java:457)
	at sun.nio.fs.UnixFileSystemProvider.move(UnixFileSystemProvider.java:262)
	at java.nio.file.Files.move(Files.java:1395)
	at com.google.devtools.build.buildjar.instrumentation.JacocoInstrumentationProcessor$1.visitFile(JacocoInstrumentationProcessor.java:127)
	at com.google.devtools.build.buildjar.instrumentation.JacocoInstrumentationProcessor$1.visitFile(JacocoInstrumentationProcessor.java:104)
	at java.nio.file.Files.walkFileTree(Files.java:2670)
	at java.nio.file.Files.walkFileTree(Files.java:2742)
	at com.google.devtools.build.buildjar.instrumentation.JacocoInstrumentationProcessor.instrumentRecursively(JacocoInstrumentationProcessor.java:102)
	at com.google.devtools.build.buildjar.instrumentation.JacocoInstrumentationProcessor.processRequest(JacocoInstrumentationProcessor.java:89)
	at com.google.devtools.build.buildjar.SimpleJavaLibraryBuilder.buildJar(SimpleJavaLibraryBuilder.java:157)
	at com.google.devtools.build.buildjar.SimpleJavaLibraryBuilder.run(SimpleJavaLibraryBuilder.java:134)
	at com.google.devtools.build.buildjar.BazelJavaBuilder.processRequest(BazelJavaBuilder.java:105)
	at com.google.devtools.build.buildjar.BazelJavaBuilder.runPersistentWorker(BazelJavaBuilder.java:67)
	at com.google.devtools.build.buildjar.BazelJavaBuilder.main(BazelJavaBuilder.java:45)
Target //src/test/java/com/google/devtools/build/lib:java-rules-tests failed to build

bazel coverage --experimental_java_coverage src/test/java/com/google/devtools/build/lib:java-rules-tests :

ERROR: /Users/elenairina/bazel/src/main/java/com/google/devtools/build/lib/windows/jni/BUILD:49:1: Building src/main/java/com/google/devtools/build/lib/windows/jni/libprocesses.jar (1 source file) failed (Exit 1)
java.io.FileNotFoundException: /private/var/tmp/_bazel_elenairina/f2f47067c84d3ffa2025ad2073174c18/execroot/io_bazel/bazel-out/darwin-fastbuild/coverage-metadata/com/google/devtools/build/lib/windows (No such file or directory)
	at com.google.devtools.build.buildjar.jarhelper.JarHelper.copyEntry(JarHelper.java:176)
	at com.google.devtools.build.buildjar.jarhelper.JarCreator.execute(JarCreator.java:234)
	at com.google.devtools.build.buildjar.SimpleJavaLibraryBuilder.buildJar(SimpleJavaLibraryBuilder.java:160)
	at com.google.devtools.build.buildjar.SimpleJavaLibraryBuilder.run(SimpleJavaLibraryBuilder.java:134)
	at com.google.devtools.build.buildjar.BazelJavaBuilder.processRequest(BazelJavaBuilder.java:105)
	at com.google.devtools.build.buildjar.BazelJavaBuilder.runPersistentWorker(BazelJavaBuilder.java:67)
	at com.google.devtools.build.buildjar.BazelJavaBuilder.main(BazelJavaBuilder.java:45)
Target //src/test/java/com/google/devtools/build/lib:java-rules-tests failed to build

robfig · 2018-03-22T20:53:09Z

Can I ask why this is not a higher priority? It seems like sort of a big issue in running coverage, and I was under the impression bazel considered java coverage to be production ready. For example, we have to run all of our coverage with j=1, which is a LOT slower than when running the equivalent bazel test at full parallelism. This results in none of the teams wanting to use it.

Or, is there a workaround that I'm not seeing? How do others run coverage without specifying j=1?

dhalperi · 2018-05-20T08:55:37Z

Still happening on 0.13.0

robfig · 2018-06-01T21:42:40Z

Still happening on 0.14.0

@iirina , since I'm not sure you saw my previous question, can you advise why it's a low priority when it seems like a major blocker for calling java coverage "production ready"? How do folks in the bazel project run coverage? I can't imagine you pass --jobs 1

iirina · 2018-06-05T14:35:42Z

Hi Rob,

I'm sorry for the silence and the terrible state of coverage. A bit of coverage context: Bazel rolled-out coverage in a poor, minimalist way and unfortunately it was not production ready (see #1118). This quarter we had the resources to prioritize working on coverage and we now have a design doc for improving it. Also unfortunately, Bazel doesn't run coverage for itself ATM. That will change (see one of the requirements in the doc). All in all, the situation is bad and we are now trying to make it better.

I'm working on a fix for this issue right now. Given that the first RC was already cut for Bazel 0.15.0 already started, the fix should be in 0.16.0.

robfig · 2018-06-05T15:52:18Z

Hi Irina,
Thanks for the response! The design doc looks very good, and I'm excited at the progress. This particular issue preventing us from being able to reliably run bazel coverage is the main pain point in the current situation, and we can migrate to the new format when it's ready. I'm very pleased to hear that a fix is being worked on, and I'll probably cut a personal release as soon as the fix goes in.
Many thanks,
Rob

iirina · 2018-06-08T12:46:51Z

@robfig The fix is now in. It would be great if you could cut a RC that includes 4001ddd and see if the issue still persists. I tested it with bazel coverage src/test/java/com/google/devtools/build/lib:java-rules-tests and didn't get breakages anymore.

robfig · 2018-06-10T15:27:22Z

@iirina The concurrency-related error has indeed gone away! That's a big step and will fix the vast majority of our issues. Thank you very much!

I do still see the other error from the original report. It seems to be triggered by instrumentation of an error-prone check.

Here's the scenario:

# //src/com/corp/util/i18n/BUILD
java_library(
    name = "i18n",
    srcs = glob(["*.java"]),
    exported_plugins = [
        "//src/com/corp/util/i18n/bugpatterns",
    ],
   ...

and

# //src/com/corp/util/i18n/bugpatterns/BUILD
java_plugin(
    name = "bugpatterns",
    srcs = glob(["*.java"]),
   ...

Here's the full error:

$ ../bazel/bazel-bin/src/bazel coverage test/com/corp/util/...

ERROR: /Users/robfig/alpha/src/com/corp/util/callable/BUILD:3:1: Building src/com/corp/util/callable/libcallable.jar (1 source file) failed (Exit 1): java failed: error executing command
  (cd /private/var/tmp/_bazel_robfig/1db08896ecff7e5e96a745981c3b77e6/execroot/corp && \
  exec env - \
    LC_CTYPE=en_US.UTF-8 \
  external/local_jdk/bin/java -Xbootclasspath/p:external/bazel_tools/third_party/java/jdk/langtools/javac-9+181-r4173-1.jar -jar external/bazel_tools/tools/jdk/JavaBuilder_deploy.jar @bazel-out/darwin-fastbuild/bin/src/com/corp/util/callable/libcallable.jar-2.params)
An exception has occurred in the compiler ((version info not available)). Please file a bug against the Java compiler via the Java bug reporting page (http://bugreport.java.com) after checking the Bug Database (http://bugs.java.com) for duplicates. Include your program and the following diagnostic in your report. Thank you.
java.util.ServiceConfigurationError: com.google.errorprone.bugpatterns.BugChecker: Provider com.corp.util.i18n.bugpatterns.I18nNonConstant could not be instantiated
	at java.util.ServiceLoader.fail(ServiceLoader.java:232)
	at java.util.ServiceLoader.access$100(ServiceLoader.java:185)
	at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:384)
	at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404)
	at java.util.ServiceLoader$1.next(ServiceLoader.java:480)
	at com.google.common.collect.TransformedIterator.next(TransformedIterator.java:47)
	at com.google.errorprone.scanner.ScannerSupplier.fromBugCheckerClasses(ScannerSupplier.java:71)
	at com.google.errorprone.ErrorPronePlugins.loadPlugins(ErrorPronePlugins.java:47)
	at com.google.errorprone.ErrorProneAnalyzer.lambda$scansPlugins$0(ErrorProneAnalyzer.java:76)
	at com.google.common.base.Suppliers$NonSerializableMemoizingSupplier.get(Suppliers.java:164)
	at com.google.errorprone.ErrorProneAnalyzer.finished(ErrorProneAnalyzer.java:152)
	at com.google.devtools.build.buildjar.javac.plugins.errorprone.ErrorPronePlugin.postFlow(ErrorPronePlugin.java:110)
	at com.google.devtools.build.buildjar.javac.BlazeJavaCompiler.flow(BlazeJavaCompiler.java:112)
	at com.sun.tools.javac.main.JavaCompiler.flow(JavaCompiler.java:1363)
	at com.sun.tools.javac.main.JavaCompiler.compile(JavaCompiler.java:959)
	at com.sun.tools.javac.api.JavacTaskImpl.lambda$doCall$0(JavacTaskImpl.java:100)
	at com.sun.tools.javac.api.JavacTaskImpl.handleExceptions(JavacTaskImpl.java:142)
	at com.sun.tools.javac.api.JavacTaskImpl.doCall(JavacTaskImpl.java:96)
	at com.sun.tools.javac.api.JavacTaskImpl.call(JavacTaskImpl.java:90)
	at com.google.devtools.build.buildjar.javac.BlazeJavacMain.compile(BlazeJavacMain.java:110)
	at com.google.devtools.build.buildjar.SimpleJavaLibraryBuilder$2.invokeJavac(SimpleJavaLibraryBuilder.java:121)
	at com.google.devtools.build.buildjar.ReducedClasspathJavaLibraryBuilder.compileSources(ReducedClasspathJavaLibraryBuilder.java:54)
	at com.google.devtools.build.buildjar.SimpleJavaLibraryBuilder.compileJavaLibrary(SimpleJavaLibraryBuilder.java:124)
	at com.google.devtools.build.buildjar.SimpleJavaLibraryBuilder.run(SimpleJavaLibraryBuilder.java:132)
	at com.google.devtools.build.buildjar.BazelJavaBuilder.processRequest(BazelJavaBuilder.java:105)
	at com.google.devtools.build.buildjar.BazelJavaBuilder.runPersistentWorker(BazelJavaBuilder.java:67)
	at com.google.devtools.build.buildjar.BazelJavaBuilder.main(BazelJavaBuilder.java:45)
Caused by: java.lang.NoClassDefFoundError: org/jacoco/agent/rt/internal_b0d6a23/Offline
	at com.corp.util.i18n.bugpatterns.I18nNonConstant.$jacocoInit(I18nNonConstant.java)
	at com.corp.util.i18n.bugpatterns.I18nNonConstant.<clinit>(I18nNonConstant.java)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at java.lang.Class.newInstance(Class.java:442)
	at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:380)
	... 24 more
Caused by: java.lang.ClassNotFoundException: org.jacoco.agent.rt.internal_b0d6a23.Offline
	at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	... 32 more
INFO: Elapsed time: 18.145s, Critical Path: 8.44s
INFO: 14 processes: 10 darwin-sandbox, 1 local, 3 worker.
FAILED: Build did NOT complete successfully

It seems like coverage instrumentation should ignore java_plugin targets?

iirina · 2018-06-11T06:22:19Z

@iirina The concurrency-related error has indeed gone away! That's a big step and will the vast majority of our issues. Thank you very much!

Yay!

I do still see the other error from the original report. It seems to be triggered by instrumentation of an error-prone check.

Can you open a new issue for this bug and cc me on it? In the meantime could you also try running coverage with --experimental_java_coverage and tell me if you get the same error? Thanks!

robfig · 2018-06-19T02:16:32Z

Sadly, I do get the same error. Opened #5426, thank you!

for each test instead of the same directory for all the tests. The previous implementation was using one directory for instrumenting the classes of a jar. For each each jar the metadata directory was deleted if it already existed. This is problematic for local execution when multiple tests are run in parallel because some threads will try to delete the directory and some will try to perform read/write operations on it. This is an important fix for Bazel coverage users. Fixes bazelbuild#4398. RELNOTES: Java coverage works now with multiple jobs. PiperOrigin-RevId: 199764483

jin added type: bug category: rules > java labels Jan 9, 2018

jin assigned iirina Jan 9, 2018

iirina added the P2 We'll consider working on this in future. (Assignee optional) label Feb 8, 2018

iirina added P1 I'll work on this now. (Assignee required) and removed P2 We'll consider working on this in future. (Assignee optional) labels Jun 5, 2018

dhalperi mentioned this issue Jun 5, 2018

Java code coverage for sh_tests missing #5331

Closed

iirina added the coverage label Jun 6, 2018

bazel-io closed this as completed in 4001ddd Jun 8, 2018

robfig mentioned this issue Jun 19, 2018

Compiler exception running coverage: ClassNotFoundException: org.jacoco.agent.rt.internal_b0d6a23.Offline #5426

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Exception running bazel coverage on java tests #4398

Exception running bazel coverage on java tests #4398

robfig commented Jan 8, 2018 •

edited

Loading

robfig commented Jan 11, 2018 •

edited

Loading

robfig commented Jan 17, 2018

robfig commented Jan 24, 2018

robfig commented Feb 2, 2018

robfig commented Feb 5, 2018

robfig commented Feb 5, 2018

iirina commented Feb 7, 2018

robfig commented Feb 7, 2018

iirina commented Feb 8, 2018

robfig commented Feb 8, 2018 •

edited

Loading

iirina commented Feb 8, 2018

iirina commented Feb 8, 2018

robfig commented Mar 22, 2018

dhalperi commented May 20, 2018

robfig commented Jun 1, 2018

iirina commented Jun 5, 2018

robfig commented Jun 5, 2018

iirina commented Jun 8, 2018

robfig commented Jun 10, 2018 •

edited

Loading

iirina commented Jun 11, 2018

robfig commented Jun 19, 2018

Exception running bazel coverage on java tests #4398

Exception running bazel coverage on java tests #4398

Comments

robfig commented Jan 8, 2018 • edited Loading

Description of the problem / feature request:

Feature requests: what underlying problem are you trying to solve with this feature?

Bugs: what's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

What operating system are you running Bazel on?

What's the output of bazel info release?

Have you found anything relevant by searching the web?

robfig commented Jan 11, 2018 • edited Loading

robfig commented Jan 17, 2018

robfig commented Jan 24, 2018

robfig commented Feb 2, 2018

robfig commented Feb 5, 2018

robfig commented Feb 5, 2018

iirina commented Feb 7, 2018

robfig commented Feb 7, 2018

iirina commented Feb 8, 2018

robfig commented Feb 8, 2018 • edited Loading

iirina commented Feb 8, 2018

iirina commented Feb 8, 2018

robfig commented Mar 22, 2018

dhalperi commented May 20, 2018

robfig commented Jun 1, 2018

iirina commented Jun 5, 2018

robfig commented Jun 5, 2018

iirina commented Jun 8, 2018

robfig commented Jun 10, 2018 • edited Loading

iirina commented Jun 11, 2018

robfig commented Jun 19, 2018

robfig commented Jan 8, 2018 •

edited

Loading

What's the output of `bazel info release`?

robfig commented Jan 11, 2018 •

edited

Loading

robfig commented Feb 8, 2018 •

edited

Loading

robfig commented Jun 10, 2018 •

edited

Loading