Conflicting version of R8/D8 in Dexer/Desugar classpath

[mauriciogg]

Description of the bug:

We started seeing the following errors during dex merging when using D8

23:01:25 [Bazel] Error in bazel-out/darwin_x86_64-fastbuild/bin/apps/hermosa/matador-services/settings/dexsplits/hermosa-settings-service-debug/9.shard.zip:kotlinx/coroutines/rx3/RxSingleKt$$ExternalSyntheticLambda0.class.dex:
23:01:25 [Bazel] Attempt at compiling intermediate artifact without its context
23:01:25 [Bazel] Merge failed: Compilation failed to complete, origin: bazel-out/darwin_x86_64-fastbuild/bin/apps/hermosa/matador-services/settings/dexsplits/hermosa-settings-service-debug/9.shard.zip:kotlinx/coroutines/rx3/RxSingleKt$$ExternalSyntheticLambda0.class.dex
23:01:25 [Bazel] Exception in thread "main" java.lang.RuntimeException: Merge failed: Compilation failed to complete, origin: bazel-out/darwin_x86_64-fastbuild/bin/apps/hermosa/matador-services/settings/dexsplits/hermosa-settings-service-debug/9.shard.zip:kotlinx/coroutines/rx3/RxSingleKt$$ExternalSyntheticLambda0.class.dex
23:01:25 [Bazel] at com.google.devtools.build.android.r8.DexFileMerger.main(DexFileMerger.java:412)
23:01:25 [Bazel] Caused by: com.android.tools.r8.CompilationFailedException: Compilation failed to complete, origin: bazel-out/darwin_x86_64-fastbuild/bin/apps/hermosa/matador-services/settings/dexsplits/hermosa-settings-service-debug/9.shard.zip:kotlinx/coroutines/rx3/RxSingleKt$$ExternalSyntheticLambda0.class.dex
23:01:25 [Bazel] at Version.fakeStackEntry(Version_8.1.56.java:0)
23:01:25 [Bazel] at com.android.tools.r8.M.a(R8_8.1.56_756d1f50f618dd1c39c000f11defb367a21e9e866e3401b884be16c0950f6f79:5)
23:01:25 [Bazel] at com.android.tools.r8.utils.R0.a(R8_8.1.56_756d1f50f618dd1c39c000f11defb367a21e9e866e3401b884be16c0950f6f79:81)
23:01:25 [Bazel] at com.android.tools.r8.utils.R0.a(R8_8.1.56_756d1f50f618dd1c39c000f11defb367a21e9e866e3401b884be16c0950f6f79:32)
23:01:25 [Bazel] at com.android.tools.r8.utils.R0.a(R8_8.1.56_756d1f50f618dd1c39c000f11defb367a21e9e866e3401b884be16c0950f6f79:31)
23:01:25 [Bazel] at com.android.tools.r8.utils.R0.b(R8_8.1.56_756d1f50f618dd1c39c000f11defb367a21e9e866e3401b884be16c0950f6f79:2)
23:01:25 [Bazel] at com.android.tools.r8.D8.run(R8_8.1.56_756d1f50f618dd1c39c000f11defb367a21e9e866e3401b884be16c0950f6f79:6)
23:01:25 [Bazel] at com.google.devtools.build.android.r8.DexFileMerger.run(DexFileMerger.java:395)
23:01:25 [Bazel] at com.google.devtools.build.android.r8.DexFileMerger.main(DexFileMerger.java:409)
23:01:25 [Bazel] Caused by: com.android.tools.r8.utils.b: Attempt at compiling intermediate artifact without its context
23:01:25 [Bazel] at com.android.tools.r8.utils.O2.a(R8_8.1.56_756d1f50f618dd1c39c000f11defb367a21e9e866e3401b884be16c0950f6f79:21)
23:01:25 [Bazel] at com.android.tools.r8.utils.O2.error(R8_8.1.56_756d1f50f618dd1c39c000f11defb367a21e9e866e3401b884be16c0950f6f79:1)
23:01:25 [Bazel] at com.android.tools.r8.synthesis.D.a(R8_8.1.56_756d1f50f618dd1c39c000f11defb367a21e9e866e3401b884be16c0950f6f79:59)
23:01:25 [Bazel] at com.android.tools.r8.D8.d(R8_8.1.56_756d1f50f618dd1c39c000f11defb367a21e9e866e3401b884be16c0950f6f79:26)
23:01:25 [Bazel] at com.android.tools.r8.D8.a(R8_8.1.56_756d1f50f618dd1c39c000f11defb367a21e9e866e3401b884be16c0950f6f79:20)
23:01:25 [Bazel] at com.android.tools.r8.utils.R0.a(R8_8.1.56_756d1f50f618dd1c39c000f11defb367a21e9e866e3401b884be16c0950f6f79:28)
23:01:25 [Bazel] ... 4 more
23:01:25 [Bazel] Suppressed: java.lang.RuntimeException: com.android.tools.r8.utils.b: Attempt at compiling intermediate artifact without its context
23:01:25 [Bazel] at com.android.tools.r8.utils.O2.a(R8_8.1.56_756d1f50f618dd1c39c000f11defb367a21e9e866e3401b884be16c0950f6f79:29)
23:01:25 [Bazel] at com.android.tools.r8.dex.k.a(R8_8.1.56_756d1f50f618dd1c39c000f11defb367a21e9e866e3401b884be16c0950f6f79:225)
23:01:25 [Bazel] at com.android.tools.r8.D8.d(R8_8.1.56_756d1f50f618dd1c39c000f11defb367a21e9e866e3401b884be16c0950f6f79:149)
23:01:25 [Bazel] ... 6 more
23:01:25 [Bazel] Caused by: [CIRCULAR REFERENCE: com.android.tools.r8.utils.b: Attempt at compiling intermediate artifact without its context]

I did some investigation and the META-INF/synthetic-contexts.map in the .jar.dex.zip is incomplete. It looks under some circumstances an older version of D8 is being used during dexing. The version of R8 being used, is an older version which does not output information about synthetic classes and their contexts. The version that should be used should be the one pointed by android_gmaven_r8, but instead is the one bundled by the @maven_android//:com_android_tools_build_builder_file dep which was added as dependency in this commit bazelbuild/bazel@aaf94c6#diff-bdb5a620f62ba1e7228456cde29d508737679c9196143db3641b22bb3ca2fd2aR128. If com_android_tools_build_builde happens to appear before than the R8 version coming from android_gmave_r8 dep in the dexer classpath, then dexmerging is likely to fail due to missing synthetic classes info.
The r8 target shipped with the android tools depends on the gradle builder jar via //src/tools/android/java/com/google/devtools/build/android:all_android_tools which in turn transitively depends on the gradle jar. The gradle dependency needs to be teased out in order to avoid causing classpath conflicts.
The version used seems to be non-deterministic and depends on which commit we are building our app . Furthermore we've only encountered this bug in our linux builds and we have not been able to repro on Mac builds.

Which category does this issue belong to?

Android

What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

This bug is not deterministic and we are only able to repro at specific commits when building our app, but its easy to check the bundled version of R8 in the gradle jar. Furthermore we've only encountered this bug in our linux builds and we have not been able to repro on Mac builds.

Which operating system are you running Bazel on?

Linux

What is the output of bazel info release?

release 7.0.2.13 (internal release)

If bazel info release returns development version or (@non-git), tell us how you built Bazel.

Built from internal forked version of bazel. Forked at this commit from upstream bazelbuild/bazel@252d363

What's the output of git remote get-url origin; git rev-parse master; git rev-parse HEAD ?

No response

Is this a regression? If yes, please try to identify the Bazel commit where the bug was introduced.

bazelbuild/bazel@aaf94c6

Have you found anything relevant by searching the web?

No response

Any other information, logs, or outputs that you want to share?

No response

[nkoroste]

fyi @ahumesky

[ricowind]

I agree with the reporter here, we certainly have some issues with duplicated versions of R8 and that can easily cause these issues. Unsure why it is non deterministic.

This bug is not deterministic and we are only able to repro at specific commits when building our app, but its easy to check the bundled version of R8 in the gradle jar. Furthermore we've only encountered this bug in our linux builds and we have not been able to repro on Mac builds.

If I look at the R8 version bundled in that jar it does not match with the stack trace:

~$ java -cp ~/Downloads/builder-7.4.2.jar com.android.tools.r8.R8 --version
R8 4.0.52 (build e70cabb270a305828df5e7e2ab4c7c299a70ea2e from go/r8bot (luci-r8-custom-ci-bionic-31-0b08))

I don't think this version have the fixes Ian did for synthetics, but the stack trace also says differently:
at Version.fakeStackEntry(Version_8.1.56.java:0)
And (contrast to bazelbuild/bazel#16368 (comment) ) I can actually retrace that stack and get:

 Caused by: com.android.tools.r8.CompilationFailedException: Compilation failed to complete, origin: bazel-out/darwin_x86_64-fastbuild/bin/apps/hermosa/matador-services/settings/dexsplits/hermosa-settings-service-debug/9.shard.zip:kotlinx/coroutines/rx3/RxSingleKt$$ExternalSyntheticLambda0.class.dex
 at Version.fakeStackEntry(Version_8.1.56.java)
 at com.android.tools.r8.InternalCompilationFailedExceptionUtils.create(InternalCompilationFailedExceptionUtils.java:29)
 at com.android.tools.r8.utils.ExceptionUtils.failWithFakeEntry(ExceptionUtils.java:144)
 at com.android.tools.r8.utils.ExceptionUtils.failCompilation(ExceptionUtils.java:89)
 at com.android.tools.r8.utils.ExceptionUtils.withCompilationHandler(ExceptionUtils.java:83)
 at com.android.tools.r8.utils.ExceptionUtils.withD8CompilationHandler(ExceptionUtils.java:64)
 at com.android.tools.r8.D8.run(D8.java:97)
 at com.google.devtools.build.android.r8.DexFileMerger.run(DexFileMerger.java:395)
 at com.google.devtools.build.android.r8.DexFileMerger.main(DexFileMerger.java:409)
 Caused by: com.android.tools.r8.utils.AbortException: Attempt at compiling intermediate artifact without its context
 at com.android.tools.r8.utils.Reporter.handleDiagnostic(Reporter.java:81)
 at com.android.tools.r8.utils.Reporter.error(Reporter.java:111)
 at com.android.tools.r8.synthesis.SyntheticItems.collectSyntheticInputs(SyntheticItems.java:315)
 at com.android.tools.r8.D8.run(D8.java:219)
 at com.android.tools.r8.D8.lambda$run$0(D8.java:101)
 at com.android.tools.r8.utils.ExceptionUtils.withCompilationHandler(ExceptionUtils.java:80)
 ... 4 more
 Suppressed: java.lang.RuntimeException: com.android.tools.r8.utils.AbortException: Attempt at compiling intermediate artifact without its context
 at com.android.tools.r8.utils.Reporter.failIfPendingErrors(Reporter.java:136)
 at com.android.tools.r8.dex.ApplicationWriter.write(ApplicationWriter.java:434)
 at com.android.tools.r8.D8.run(D8.java:331)
 ... 6 more

[ricowind]

You can validate what version of R8 was used to build the shards by running:

 $ java -cp ~/Downloads/builder-7.4.2.jar com.android.tools.r8.ExtractMarker shard.jar

[nkoroste]

$ java -cp ~/Downloads/builder-7.4.2.jar com.android.tools.r8.ExtractMarker shard.jar

I managed to repro this as well. I just checked the version of the jar using your method and it does match the version (8.1.51) of the stack trace in my case.

[mauriciogg]

I've finally got a chance to look at this yesterday. My original assumptions were completely wrong. I've seen the classpath issue happen with gradle builds so I assumed this was the same issue. In reality, the issue here is that the workers cache for dexbuilder completely ignores the synthetic context info, so if an entry is cached it will be used the cached dex content but will be archived without the synthetic info. I've added a fix for that here bazelbuild/bazel#20411

[ahumesky]

Ah, tricky! Thanks for looking into this