Bazel HEAD OOMs during analysis phase when JavaPluginInfo provides Java outputs information

[Bencodes]

Description of the problem / feature request:

Bazel HEAD runs out of memory during the analysis phase of a build.

I did a git bisect tracking the issue back to this commit f73e28c as the culprit.

The stacktrace that eventually comes out:

(19:11:00) FATAL: bazel ran out of memory and crashed. Printing stack trace:
net.starlark.java.eval.Starlark$UncheckedEvalError: OutOfMemoryError thrown during Starlark evaluation (//instant-features/rider/last-mile/fieldwork/stationdetail/plugins:lib_kt)
        at <starlark>.compile(<builtin>:0)
        at <starlark>._run_kt_java_builder_actions(/private/var/tmp/_bazel_blee/499a001013731d09bffd82f8601a3161/external/io_bazel_rules_kotlin/kotlin/internal/jvm/compile.bzl:780)
        at <starlark>.kt_jvm_produce_jar_actions(/private/var/tmp/_bazel_blee/499a001013731d09bffd82f8601a3161/external/io_bazel_rules_kotlin/kotlin/internal/jvm/compile.bzl:517)
        at <starlark>.kt_jvm_library_impl(/private/var/tmp/_bazel_blee/499a001013731d09bffd82f8601a3161/external/io_bazel_rules_kotlin/kotlin/internal/jvm/impl.bzl:235)
Caused by: java.lang.OutOfMemoryError: Java heap space
        at java.base/java.util.Arrays.copyOf(Unknown Source)
        at com.google.common.collect.ImmutableList.asImmutableList(ImmutableList.java:371)
        at com.google.common.collect.ImmutableList$Builder.build(ImmutableList.java:871)
        at com.google.devtools.build.lib.rules.java.JavaPluginInfo.merge(JavaPluginInfo.java:179)
        at com.google.devtools.build.lib.rules.java.JavaInfoBuildHelper.mergeExportedJavaPluginInfo(JavaInfoBuildHelper.java:238)
        at com.google.devtools.build.lib.rules.java.JavaInfoBuildHelper.createJavaCompileAction(JavaInfoBuildHelper.java:298)
        at com.google.devtools.build.lib.rules.java.JavaStarlarkCommon.createJavaCompileAction(JavaStarlarkCommon.java:118)
        at jdk.internal.reflect.GeneratedMethodAccessor163.invoke(Unknown Source)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
        at java.base/java.lang.reflect.Method.invoke(Unknown Source)
        at net.starlark.java.eval.MethodDescriptor.call(MethodDescriptor.java:162)
        at net.starlark.java.eval.BuiltinFunction.fastcall(BuiltinFunction.java:77)
        at net.starlark.java.eval.Starlark.fastcall(Starlark.java:619)
        at net.starlark.java.eval.Eval.evalCall(Eval.java:672)
        at net.starlark.java.eval.Eval.eval(Eval.java:489)
        at net.starlark.java.eval.Eval.execAssignment(Eval.java:109)
        at net.starlark.java.eval.Eval.exec(Eval.java:268)
        at net.starlark.java.eval.Eval.execStatements(Eval.java:82)
        at net.starlark.java.eval.Eval.execIf(Eval.java:205)
        at net.starlark.java.eval.Eval.exec(Eval.java:283)
        at net.starlark.java.eval.Eval.execStatements(Eval.java:82)
        at net.starlark.java.eval.Eval.execFunctionBody(Eval.java:66)
        at net.starlark.java.eval.StarlarkFunction.fastcall(StarlarkFunction.java:191)
        at net.starlark.java.eval.Starlark.fastcall(Starlark.java:619)
        at net.starlark.java.eval.Eval.evalCall(Eval.java:672)
        at net.starlark.java.eval.Eval.eval(Eval.java:489)
        at net.starlark.java.eval.Eval.execAssignment(Eval.java:109)
        at net.starlark.java.eval.Eval.exec(Eval.java:268)
        at net.starlark.java.eval.Eval.execStatements(Eval.java:82)
        at net.starlark.java.eval.Eval.execIf(Eval.java:205)
        at net.starlark.java.eval.Eval.exec(Eval.java:283)
        at net.starlark.java.eval.Eval.execStatements(Eval.java:82)

Bugs: what's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

Create a large java project that has lots of java plugins attached to each target. I'm able to repro with one of our smaller apps which has about 583 java targets, most of which have annotation processors.

[Bencodes]

@comius FYI I think you added this commit.

[comius]

I would need a reproducer for this, can you provide one?

It seems though rules_kotlin are at fault, what version are you using? Does changing the rules_kotlin version make things better/worse?

Is there something useful in the heapdump? You're looking for a large number of JavaPluginInfo/JavaOutputs. To get a heap dump increase memory using --host_jvm_args=-Xmx18g and run jmap -histo:live $BAZEL_PID.

Ideally I would compare a heapdump before and after the commit f73e28c.

[Bencodes]

It seems though rules_kotlin are at fault, what version are you using? Does changing the rules_kotlin version make things better/worse?

We are using an up to date version of rules Kotlin that has the JavaPluginInfo patches in it.

Is there something useful in the heapdump? You're looking for a large number of JavaPluginInfo/JavaOutputs. To get a heap dump increase memory using --host_jvm_args=-Xmx18g and run jmap -histo:live $BAZEL_PID.

I'll work on getting a heap dump.

[Bencodes]

Heap for 5.0.0 RC2 jmapd-5.0.0-RC2.txt
Heap for 5.0.0 RC2 with f73e28c reverted jmapd-5.0.0-RC2-with-revert.txt

[Bencodes]

The stack trace is a bit off compared to rules_kotlin HEAD. The call at io_bazel_rules_kotlin/kotlin/internal/jvm/compile.bzl:780 is the call to java_info = java_common.compile(...)

[meisterT]

Cc @Wyverald

[comius]

Comparison of heap dumps show insane increase in object arrays and some increase in integer arrays:

 objsize  chg   instances       space KB    class name
------------------------------------------------------
      99 +20426     -435368    26237030 KB    [Ljava.lang.Object;
      44 +1161      -85749       71739 KB    [I

merge call using a list for javaOutputs looks to be at fault here:
f73e28c#diff-f5262116f466c897bb45cfb13277e052945b108e22245b50b1d98cc6f9d3f398R172

[comius]

Can you check if #14288 fixes it? Perhaps provide new heapdump.

[Bencodes]

The heap looks much better with that commit pulled in. jmapd-patched.txt

The heap for a larger app jmapd-patched-larger-app.txt

[comius]

Comparing jmap reverted and patched, there is even 34MB improvement.

[Wyverald]

Thanks for flagging @meisterT. Let's get the fix cherrypicked into the rc2 branch.