Windows, Java: non-fatal Java header compilation error while building Bazel

[laszlocsomor]

Description of the problem / feature request / question:

Java worker fails with "The filename or extension is too long." error, though this doesn't fail the whole build. The path is actually well below 259 characters, so the error message seems to be misguided.

INFO: From Compiling Java headers src/main/java/com/google/devtools/build/lib/libbuild-base-hjar.jar (441 files):
Action failed to execute: java.io.IOException: ERROR: src/main/native/windows/processes-jni.cc(410): nativeCreateProcess(D:\bazel_build\execroot\io_bazel\external\local_jdk\bin\java.exe): The filename or extension is too long.

If possible, provide a minimal example to reproduce the problem:

subst d: c:\d_drive

mkdir d:\os

d:

cd d:\os

git clone https://github.com/bazelbuild/bazel.git

git checkout 2686e4e301dec8d30b901e8fcdce70dd83cec69c

c:\work\bazel-0.7.0\bazel.exe --output_base=d:\bazel_build build --stamp --action_env=USE_DYNAMIC_CRT=1 --action_env=NO_MSVC_WRAPPER=1 --compilation_mode opt --verbose_failures src:bazel.exe

D:/bazel_build/execroot/io_bazel/bazel-out/msvc_x64-opt/bin/src/bazel.exe --output_base=d:\bazel_build build --stamp --action_env=USE_DYNAMIC_CRT=1 --action_env=NO_MSVC_WRAPPER=1 --compilation_mode opt --verbose_failures src:bazel.exe

Environment info

• Operating System: Windows 10

• Bazel version (output of bazel info release): 0.7.0, then a dev version

Have you found anything relevant by searching the web?

Found this while attempting to repro #4035

[laszlocsomor]

This is still not fixed, see first message on #4924.

I found the culprit:

• JavaHeaderCompileAction creates a "two-tiered spawn":

  bazel/src/main/java/com/google/devtools/build/lib/rules/java/JavaHeaderCompileAction.java

  Lines 472 to 476 in 73088a8

  	// The action doesn't require annotation processing, javac-turbine fallback is enabled, and
  	// the target distinguishes between direct and transitive deps. Try a two-tiered spawn
  	// the invokes turbine with direct deps, and falls back to javac-turbine on failures to
  	// produce better diagnostics. (At the cost of slower failed actions and a larger
  	// cache footprint.)

• the two-tiered spawn runs one command, and upon failure it falls back on another one:

  bazel/src/main/java/com/google/devtools/build/lib/rules/java/JavaHeaderCompileAction.java

  Lines 145 to 153 in 73088a8

  	try {
  	return context.exec(spawn, actionExecutionContext);
  	} catch (ExecException e) {
  	// if the direct input spawn failed, try again with transitive inputs to produce better
  	// better messages
  	try {
  	return context.exec(
  	getSpawn(actionExecutionContext.getClientEnv()),
  	actionExecutionContext);

• the build reports an error but succeeds on Windows because the first command's length is above the 32K limit (as the error says) but the second one's isn't -- I found this by logging the lengths of spawn.arguments() and of getSpawn(...).arguments():

  bazel/src/main/java/com/google/devtools/build/lib/rules/java/JavaHeaderCompileAction.java

  Line 143 in 73088a8

  Spawn spawn = getDirectSpawn();

  bazel/src/main/java/com/google/devtools/build/lib/rules/java/JavaHeaderCompileAction.java

  Line 152 in 73088a8

  getSpawn(actionExecutionContext.getClientEnv()),

Solution:

• JavaHeaderCompileAction must compute the length of directCommandLine:

  bazel/src/main/java/com/google/devtools/build/lib/rules/java/JavaHeaderCompileAction.java

  Line 492 in 73088a8

  directCommandLine,

• JavaHeaderCompileAction must compare that length to the platform command line length limit, and create a parameter file if needed -- /cc @cushon @iirina
• to compute the length of the CustomCommandLine, we have multiple options:
  • naive: flatten it and observe the length
  • smart: implement a CountingCustomCommandLine that computes the length in its Builder -- /cc @tomlu: WDYT of this?

[laszlocsomor]

This problem isn't specific to Windows, Bazel might trigger it on other platforms too, if the two-tiered spawn's first command is longer than the platform's limit but the second one isn't.

[laszlocsomor]

The failing command is in turbine-failure.txt.

Using a parameter file works:

C:\tmp3\o6Hbs7N0\execroot\io_bazel>type c:\tmp\javah-params
--output
c:/tmp/javah-libbuild-base-hjar.jar
--output_deps
c:/tmp/javah-libbuild-base-hjar.jdeps
--temp_dir
c:/tmp/javahlibbuild-base-hjar_temp
--bootclasspath
bazel-out/host/genfiles/external/bazel_tools/tools/jdk/platformclasspath.jar
--sources
src/main/java/com/google/devtools/build/lib/analysis/ActionsProvider.java
src/main/java/com/google/devtools/build/lib/analysis/AliasProvider.java
src/main/java/com/google/devtools/build/lib/analysis/AnalysisEnvironment.java
...
bazel-out/x64_windows-fastbuild/genfiles/third_party/_ijar/guava/third_party/guava/guava-24.1-jre-ijar.jar
bazel-out/x64_windows-fastbuild/genfiles/third_party/_ijar/jsr305/third_party/jsr305/jsr-305-ijar.jar

C:\tmp3\o6Hbs7N0\execroot\io_bazel>"C:\tmp3\o6hbs7n0\execroot\io_bazel\external\local_jdk\bin\java.exe" -Xverify:none -Xbootclasspath/p:external/bazel_tools/third_party/java/jdk/langtools/javac-9+181-r4173-1.jar -jar external/bazel_tools/tools/jdk/turbine_deploy.jar @c:/tmp/javah-params

[laszlocsomor]

I talked with @aehlig and @dslomov about this problem. Here's what we thought out:

• Evaluating the CustomCommandLine to count its length beats the purpose of using one, and a CountingCustomCommandLine (or its Builder) would still need to flatten NestedSets. However that is exactly what SpawnAction.Builder.buildCommandLine() does:

  bazel/src/main/java/com/google/devtools/build/lib/analysis/actions/SpawnAction.java

  Line 689 in 73088a8

  totalLen += getCommandLineSize(commandLineAndParamFileInfo.commandLine);

• The ...SpawnRunner objects could provide ground truth of the platform limitations such as command length limits, but they lack the ability to turn a specific long command line into a short parameter-file using one, nor do they know whether the executable supports params files anyway. Only the ConfiguredTarget creating the Action knows all this.
• Therefore: if the rule anticipates its action exceeding the command length limit, then let the rule create the action with a fallback command line that's guaranteed to be short. We already have a similar machinery in SpawnAction.Builder (see .buildCommandLine() earlier), but we need a generic, Spawn-level solution. The SpawnRunner would evaluate the main command line, compare to the limit, and respectively use the main command line or create a parameter file and use the alternative command line.
• Keeping two command lines in sync is error-prone, let's maybe add a state-changing method to CustomCommandLine.Builder that says "all following arguments may go into a params file, and here's the argument to pass this params file".

@tomlu , WDYT?

[tomlu]

I have a design doc out for review request that would address this, but as of yet nobody has been looking at it :( Look at our internal queue. We could promote this document to a public one with some light scrubbing (eg. project names and benchmarks).

…

On Wed, Mar 28, 2018 at 7:00 AM, László Csomor ***@***.***> wrote: I talked with @aehlig <https://github.com/aehlig> and @dslomov <https://github.com/dslomov> about this problem. Here's what we thought out: - Evaluating the CustomCommandLine to count its length beats the purpose of using one, and a CountingCustomCommandLine (or its Builder) would still need to flatten NestedSets. However that is exactly what SpawnAction.Builder.buildCommandLine() does: https://github.com/bazelbuild/bazel/blob/73088a8a010232ae540080eacb7815 3612c1ce93/src/main/java/com/google/devtools/build/lib/ analysis/actions/SpawnAction.java#L689 <https://github.com/bazelbuild/bazel/blob/73088a8a010232ae540080eacb78153612c1ce93/src/main/java/com/google/devtools/build/lib/analysis/actions/SpawnAction.java#L689> - The ...SpawnRunner objects could provide ground truth of the platform limitations such as command length limits, but they lack the ability to turn a specific long command line into a short parameter-file using one, nor do they know whether the executable supports params files anyway. Only the ConfiguredTarget creating the Action knows all this. - Therefore: if the rule anticipates its action exceeding the command length limit, then let the rule create the action with a fallback command line that's guaranteed to be short. We already have a similar machinery in SpawnAction.Builder (see .buildCommandLine() earlier), but we need a generic, Spawn-level solution. The SpawnRunner would evaluate the main command line, compare to the limit, and respectively use the main command line or create a parameter file and use the alternative command line. - Keeping two command lines in sync is error-prone, let's maybe add a state-changing method to CustomCommandLine.Builder that says "all following arguments may go into a params file, and here's the argument to pass this params file". @tomlu <https://github.com/tomlu> , WDYT? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#4096 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABbnSmlH1vtt5veWJj9-dd2cPUUVNGe1ks5ti21OgaJpZM4Qe4ER> .

[laszlocsomor]

Sorry, I can't look at the doc, just don't have the mental bandwidth for it.

[cushon]

@tomlu's doc looks great to me. What is the priority of this bug? Can we wait for the principled solution?

Conditionally creating a params file the way that is currently supported has unacceptable performance overhead. Unconditionally creating a params file would address this bug, but also has performance overhead.

[laszlocsomor]

@tomlu's doc looks great to me. What is the priority of this bug? Can we wait for the principled solution?

I don't think it's urgent, because Bazel still builds, but it displays this error message. I guess we can wait

[cushon]

Deferred params files happened, so this should be easy to fix if it's still happening. We aren't doing the two-tiered spawn thing for header compilation anymore, though, so maybe it's no longer a problem?

[laszlocsomor]

I haven't seen this issue in a long time, maybe it was accidentally fixed. Setting P3.

[meisterT]

Closing the issue since it seems to be fixed.