Race conditions with local action deduplication

[fmeum]

While testing path mapping on Bazel itself with 7.3.0, I just encountered the following exceptions:

java.lang.RuntimeException: Unrecoverable error while evaluating node 'ActionLookupData0{actionLookupKey=ConfiguredTargetKey{label=//src/main/java/com/google/devtools/build/lib/sandbox/cgroups/controller:v2, config=BuildConfigurationKey[153e20f2e3739e0fdcb3b5e06c3ec8614ec4e43b5ff5f93c78bf2a56108ee671]}, actionIndex=0}' (requested by nodes 'ArtifactNestedSetKey[4]@42667357', 'ArtifactNestedSetKey[3]@612534160', 'ArtifactNestedSetKey[8]@776188326')
	at com.google.devtools.build.skyframe.AbstractParallelEvaluator$Evaluate.run(AbstractParallelEvaluator.java:550)
	at com.google.devtools.build.lib.concurrent.AbstractQueueVisitor$WrappedRunnable.run(AbstractQueueVisitor.java:414)
	at java.base/java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(Unknown Source)
	at java.base/java.util.concurrent.ForkJoinTask.doExec(Unknown Source)
	at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(Unknown Source)
	at java.base/java.util.concurrent.ForkJoinPool.scan(Unknown Source)
	at java.base/java.util.concurrent.ForkJoinPool.runWorker(Unknown Source)
	at java.base/java.util.concurrent.ForkJoinWorkerThread.run(Unknown Source)
Caused by: java.lang.IllegalStateException: Missing original path for mapped path bazel-out/darwin_arm64-fastbuild/bin/src/main/java/com/google/devtools/build/lib/sandbox/cgroups/controller/libcontroller-hjar.jar in bazel-out/darwin_arm64-opt-exec-ST-fad1763555eb/bin/src/main/java/com/google/devtools/build/lib/sandbox/cgroups/controller/libv2-hjar.jdeps
jdeps: dependency {
  path: "bazel-out/darwin_arm64-fastbuild/bin/src/main/java/com/google/devtools/build/lib/sandbox/cgroups/controller/libcontroller-hjar.jar"
  kind: EXPLICIT
}
rule_label: "//src/main/java/com/google/devtools/build/lib/sandbox/cgroups/controller:v2"
success: true

path map: {bazel-out/cfg/bin/external/rules_java~/toolchains/platformclasspath.jar=bazel-out/darwin_arm64-opt-exec-ST-fad1763555eb/bin/external/rules_java~/toolchains/platformclasspath.jar, bazel-out/cfg/bin/src/main/java/com/google/devtools/build/lib/sandbox/cgroups/controller/libcontroller-hjar.jar=bazel-out/darwin_arm64-opt-exec-ST-fad1763555eb/bin/src/main/java/com/google/devtools/build/lib/sandbox/cgroups/controller/libcontroller-hjar.jar, bazel-out/cfg/bin/external/rules_jvm_external~~maven~maven/com/google/errorprone/error_prone_annotations/2.24.1/header_error_prone_annotations-2.24.1.jar=bazel-out/darwin_arm64-opt-exec-ST-fad1763555eb/bin/external/rules_jvm_external~~maven~maven/com/google/errorprone/error_prone_annotations/2.24.1/header_error_prone_annotations-2.24.1.jar, bazel-out/cfg/bin/external/rules_jvm_external~~maven~maven/com/google/errorprone/error_prone_type_annotations/2.23.0/header_error_prone_type_annotations-2.23.0.jar=bazel-out/darwin_arm64-opt-exec-ST-fad1763555eb/bin/external/rules_jvm_external~~maven~maven/com/google/errorprone/error_prone_type_annotations/2.23.0/header_error_prone_type_annotations-2.23.0.jar, bazel-out/cfg/bin/external/rules_jvm_external~~maven~maven/com/github/stephenc/jcip/jcip-annotations/1.0-1/header_jcip-annotations-1.0-1.jar=bazel-out/darwin_arm64-opt-exec-ST-fad1763555eb/bin/external/rules_jvm_external~~maven~maven/com/github/stephenc/jcip/jcip-annotations/1.0-1/header_jcip-annotations-1.0-1.jar, bazel-out/cfg/bin/external/rules_jvm_external~~maven~maven/com/google/code/findbugs/jsr305/3.0.2/header_jsr305-3.0.2.jar=bazel-out/darwin_arm64-opt-exec-ST-fad1763555eb/bin/external/rules_jvm_external~~maven~maven/com/google/code/findbugs/jsr305/3.0.2/header_jsr305-3.0.2.jar, bazel-out/cfg/bin/external/rules_jvm_external~~maven~maven/com/google/guava/guava/33.0.0-jre/header_guava-33.0.0-jre.jar=bazel-out/darwin_arm64-opt-exec-ST-fad1763555eb/bin/external/rules_jvm_external~~maven~maven/com/google/guava/guava/33.0.0-jre/header_guava-33.0.0-jre.jar, bazel-out/cfg/bin/external/rules_jvm_external~~maven~maven/com/google/guava/failureaccess/1.0.2/header_failureaccess-1.0.2.jar=bazel-out/darwin_arm64-opt-exec-ST-fad1763555eb/bin/external/rules_jvm_external~~maven~maven/com/google/guava/failureaccess/1.0.2/header_failureaccess-1.0.2.jar, bazel-out/cfg/bin/external/rules_jvm_external~~maven~maven/com/google/guava/listenablefuture/9999.0-empty-to-avoid-conflict-with-guava/header_listenablefuture-9999.0-empty-to-avoid-conflict-with-guava.jar=bazel-out/darwin_arm64-opt-exec-ST-fad1763555eb/bin/external/rules_jvm_external~~maven~maven/com/google/guava/listenablefuture/9999.0-empty-to-avoid-conflict-with-guava/header_listenablefuture-9999.0-empty-to-avoid-conflict-with-guava.jar, bazel-out/cfg/bin/external/rules_jvm_external~~maven~maven/com/google/j2objc/j2objc-annotations/2.8/header_j2objc-annotations-2.8.jar=bazel-out/darwin_arm64-opt-exec-ST-fad1763555eb/bin/external/rules_jvm_external~~maven~maven/com/google/j2objc/j2objc-annotations/2.8/header_j2objc-annotations-2.8.jar, bazel-out/cfg/bin/external/rules_jvm_external~~maven~maven/org/checkerframework/checker-qual/3.42.0/header_checker-qual-3.42.0.jar=bazel-out/darwin_arm64-opt-exec-ST-fad1763555eb/bin/external/rules_jvm_external~~maven~maven/org/checkerframework/checker-qual/3.42.0/header_checker-qual-3.42.0.jar}
	at com.google.devtools.build.lib.rules.java.JavaCompileAction.createFullOutputDeps(JavaCompileAction.java:796)
	at com.google.devtools.build.lib.rules.java.JavaHeaderCompileAction.afterExecute(JavaHeaderCompileAction.java:147)
	at com.google.devtools.build.lib.analysis.actions.SpawnAction.execute(SpawnAction.java:262)
	at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor$ActionRunner.executeAction(SkyframeActionExecutor.java:1159)
	at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor$ActionRunner.run(SkyframeActionExecutor.java:1076)
	at com.google.devtools.build.lib.skyframe.ActionExecutionState.runStateMachine(ActionExecutionState.java:165)
	at com.google.devtools.build.lib.skyframe.ActionExecutionState.getResultOrDependOnFuture(ActionExecutionState.java:94)
	at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor.executeAction(SkyframeActionExecutor.java:573)
	at com.google.devtools.build.lib.skyframe.ActionExecutionFunction.checkCacheAndExecuteIfNeeded(ActionExecutionFunction.java:862)
	at com.google.devtools.build.lib.skyframe.ActionExecutionFunction.computeInternal(ActionExecutionFunction.java:334)
	at com.google.devtools.build.lib.skyframe.ActionExecutionFunction.compute(ActionExecutionFunction.java:172)
	at com.google.devtools.build.skyframe.AbstractParallelEvaluator$Evaluate.run(AbstractParallelEvaluator.java:461)
	... 7 more

and

java.io.FileNotFoundException: /private/var/tmp/_bazel_fmeum/507738cfc7e6cde00e4a0230e9aa0722/execroot/_main/bazel-out/_tmp/actions/remote/175.tmp (No such file or directory)
	at com.google.devtools.build.lib.unix.NativePosixFiles.lstat(Native Method)
	at com.google.devtools.build.lib.unix.UnixFileSystem.statInternal(UnixFileSystem.java:212)
	at com.google.devtools.build.lib.unix.UnixFileSystem.stat(UnixFileSystem.java:201)
	at com.google.devtools.build.lib.vfs.Path.stat(Path.java:290)
	at com.google.devtools.build.lib.vfs.FileSystemUtils.moveFile(FileSystemUtils.java:456)
	at com.google.devtools.build.lib.remote.RemoteExecutionService.moveOutputsToFinalLocation(RemoteExecutionService.java:878)
	at com.google.devtools.build.lib.remote.RemoteExecutionService.waitForAndReuseOutputs(RemoteExecutionService.java:1473)
	at com.google.devtools.build.lib.remote.RemoteSpawnCache.lookup(RemoteSpawnCache.java:169)
	at com.google.devtools.build.lib.exec.AbstractSpawnStrategy.exec(AbstractSpawnStrategy.java:151)
	at com.google.devtools.build.lib.exec.AbstractSpawnStrategy.exec(AbstractSpawnStrategy.java:118)
	at com.google.devtools.build.lib.exec.SpawnStrategyResolver.exec(SpawnStrategyResolver.java:45)
	at com.google.devtools.build.lib.rules.java.JavaCompileAction.execute(JavaCompileAction.java:425)
	at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor$ActionRunner.executeAction(SkyframeActionExecutor.java:1155)
	at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor$ActionRunner.run(SkyframeActionExecutor.java:1068)
	at com.google.devtools.build.lib.skyframe.ActionExecutionState.runStateMachine(ActionExecutionState.java:166)
	at com.google.devtools.build.lib.skyframe.ActionExecutionState.getResultOrDependOnFuture(ActionExecutionState.java:95)
	at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor.executeAction(SkyframeActionExecutor.java:559)
	at com.google.devtools.build.lib.skyframe.ActionExecutionFunction.checkCacheAndExecuteIfNeeded(ActionExecutionFunction.java:902)
	at com.google.devtools.build.lib.skyframe.ActionExecutionFunction.computeInternal(ActionExecutionFunction.java:349)
	at com.google.devtools.build.lib.skyframe.ActionExecutionFunction.compute(ActionExecutionFunction.java:190)
	at com.google.devtools.build.skyframe.AbstractParallelEvaluator$Evaluate.run(AbstractParallelEvaluator.java:467)
	at com.google.devtools.build.lib.concurrent.AbstractQueueVisitor$WrappedRunnable.run(AbstractQueueVisitor.java:435)
	at java.base/java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1423)
	at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:387)
	at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1312)
	at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1843)
	at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1808)
	at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:188)

Originally posted by @fmeum in #22658 (comment)

[fmeum]

@bazel-io fork 7.3.1

[fmeum]

The second issue is fixed by #23296.

The first one is more difficult to solve: the Java compilation action runs two spawns and modifies or deletes outputs in multiple cases, both before falling back to the second spawn and after each spawn to rewrite the .jdeps file.

@tjgq Do you happen to know how remote cache uploads interact with this logic? It seems like either they are robust against this kind of race and then local execution deduplication could reuse the same logic or they are potentially affected by this as well.

[fmeum]

With the small fix in #23296 as well as #23307, which temporarily disables deduplication for Java actions, this problem should be gone at HEAD. If they get merged in time, we could even cherry-pick the rollforward with these two fixes into 7.3.1.

[fmeum]

@bazel-io fork 7.3.1

[fmeum]

@iancha1992 Please consider this a (very) soft blocker for 7.3.1 only: it would be great to get in if ready in time, but there is no need to block the release for it.

[tjgq]

@tjgq Do you happen to know how remote cache uploads interact with this logic? It seems like either they are robust against this kind of race and then local execution deduplication could reuse the same logic or they are potentially affected by this as well.

I would not expect remote uploads to be robust; the upload logic assumes that, once an output file is created, it will not disappear for the remainder of the build.

This is also a problem for async uploading. At some point I'd like to find a general solution, but I think for the time being we should special-case our way out of it (as you're already doing in #23307).