Concurrency issue when using GradleBuild task in multiproject setup

[davidburstromspotify]

JavaCompile tasks can fail randomly if the Spotless plugin is applied on subprojects AND a GradleBuild task is executing on the same project in parallel.

Gradle version 7.3.3
Spotless Gradle plugin version 6.1.2
Gradle build scan: https://gradle.com/s/enf5wof4kfw7i
Public repo: https://github.com/davidburstromspotify/spotless-issue-1087

I have a feeling the plugin causes some cross talk between the outer and inner build.

org.gradle.api.tasks.TaskExecutionException: Execution failed for task ':subproject5:compileJava'.
        at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.lambda$executeIfValid$1(ExecuteActionsTaskExecuter.java:145)
        at org.gradle.internal.Try$Failure.ifSuccessfulOrElse(Try.java:282)
        at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.executeIfValid(ExecuteActionsTaskExecuter.java:143)
        at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.execute(ExecuteActionsTaskExecuter.java:131)
        ...
Caused by: java.lang.NullPointerException
        at java.util.concurrent.ConcurrentHashMap.get(ConcurrentHashMap.java:936)
        at org.gradle.api.internal.tasks.compile.tooling.JavaCompileTaskSuccessResultPostProcessor.findTaskOperationId(JavaCompileTaskSuccessResultPostProcessor.java:65)
        at org.gradle.api.internal.tasks.compile.tooling.JavaCompileTaskSuccessResultPostProcessor.findTaskOperationId(JavaCompileTaskSuccessResultPostProcessor.java:69)
        at org.gradle.api.internal.tasks.compile.tooling.JavaCompileTaskSuccessResultPostProcessor.finished(JavaCompileTaskSuccessResultPostProcessor.java:58)
        ...

[nedtwigg]

I think I know why this is happening. We create a single task in the root project, :spotlessInternalRegisterDependencies. The problem is that in a composite build, there will be two different Spotless plugins, each with a classloader from their own build, each trying to register the same task.

The way that we resolved that was:

• try to register :spotlessInternalRegisterDependencies
• if that fails, then register :spotlessInternalRegisterDependencies${System.identityHashCode(RegisterDependenciesTask.class)}

spotless/plugin-gradle/src/main/java/com/diffplug/gradle/spotless/SpotlessExtension.java

Lines 242 to 252 in 804a73a

	TaskProvider<RegisterDependenciesTask> findRegisterDepsTask() {
	try {
	return findRegisterDepsTask(RegisterDependenciesTask.TASK_NAME);
	} catch (Exception e) {
	// in a composite build there can be multiple Spotless plugins on the classpath, and they will each try to register
	// a task on the root project with the same name. That will generate casting errors, which we can catch and try again
	// with an identity-specific identifier.
	// https://github.com/diffplug/spotless/pull/1001 for details
	return findRegisterDepsTask(RegisterDependenciesTask.TASK_NAME + System.identityHashCode(RegisterDependenciesTask.class));
	}
	}

The exception being caught above is thrown on this line:

spotless/plugin-gradle/src/main/java/com/diffplug/gradle/spotless/SpotlessExtension.java

Line 259 in 804a73a

return rootProjectTasks.named(taskName, RegisterDependenciesTask.class);

I think I have an easy solution (I feel like there's an air bubble of "Gradle demands to declare dependencies in the root project" and Spotless is pushing that bubble around, but we do seem to have concretely ironed out some cases, and here we can iron out another.

[nedtwigg]

I was wrong. I added some very simple printlns: 72919ce

And this is the output I get over and over:

./gradlew build          

> Configure project :
~~~ REGISTER spotlessInternalRegisterDependencies

> Configure project :spotless-issue-1087
~~~ REGISTER spotlessInternalRegisterDependencies

> Task :spotless-issue-1087:help

The important thing is that "FAILED TO REGISTER" is never printed, and it is strange that "REGISTER" succeeds twice. Not sure what to make of that...

[davidburstromspotify]

You probably also already noticed this, but it's not even necessary to run any Spotless tasks:
./gradlew gradleBuild compileJava is enough to provoke the issue. But as you point out, task registration will happen regardless.
https://scans.gradle.com/s/3cxzpd6plgo22/timeline

[nedtwigg]

This seems like a Gradle bug to me. I wonder if something as simple as this could reproduce it:

public class BugPlugin implements Plugin<Project> {
  static final String BUG_TASK = "bug";

  TaskProvider<BugTask> bugTask;

  @Override
  public void apply(Project project) {
    if (!project.rootProject.tasks.names.contains(BUG_TASK)) {
        bugTask = rootProjectTasks.register(BUG_TASK, BugTask.class, BugTask::setup);
    } else {
        bugTask = rootProjectTasks.named(BUG_TASK, BugTask.class);
    }
}

[davidburstromspotify]

It could certainly be a Gradle bug, would be good to rule out Spotless from the equation.

[davidburstromspotify]

I tried with

subprojects {
    if (!project.rootProject.tasks.names.contains("dummy")) {
        project.rootProject.tasks.register("dummy", DefaultTask::class.java)
    } else {
        println("contains dummy")
    }
}   

which should be the equivalent of applying the plugin, but it didn't provoke any issue.

Looking at the debug logs when using the Spotless plugin, it looks like Gradle registers one :spotlessInternalRegisterDependencies and one :spotless-issue-1087:spotlessInternalRegisterDependencies, which indicates they apply on different projects (the latter being considered the root project for the GradleBuild invocation). What's the risk of registering the task twice in one Gradle invocation?

[davidburstromspotify]

The fact that there's a reference to the Gradle instance makes my spidey-sense tingle, though I'm not sure if or how that can introduce any unexpected cross-talk:

https://github.com/diffplug/spotless/blob/main/plugin-gradle/src/main/java/com/diffplug/gradle/spotless/RegisterDependenciesTask.java#L65

[nedtwigg]

Maybe instead of DefaultTask use BugTask, and have that BugTask create a shared build since that's what Spotless does?

the latter being considered the root project for the GradleBuild invocation

I agree that seems to be the case, but I don't understand why that is happening / is possible. Seems like a very bad design to have "which project is the root" depend on the context of how it was called.

[Cimballi]

I also faced the same bug, I cannot help you much but it would be great if you could solve it !

[JamesXNelson]

fwiw, we are now seeing this as well in gradle 7.4.2 (I know, I know...)

[nedtwigg]

We've made huge changes in 7.0.0.BETA4. I'm going to close this as stale, if the issue persists in later versions please comment and I'll reopen.