[BUG] Investigate/remove use of empty docker-export/build.gradle file

[kartg]

Describe the bug
This is a follow up to the PR #4723, which aimed to remove all unnecessary empty build.gradle files. The original change resulted in a failing Gradle check:

....
> Task :distribution:docker:docker-ppc64le-export:exportDockerImage
> Task :distribution:docker:preProcessFixture
> Task :distribution:docker:docker-export:exportDockerImage FAILED
....
* What went wrong:
Execution failed for task ':distribution:docker:docker-export:exportDockerImage'.
> A problem occurred starting process 'command 'docker''

I was able to repro this locally:

$ ./gradlew --stacktrace :distribution:docker:docker-export:exportDockerImage
...
* Exception is:
org.gradle.api.tasks.TaskExecutionException: Execution failed for task ':distribution:docker:docker-export:exportDockerImage'.
        at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.lambda$executeIfValid$1(ExecuteActionsTaskExecuter.java:149)
        at org.gradle.internal.Try$Failure.ifSuccessfulOrElse(Try.java:282)
        at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.executeIfValid(ExecuteActionsTaskExecuter.java:147)
        at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.execute(ExecuteActionsTaskExecuter.java:135)
        at org.gradle.api.internal.tasks.execution.FinalizePropertiesTaskExecuter.execute(FinalizePropertiesTaskExecuter.java:46)
        at org.gradle.api.internal.tasks.execution.ResolveTaskExecutionModeExecuter.execute(ResolveTaskExecutionModeExecuter.java:51)
        at org.gradle.api.internal.tasks.execution.SkipTaskWithNoActionsExecuter.execute(SkipTaskWithNoActionsExecuter.java:57)
        at org.gradle.api.internal.tasks.execution.SkipOnlyIfTaskExecuter.execute(SkipOnlyIfTaskExecuter.java:74)
        at org.gradle.api.internal.tasks.execution.CatchExceptionTaskExecuter.execute(CatchExceptionTaskExecuter.java:36)
        at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter$1.executeTask(EventFiringTaskExecuter.java:77)
        at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter$1.call(EventFiringTaskExecuter.java:55)
        at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter$1.call(EventFiringTaskExecuter.java:52)
        at org.gradle.internal.operations.DefaultBuildOperationRunner$CallableBuildOperationWorker.execute(DefaultBuildOperationRunner.java:204)
        at org.gradle.internal.operations.DefaultBuildOperationRunner$CallableBuildOperationWorker.execute(DefaultBuildOperationRunner.java:199)
        at org.gradle.internal.operations.DefaultBuildOperationRunner$2.execute(DefaultBuildOperationRunner.java:66)
        at org.gradle.internal.operations.DefaultBuildOperationRunner$2.execute(DefaultBuildOperationRunner.java:59)
        at org.gradle.internal.operations.DefaultBuildOperationRunner.execute(DefaultBuildOperationRunner.java:157)
        at org.gradle.internal.operations.DefaultBuildOperationRunner.execute(DefaultBuildOperationRunner.java:59)
        at org.gradle.internal.operations.DefaultBuildOperationRunner.call(DefaultBuildOperationRunner.java:53)
        at org.gradle.internal.operations.DefaultBuildOperationExecutor.call(DefaultBuildOperationExecutor.java:73)
        at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter.execute(EventFiringTaskExecuter.java:52)
        at org.gradle.execution.plan.LocalTaskNodeExecutor.execute(LocalTaskNodeExecutor.java:42)
        at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$InvokeNodeExecutorsAction.execute(DefaultTaskExecutionGraph.java:337)
        at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$InvokeNodeExecutorsAction.execute(DefaultTaskExecutionGraph.java:324)
        at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$BuildOperationAwareExecutionAction.execute(DefaultTaskExecutionGraph.java:317)
        at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$BuildOperationAwareExecutionAction.execute(DefaultTaskExecutionGraph.java:303)
        at org.gradle.execution.plan.DefaultPlanExecutor$ExecutorWorker.execute(DefaultPlanExecutor.java:463)
        at org.gradle.execution.plan.DefaultPlanExecutor$ExecutorWorker.run(DefaultPlanExecutor.java:380)
        at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64)
        at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:49)
Caused by: org.gradle.process.internal.ExecException: A problem occurred starting process 'command 'docker''
        at org.gradle.process.internal.DefaultExecHandle.execExceptionFor(DefaultExecHandle.java:241)
        at org.gradle.process.internal.DefaultExecHandle.setEndStateInfo(DefaultExecHandle.java:218)
        at org.gradle.process.internal.DefaultExecHandle.failed(DefaultExecHandle.java:370)
        at org.gradle.process.internal.ExecHandleRunner.run(ExecHandleRunner.java:87)
        at org.gradle.internal.operations.CurrentBuildOperationPreservingRunnable.run(CurrentBuildOperationPreservingRunnable.java:42)
        at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64)
        at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:49)
Caused by: net.rubygrapefruit.platform.NativeException: Could not start 'docker'
        at net.rubygrapefruit.platform.internal.DefaultProcessLauncher.start(DefaultProcessLauncher.java:27)
        at net.rubygrapefruit.platform.internal.WrapperProcessLauncher.start(WrapperProcessLauncher.java:36)
        at org.gradle.process.internal.ExecHandleRunner.startProcess(ExecHandleRunner.java:98)
        at org.gradle.process.internal.ExecHandleRunner.run(ExecHandleRunner.java:71)
        ... 3 more
Caused by: java.io.IOException: Cannot run program "docker" (in directory "<redacted-OS-root-path>/distribution/docker/docker-export"): error=2, No such file or directory
        at net.rubygrapefruit.platform.internal.DefaultProcessLauncher.start(DefaultProcessLauncher.java:25)
        ... 6 more
Caused by: java.io.IOException: error=2, No such file or directory
        ... 7 more

Reverting the removal of the distribution/docker/docker-export/build.gradle allowed Gradle check to succeed, even though the file is empty. The PR was merged, but the root-cause for why this file is is required is still unclear. This issue tracks the investigation work to resolve this and potentially remove this empty file.

To Reproduce
Steps to reproduce the behavior:

1. Check out the main branch and run ./gradlew clean
2. Run ./gradlew :distribution:docker:docker-export:exportDockerImage and validate that this succeeds
3. Remove the empty gradle file - rm distribution/docker/docker-export/build.gradle
4. Rerun ./gradlew :distribution:docker:docker-export:exportDockerImage and observe that it now fails

Expected behavior
./gradlew :distribution:docker:docker-export:exportDockerImage should pass even with the empty file removed

Plugins
N/A

Screenshots
N/A

Host/Environment (please complete the following information):
My local development environnment is MacOS Monterey 12.6.6

Additional context
N/A

[dbwiddis]

I've spent a few hours digging into this. Some notes so far:

• I was initially unable to repro the problem
• I traced through enough of the stack trace to see the problem occurs when trying to run docker inside a non-existent directory (so it's not the lack of empty build.gradle causing the problem, it's the directory being absent)
• The problem occurs in the gradle-docker-compose-plugin.
• I found a very similar stack trace associated with a different command but same root cause in this issue, which notes it's a problem on JDK17 and not JDK11 (Of note I'm running on JDK 19.0.1 right now.) Cannot run program "docker-compose" - No such file or directory avast/gradle-docker-compose-plugin#406
• Based on this, I changed my JAVA_HOME to JDK 17.0.5 and tried again. I got a different error:

> Task :libs:opensearch-core:compileJava
/Users/danielwiddis/git/OpenSearch/libs/core/src/main/java/org/opensearch/core/common/util/CollectionUtils.java:143: warning: [unchecked] unchecked generic array creation for varargs parameter of type Iterator<? extends Object>[]
            return () -> Iterators.concat(map.keySet().iterator(), map.values().iterator());
                                         ^
error: warnings found and -Werror specified
Note: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
1 error
1 warning

> Task :libs:opensearch-core:compileJava FAILED

• deferring troubleshooting of that I switched back to JDK 19.0.1, and am now able to reproduce the error
• based on linked issue tried with JDK 11.0.17, still reproduced the error
• ran it with debug, relevant lines show the command being executed that fails (the working directory doesn't exist)

2023-08-05T15:17:33.728-0700 [DEBUG] [org.gradle.internal.execution.steps.CreateOutputsStep] Ensuring parent directory exists for property $1 at /Users/danielwiddis/git/OpenSearch/distribution/docker/build/opensearch_test_3.0.0-SNAPSHOT.docker.tar
2023-08-05T15:17:33.728-0700 [DEBUG] [org.gradle.api.internal.tasks.execution.TaskExecution] Executing actions for task ':distribution:docker:docker-export:exportDockerImage'.
2023-08-05T15:17:33.729-0700 [INFO] [org.gradle.process.internal.DefaultExecHandle] Starting process 'command 'docker''. Working directory: /Users/danielwiddis/git/OpenSearch/distribution/docker/docker-export Command: docker save -o /Users/danielwiddis/git/OpenSearch/distribution/docker/build/opensearch_test_3.0.0-SNAPSHOT.docker.tar opensearch:test
2023-08-05T15:17:33.729-0700 [DEBUG] [org.gradle.process.internal.DefaultExecHandle] Changing state to: STARTING
2023-08-05T15:17:33.729-0700 [DEBUG] [org.gradle.process.internal.DefaultExecHandle] Waiting until process started: command 'docker'.
2023-08-05T15:17:33.736-0700 [DEBUG] [org.gradle.process.internal.DefaultExecHandle] Changing state to: FAILED
2023-08-05T15:17:33.736-0700 [DEBUG] [org.gradle.process.internal.DefaultExecHandle] Process 'command 'docker'' finished with exit value -1 (state: FAILED)

[dbwiddis]

It appears the inability to repro initially was because the gradle daemon does some sort of caching. By running ./gradlew --stop in between repro steps 3 and 4 above it's reproducible on JDK 17 and JDK 19, but not on JDK 11.

[dbwiddis]

Here's where the failing executable is defined.

https://github.com/opensearch-project/OpenSearch/blob/4c5981091fcff67830bc6a9f2f9dee9469dffae7/distribution/docker/build.gradle#L262C1-L273

    tasks.register(exportTaskName, LoggedExec) {
      inputs.file("${parent.projectDir}/build/markers/${buildTaskName}.marker")
      executable 'docker'
      outputs.file(tarFile)
      args "save",
        "-o",
        tarFile,
        "opensearch:test"

      dependsOn(parent.path + ":" + buildTaskName)
      onlyIf { Architecture.current() == architecture && !OperatingSystem.current().isWindows()}
    }

[dbwiddis]

After testing this many different ways, my conclusions:

• Unable to repro by just deleting build.gradle
• Repro 100% on macOS by deleting the entire directory, on JDK17+

It seems the directory's existence is required. The code eventually calls ProcessBuilder.start(), which indicates the error is expected (if not consistent):

Starting an operating system process is highly system-dependent. Among the many things that can go wrong are:

• The operating system program file was not found.
• Access to the program file was denied.
• The working directory does not exist.
• Invalid character in command argument, such as NUL.

This doesn't explain why there were initially problems without build.gradle but it may be related to the macOS default of not including /usr/local/bin on the PATH, which is where (a link to) docker is typically installed.

[reta]

@kartg @dbwiddis Closing this one , we do have an explanation for reasons why we need pre-created folders, see please #11136 (review)