[SPARK-37529][K8S][TESTS][FOLLOWUP] Allow dev-run-integration-tests.sh to take a custom Dockerfile #34818

sarutak · 2021-12-06T08:46:44Z

What changes were proposed in this pull request?

This PR changes dev-run-integration-tests.sh to allow it to take a custom Dockerfile like #34790 did.
With this change, this script accepts --docker-file option, which takes a path to a custom Dockerfile.

$ ./dev/run-integration-tests.sh --docker-file /path/to/dockerfile

Why are the changes needed?

As of #34790, we can specify a custom Dockerfile by spark.kubernetes.test.dockerFile property when we run the K8s integration tests using Maven.
We can run the integration test via dev-run-integration-tests.sh but there is no way to specify a custom Dockerfile.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Confirmed that the K8s integration tests run with the following command using Dockerfile.java17.

cd resource-managers/kubernetes/integration-tests
./dev/dev-run-integration-tests.sh --docker-file ../docker/src/main/dockerfiles/spark/Dockerfile.java17

…erfile.

sarutak · 2021-12-06T08:47:16Z

resource-managers/kubernetes/integration-tests/README.md

    <td><code>spark-r</code></td>
  </tr>
+  <tr>
+    <td><code>spark.kubernetes.test.dockerFile</code></td>


This part is what I forgot to add in #34790 .

sarutak · 2021-12-06T08:49:19Z

resource-managers/kubernetes/integration-tests/dev/dev-run-integration-tests.sh

 )

-$TEST_ROOT_DIR/build/mvn install -f $TEST_ROOT_DIR/pom.xml -pl resource-managers/kubernetes/integration-tests $BUILD_DEPENDENCIES_MVN_FLAG -Pscala-$SCALA_VERSION -P$HADOOP_PROFILE -Pkubernetes -Pkubernetes-integration-tests ${properties[@]}
+(cd $TEST_ROOT_DIR; ./build/mvn install -pl resource-managers/kubernetes/integration-tests $BUILD_DEPENDENCIES_MVN_FLAG -Pscala-$SCALA_VERSION -P$HADOOP_PROFILE -Pkubernetes -Pkubernetes-integration-tests ${properties[@]})


I noticed that all the examples of dev-run-integration-tests.sh in integration-tests/README.md don't work because scala-style-config.xml is not found in integration-tests.
This change is to resolve the issue.

Does this work? I'm testing your PR and hit the following.

$ cd resource-managers/kubernetes/integration-tests $ ./dev/dev-run-integration-tests.sh --docker-file ../docker/src/main/dockerfiles/spark/Dockerfile.java17 ++ git rev-parse --show-toplevel + TEST_ROOT_DIR=/Users/dongjoon/APACHE/spark-merge + DEPLOY_MODE=minikube + IMAGE_REPO=docker.io/kubespark + SPARK_TGZ=N/A + IMAGE_TAG=N/A + JAVA_IMAGE_TAG= + BASE_IMAGE_NAME= + JVM_IMAGE_NAME= + PYTHON_IMAGE_NAME= + R_IMAGE_NAME= + DOCKER_FILE= + SPARK_MASTER= + NAMESPACE= + SERVICE_ACCOUNT= + CONTEXT= + INCLUDE_TAGS=k8s + EXCLUDE_TAGS= + JAVA_VERSION=8 + BUILD_DEPENDENCIES_MVN_FLAG=-am + HADOOP_PROFILE=hadoop-3.2 + MVN=/Users/dongjoon/APACHE/spark-merge/build/mvn ++ /Users/dongjoon/APACHE/spark-merge/build/mvn help:evaluate -Dexpression=scala.binary.version ++ grep -v INFO ++ grep -v WARNING ++ tail -n 1 + SCALA_VERSION=2.12 + export SCALA_VERSION + echo 2.12 2.12 + (( 2 )) + case $1 in + DOCKER_FILE=../docker/src/main/dockerfiles/spark/Dockerfile.java17 + shift + shift + (( 0 )) + properties=(-Djava.version=$JAVA_VERSION -Dspark.kubernetes.test.sparkTgz=$SPARK_TGZ -Dspark.kubernetes.test.imageTag=$IMAGE_TAG -Dspark.kubernetes.test.imageRepo=$IMAGE_REPO -Dspark.kubernetes.test.deployMode=$DEPLOY_MODE -Dtest.include.tags=$INCLUDE_TAGS) + '[' -n '' ']' + '[' -n ../docker/src/main/dockerfiles/spark/Dockerfile.java17 ']' + properties=(${properties[@]} -Dspark.kubernetes.test.dockerFile=$(realpath $DOCKER_FILE)) ++ realpath ../docker/src/main/dockerfiles/spark/Dockerfile.java17 ./dev/dev-run-integration-tests.sh: line 153: realpath: command not found

Hmm, realpath seems not to be a standard command. I tested on Pop!_OS 20.04 and CentOS 8.
I'll look for another way.

Python's os.path.realpath should work on both Linux and macOS.

No~ It's Spark bash function in util.sh.

Ah, Linux has realpath command (coreutils) but we have own realpath.

SparkQA · 2021-12-06T09:04:37Z

Test build #145947 has finished for PR 34818 at commit 8072b13.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-12-06T09:36:24Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50421/

SparkQA · 2021-12-06T10:20:39Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50421/

dongjoon-hyun · 2021-12-06T17:41:50Z

resource-managers/kubernetes/integration-tests/dev/dev-run-integration-tests.sh

+  properties=(
+    ${properties[@]}
+    -Dspark.kubernetes.test.dockerFile=$(
+      python3 -c "import os.path; print(os.path.realpath(\"$DOCKER_FILE\"))") )


@sarutak We should use realpath consistently in our Spark bash code.

It seems that the root cause was that

You can a relative path via realpath in some directory

And, we moved to another directory.

I didn't notice realpath in util.sh and I used realpath command which is installed on Linux.
OK, I'll use our own realpath here.

SparkQA · 2021-12-06T18:01:28Z

Test build #145958 has finished for PR 34818 at commit 26eca41.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-12-06T18:28:09Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50432/

SparkQA · 2021-12-06T19:27:10Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50432/

SparkQA · 2021-12-06T20:02:12Z

Test build #145963 has finished for PR 34818 at commit 167b681.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-12-06T20:32:44Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50439/

SparkQA · 2021-12-06T21:15:15Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50439/

SparkQA · 2021-12-08T09:10:30Z

Test build #146007 has finished for PR 34818 at commit b599191.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class PandasSQLStringFormatter(string.Formatter):
class SQLStringFormatter(string.Formatter):
case class ConvertTimezone(

SparkQA · 2021-12-08T09:43:01Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50483/

SparkQA · 2021-12-08T10:36:24Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50483/

dongjoon-hyun

+1, LGTM. Thank you, @sarutak . It works as described.

sarutak · 2021-12-08T16:58:18Z

@dongjoon-hyun Thank you for your advice and review !

Change dev-run-integration-tests.sh to allow to specify a custom Dock…

8072b13

…erfile.

sarutak commented Dec 6, 2021

View reviewed changes

github-actions bot added DOCS KUBERNETES labels Dec 6, 2021

Use Python's os.path.realpath instead of realpath command.

26eca41

dongjoon-hyun reviewed Dec 6, 2021

View reviewed changes

Use own realpath script function.

167b681

dongjoon-hyun self-assigned this Dec 7, 2021

Resolve conflict.

b599191

dongjoon-hyun approved these changes Dec 8, 2021

View reviewed changes

dongjoon-hyun closed this in ca9a68d Dec 8, 2021

dongjoon-hyun removed their assignment Apr 14, 2024

[SPARK-37529][K8S][TESTS][FOLLOWUP] Allow dev-run-integration-tests.sh to take a custom Dockerfile #34818

[SPARK-37529][K8S][TESTS][FOLLOWUP] Allow dev-run-integration-tests.sh to take a custom Dockerfile #34818

Uh oh!

Conversation

sarutak commented Dec 6, 2021

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

sarutak Dec 6, 2021

Choose a reason for hiding this comment

Uh oh!

sarutak Dec 6, 2021

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun Dec 6, 2021

Choose a reason for hiding this comment

Uh oh!

sarutak Dec 6, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sarutak Dec 6, 2021

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun Dec 6, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun Dec 6, 2021

Choose a reason for hiding this comment

Uh oh!

sarutak Dec 6, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Dec 6, 2021

Uh oh!

SparkQA commented Dec 6, 2021

Uh oh!

SparkQA commented Dec 6, 2021

Uh oh!

dongjoon-hyun Dec 6, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sarutak Dec 6, 2021

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Dec 6, 2021

Uh oh!

SparkQA commented Dec 6, 2021

Uh oh!

SparkQA commented Dec 6, 2021

Uh oh!

SparkQA commented Dec 6, 2021

Uh oh!

SparkQA commented Dec 6, 2021

Uh oh!

SparkQA commented Dec 6, 2021

Uh oh!

SparkQA commented Dec 8, 2021

Uh oh!

SparkQA commented Dec 8, 2021

Uh oh!

SparkQA commented Dec 8, 2021

Uh oh!

dongjoon-hyun left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sarutak commented Dec 8, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

sarutak Dec 6, 2021 •

edited

Loading

dongjoon-hyun Dec 6, 2021 •

edited

Loading

sarutak Dec 6, 2021 •

edited

Loading

dongjoon-hyun Dec 6, 2021 •

edited

Loading

dongjoon-hyun left a comment •

edited

Loading