[SPARK-37529][K8S][TESTS] Support K8s integration tests for Java 17 #34790

sarutak · 2021-12-03T05:52:25Z

What changes were proposed in this pull request?

This PR aims to support K8s integration tests for Java 17 using Maven and SBT.
The new system property spark.kubernetes.test.dockerFile is introduced to specify a Dockerfile.
By setting Dockerfile.java17 to the property, the integration tests run with Java 17.

This PR also revised the change brought by SPARK-37354 (#34628) by changing SparkBuild.scala so that it can recognize the system property spark.kubernetes.test.javaImageTag, like the integration tests with Maven do.

If both spark.kubernetes.test.dockerFile and spark.kubernetes.test.javaImageTag are set, spark.kubernetes.test.dockerFile is preferred.

Why are the changes needed?

To ensure Spark works on K8s with Java 17.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Confirmed that the intended version of Java is used for each pattern.

Neither spark.kubernetes.test.javaImageTag nor spark.kubernetes.test.dockerFile is set (SBT)

# Run the integration tests to create the container image
$ build/sbt -Pkubernetes  -Pkubernetes-integration-tests "kubernetes-integration-tests/test"

# Create and login the container.
docker run -it <hash> /bin/bash

185@b40ef8aaa56c:/opt/spark/work-dir$ java -version
openjdk version "1.8.0_312"
OpenJDK Runtime Environment (build 1.8.0_312-b07)
OpenJDK 64-Bit Server VM (build 25.312-b07, mixed mode)

Neither spark.kubernetes.test.javaImageTag nor spark.kubernetes.test.dockerFile is set (Maven)

# Run the integration tests to create the container image
$ build/mvn -Pkubernetes -Pkubernetes-integration-tests -pl resource-managers/kubernetes/integration-tests integration-test

# Create and login the container.
docker run -it <hash> /bin/bash

185@36ae5c5c21f4:/opt/spark/work-dir$ java -version
openjdk version "1.8.0_312"
OpenJDK Runtime Environment (build 1.8.0_312-b07)
OpenJDK 64-Bit Server VM (build 25.312-b07, mixed mode)

spark.kubernetes.test.javaImageTag is set (SBT):

# Run the integration tests to create the container image
$ build/sbt -Dspark.kubernetes.test.javaImageTag=11-jre-slim -Pkubernetes  -Pkubernetes-integration-tests "kubernetes-integration-tests/test"

# Create and login the container.
docker run -it <hash> /bin/bash

185@5b896d0e5dd8:/opt/spark/work-dir$ java -version
openjdk version "11.0.13" 2021-10-19
OpenJDK Runtime Environment 18.9 (build 11.0.13+8)
OpenJDK 64-Bit Server VM 18.9 (build 11.0.13+8, mixed mode, sharing)

spark.kubernetes.test.javaImageTag is set (Maven):

# Run the integration tests to create the container image
$ build/mvn -Dspark.kubernetes.test.javaImageTag=11-jre-slim -Pkubernetes -Pkubernetes-integration-tests -pl resource-managers/kubernetes/integration-tests integration-test

# Create and login the container.
docker run -it <hash> /bin/bash

185@5d3c228e7521:/opt/spark/work-dir$ java -version
openjdk version "11.0.13" 2021-10-19
OpenJDK Runtime Environment 18.9 (build 11.0.13+8)
OpenJDK 64-Bit Server VM 18.9 (build 11.0.13+8, mixed mode, sharing)

spark.kubernetes.test.dockerFile is set (SBT)

$ build/sbt -Dspark.kubernetes.test.dockerFile=resource-managers/kubernetes/docker/src/main/dockerfiles/spark/Dockerfile.java17 -Pkubernetes  -Pkubernetes-integration-tests package "kubernetes-integration-tests/test"

# Create and login the container.
docker run -it <hash> /bin/bash

185@550515c76582:/opt/spark/work-dir$ java -version
openjdk version "17.0.1" 2021-10-19
OpenJDK Runtime Environment (build 17.0.1+12-Debian-1deb11u2)
OpenJDK 64-Bit Server VM (build 17.0.1+12-Debian-1deb11u2, mixed mode, sharing)

spark.kubernetes.test.dockerFile is set (Maven)

build/mvn -Dspark.kubernetes.test.dockerFile=resource-managers/kubernetes/docker/src/main/dockerfiles/spark/Dockerfile.java17 -Pkubernetes -Pkubernetes-integration-tests -pl resource-managers/kubernetes/integration-tests integration-test

# Create and login the container.
docker run -it <hash> /bin/bash

185@be35bde0ebfa:/opt/spark/work-dir$ java -version
openjdk version "17.0.1" 2021-10-19
OpenJDK Runtime Environment (build 17.0.1+12-Debian-1deb11u2)
OpenJDK 64-Bit Server VM (build 17.0.1+12-Debian-1deb11u2, mixed mode, sharing)

Both spark.kubernetes.test.javaImageTag and spark.kubernetes.test.dockerFile are set (SBT)

$ build/sbt -Dspark.kubernetes.test.dockerFile=resource-managers/kubernetes/docker/src/main/dockerfiles/spark/Dockerfile.java17 -Dspark.kubernetes.test.javaImageTag=11-jre-slim -Pkubernetes  -Pkubernetes-integration-tests package "kubernetes-integration-tests/test"

# Create and login the container.
docker run -it <hash> /bin/bash

185@7ed1891b2921:/opt/spark/work-dir$ java -version
openjdk version "17.0.1" 2021-10-19
OpenJDK Runtime Environment (build 17.0.1+12-Debian-1deb11u2)
OpenJDK 64-Bit Server VM (build 17.0.1+12-Debian-1deb11u2, mixed mode, sharing)

Both spark.kubernetes.test.javaImageTag and spark.kubernetes.test.dockerFile are set (Maven)

$ build/mvn -Dspark.kubernetes.test.dockerFile=resource-managers/kubernetes/docker/src/main/dockerfiles/spark/Dockerfile.java17 -Dspark.kubernetes.test.javaImageTag=11-jre-slim -Pkubernetes -Pkubernetes-integration-tests -pl resource-managers/kubernetes/integration-tests integration-test

# Create and login the container.
docker run -it <hash> /bin/bash

185@84f3a8b7b4a6:/opt/spark/work-dir$ java -version
openjdk version "17.0.1" 2021-10-19
OpenJDK Runtime Environment (build 17.0.1+12-Debian-1deb11u2)
OpenJDK 64-Bit Server VM (build 17.0.1+12-Debian-1deb11u2, mixed mode, sharing)

SparkQA · 2021-12-03T07:22:56Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50360/

SparkQA · 2021-12-03T08:24:20Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50360/

SparkQA · 2021-12-03T09:53:16Z

Test build #145885 has finished for PR 34790 at commit c497abc.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-12-03T22:19:10Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50393/

SparkQA · 2021-12-03T23:18:19Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50393/

SparkQA · 2021-12-04T00:17:58Z

Test build #145917 has finished for PR 34790 at commit ce2aa41.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun

Do we need to change more? (since this is WIP)

sarutak · 2021-12-04T06:02:39Z

@dongjoon-hyun Oh, I just forgot to modify the title. Thank you !

dongjoon-hyun · 2021-12-04T08:40:03Z

resource-managers/kubernetes/integration-tests/scripts/setup-integration-test-env.sh

      SPARK_TGZ="$2"
      shift
      ;;
+    --docker-file)


Do we need to update resource-managers/kubernetes/integration-tests/README.md?

dongjoon-hyun

+1, LGTM. Thank you so much, @sarutak .

I also verified that the test runner works correctly on SBT, Zulu Java 17, Java 17 Docker Image, Mac OS (Intel), Minikube v1.24.0, K8s v1.22.3. Although I hit some test failures, those can be handled separately like updating README.md.

$ java -version
openjdk version "17.0.1" 2021-10-19 LTS
OpenJDK Runtime Environment Zulu17.30+15-CA (build 17.0.1+12-LTS)
OpenJDK 64-Bit Server VM Zulu17.30+15-CA (build 17.0.1+12-LTS, mixed mode, sharing)

$ minikube version --short
v1.24.0

$ k version --short
Client Version: v1.22.3
Server Version: v1.22.3

$ build/sbt -Dspark.kubernetes.test.dockerFile=resource-managers/kubernetes/docker/src/main/dockerfiles/spark/Dockerfile.java17 -Pkubernetes -Pkubernetes-integration-tests package "kubernetes-integration-tests/test"
Using /Users/dongjoon/.jenv/versions/zulu17 as default JAVA_HOME.
...
[info] KubernetesSuite:
[info] - Run SparkPi with no resources (14 seconds, 550 milliseconds)
[info] - Run SparkPi with no resources & statefulset allocation (13 seconds, 849 milliseconds)
[info] - Run SparkPi with a very long application name. (14 seconds, 863 milliseconds)
[info] - Use SparkLauncher.NO_RESOURCE (13 seconds, 817 milliseconds)
[info] - Run SparkPi with a master URL without a scheme. (13 seconds, 901 milliseconds)
[info] - Run SparkPi with an argument. (14 seconds, 814 milliseconds)
[info] - Run SparkPi with custom labels, annotations, and environment variables. (13 seconds, 809 milliseconds)
[info] - All pods have the same service account by default (13 seconds, 818 milliseconds)
[info] - Run extraJVMOptions check on driver (7 seconds, 754 milliseconds)
[info] - Run SparkRemoteFileTest using a remote data file *** FAILED *** (3 minutes, 4 seconds)
...
[info] - Verify logging configuration is picked from the provided SPARK_CONF_DIR/log4j.properties (13 seconds, 784 milliseconds)
[info] - Run SparkPi with env and mount secrets. (25 seconds, 40 milliseconds)
[info] - Run PySpark on simple pi.py example (15 seconds, 830 milliseconds)
[info] - Run PySpark to test a pyfiles example (17 seconds, 977 milliseconds)
[info] - Run PySpark with memory customization (15 seconds, 833 milliseconds)
[info] - Run in client mode. (11 seconds, 434 milliseconds)
[info] - Start pod creation from template (13 seconds, 893 milliseconds)
[info] - PVs with local hostpath storage on statefulsets *** FAILED *** (3 minutes, 4 seconds)
...
[info] - PVs with local hostpath and storageClass on statefulsets *** FAILED *** (3 minutes, 4 seconds)
...
[info] - PVs with local storage *** FAILED *** (3 minutes, 5 seconds)
[info] - Launcher client dependencies (41 seconds, 424 milliseconds)
[info] - SPARK-33615: Launcher client archives (26 seconds, 386 milliseconds)
[info] - SPARK-33748: Launcher python client respecting PYSPARK_PYTHON (31 seconds, 927 milliseconds)
[info] - SPARK-33748: Launcher python client respecting spark.pyspark.python and spark.pyspark.driver.python (31 seconds, 580 milliseconds)
[info] - Launcher python client dependencies using a zip file (33 seconds, 228 milliseconds)
[info] - Test basic decommissioning (48 seconds, 505 milliseconds)
[info] - Test basic decommissioning with shuffle cleanup (48 seconds, 677 milliseconds)
[info] - Test decommissioning with dynamic allocation & shuffle cleanups *** FAILED *** (4 minutes, 19 seconds)
...
[info] - Test decommissioning timeouts (47 seconds, 685 milliseconds)
[info] - Run SparkR on simple dataframe.R example *** FAILED *** (3 minutes, 4 seconds)
...
[info] Run completed in 29 minutes, 24 seconds.
[info] Total number of tests run: 30
[info] Suites: completed 1, aborted 0
[info] Tests: succeeded 24, failed 6, canceled 0, ignored 0, pending 0
[info] *** 6 TESTS FAILED ***
[error] Failed tests:
[error] 	org.apache.spark.deploy.k8s.integrationtest.KubernetesSuite
[error] (kubernetes-integration-tests / Test / test) sbt.TestsFailedException: Tests unsuccessful
[error] Total time: 1978 s (32:58), completed Dec 4, 2021, 3:33:51 PM

$ docker images | grep dev
spark-r                                   dev             636af4cb030f   4 minutes ago   1.42GB
spark-py                                  dev             2284e581ac6b   5 minutes ago   1.26GB
spark                                     dev             f45786f9002f   6 minutes ago   916MB

$ docker run -it --rm spark:dev java -version | tail -n3
openjdk version "17.0.1" 2021-10-19
OpenJDK Runtime Environment (build 17.0.1+12-Debian-1deb11u2)
OpenJDK 64-Bit Server VM (build 17.0.1+12-Debian-1deb11u2, mixed mode, sharing)

sarutak · 2021-12-05T08:27:06Z

Thank you @dongjoon-hyun . I'll update the README.md later.
I'll also look into the failed tests.

…h to take a custom Dockerfile ### What changes were proposed in this pull request? This PR changes `dev-run-integration-tests.sh` to allow it to take a custom Dockerfile like #34790 did. With this change, this script accepts `--docker-file` option, which takes a path to a custom Dockerfile. ``` $ ./dev/run-integration-tests.sh --docker-file /path/to/dockerfile ``` ### Why are the changes needed? As of #34790, we can specify a custom Dockerfile by `spark.kubernetes.test.dockerFile` property when we run the K8s integration tests using Maven. We can run the integration test via `dev-run-integration-tests.sh` but there is no way to specify a custom Dockerfile. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Confirmed that the K8s integration tests run with the following command using `Dockerfile.java17`. ``` cd resource-managers/kubernetes/integration-tests ./dev/dev-run-integration-tests.sh --docker-file ../docker/src/main/dockerfiles/spark/Dockerfile.java17 ``` Closes #34818 from sarutak/kube-integration-test-java17-2. Authored-by: Kousuke Saruta <sarutak@oss.nttdata.com> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>

Enable K8s integration tests for Java 17 with Maven.

c497abc

github-actions bot added BUILD KUBERNETES labels Dec 3, 2021

Enable K8s integration tests for Java 17 with SBT.

ce2aa41

sarutak marked this pull request as ready for review December 3, 2021 21:36

dongjoon-hyun reviewed Dec 4, 2021

View reviewed changes

sarutak changed the title ~~[WIP][SPARK-37529][K8S][TESTS] Support K8s integration tests for Java 17~~ [SPARK-37529][K8S][TESTS] Support K8s integration tests for Java 17 Dec 4, 2021

dongjoon-hyun reviewed Dec 4, 2021

View reviewed changes

dongjoon-hyun approved these changes Dec 4, 2021

View reviewed changes

dongjoon-hyun closed this in 32922ee Dec 4, 2021

sarutak mentioned this pull request Dec 6, 2021

[SPARK-37529][K8S][TESTS][FOLLOWUP] Allow dev-run-integration-tests.sh to take a custom Dockerfile #34818

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[SPARK-37529][K8S][TESTS] Support K8s integration tests for Java 17 #34790

[SPARK-37529][K8S][TESTS] Support K8s integration tests for Java 17 #34790

Uh oh!

sarutak commented Dec 3, 2021 •

edited

Loading

Uh oh!

SparkQA commented Dec 3, 2021

Uh oh!

SparkQA commented Dec 3, 2021

Uh oh!

SparkQA commented Dec 3, 2021

Uh oh!

SparkQA commented Dec 3, 2021

Uh oh!

SparkQA commented Dec 3, 2021

Uh oh!

SparkQA commented Dec 4, 2021

Uh oh!

dongjoon-hyun left a comment

Uh oh!

sarutak commented Dec 4, 2021

Uh oh!

dongjoon-hyun Dec 4, 2021

Uh oh!

dongjoon-hyun left a comment •

edited

Loading

Uh oh!

sarutak commented Dec 5, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

[SPARK-37529][K8S][TESTS] Support K8s integration tests for Java 17 #34790

[SPARK-37529][K8S][TESTS] Support K8s integration tests for Java 17 #34790

Uh oh!

Conversation

sarutak commented Dec 3, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

SparkQA commented Dec 3, 2021

Uh oh!

SparkQA commented Dec 3, 2021

Uh oh!

SparkQA commented Dec 3, 2021

Uh oh!

SparkQA commented Dec 3, 2021

Uh oh!

SparkQA commented Dec 3, 2021

Uh oh!

SparkQA commented Dec 4, 2021

Uh oh!

dongjoon-hyun left a comment

Choose a reason for hiding this comment

Uh oh!

sarutak commented Dec 4, 2021

Uh oh!

dongjoon-hyun Dec 4, 2021

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sarutak commented Dec 5, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

sarutak commented Dec 3, 2021 •

edited

Loading

dongjoon-hyun left a comment •

edited

Loading