Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SPARK-38770][K8S] Remove renameMainAppResource from baseDriverContainer #36044

Closed
wants to merge 2 commits into from

Conversation

dcoliversun
Copy link
Contributor

@dcoliversun dcoliversun commented Apr 2, 2022

What changes were proposed in this pull request?

This PR aims to simply steps to re-write primary resource in k8s spark application.

Why are the changes needed?

Re-write primary resource uses renameMainAppResource twice.

Refactoring and renameMainAppResource in configureForJava makes renameMainAppResource in baseDriverContainer useless.

Does this PR introduce any user-facing change?

No

How was this patch tested?

  • Pass the GA.
  • Pass k8s IT.
$ build/sbt -Pkubernetes -Pkubernetes-integration-tests -Dtest.exclude.tags=r -Dspark.kubernetes.test.imageRepo=kubespark "kubernetes-integration-tests/test"
[info] KubernetesSuite:
[info] - Run SparkPi with no resources (17 seconds, 443 milliseconds)
[info] - Run SparkPi with no resources & statefulset allocation (17 seconds, 858 milliseconds)
[info] - Run SparkPi with a very long application name. (30 seconds, 450 milliseconds)
[info] - Use SparkLauncher.NO_RESOURCE (18 seconds, 596 milliseconds)
[info] - Run SparkPi with a master URL without a scheme. (18 seconds, 534 milliseconds)
[info] - Run SparkPi with an argument. (21 seconds, 853 milliseconds)
[info] - Run SparkPi with custom labels, annotations, and environment variables. (14 seconds, 285 milliseconds)
[info] - All pods have the same service account by default (13 seconds, 800 milliseconds)
[info] - Run extraJVMOptions check on driver (7 seconds, 825 milliseconds)
[info] - Run SparkRemoteFileTest using a remote data file (15 seconds, 242 milliseconds)
[info] - Verify logging configuration is picked from the provided SPARK_CONF_DIR/log4j2.properties (15 seconds, 491 milliseconds)
[info] - Run SparkPi with env and mount secrets. (26 seconds, 967 milliseconds)
[info] - Run PySpark on simple pi.py example (20 seconds, 318 milliseconds)
[info] - Run PySpark to test a pyfiles example (25 seconds, 659 milliseconds)
[info] - Run PySpark with memory customization (25 seconds, 608 milliseconds)
[info] - Run in client mode. (14 seconds, 620 milliseconds)
[info] - Start pod creation from template (19 seconds, 916 milliseconds)
[info] - SPARK-38398: Schedule pod creation from template (19 seconds, 966 milliseconds)
[info] - PVs with local hostpath storage on statefulsets (22 seconds, 380 milliseconds)
[info] - PVs with local hostpath and storageClass on statefulsets (26 seconds, 935 milliseconds)
[info] - PVs with local storage (30 seconds, 75 milliseconds)
[info] - Launcher client dependencies (2 minutes, 48 seconds)
[info] - SPARK-33615: Launcher client archives (1 minute, 26 seconds)
[info] - SPARK-33748: Launcher python client respecting PYSPARK_PYTHON (1 minute, 47 seconds)
[info] - SPARK-33748: Launcher python client respecting spark.pyspark.python and spark.pyspark.driver.python (1 minute, 51 seconds)
[info] - Launcher python client dependencies using a zip file (1 minute, 51 seconds)
[info] - Test basic decommissioning (59 seconds, 765 milliseconds)
[info] - Test basic decommissioning with shuffle cleanup (1 minute, 3 seconds)
[info] - Test decommissioning with dynamic allocation & shuffle cleanups (2 minutes, 58 seconds)
[info] - Test decommissioning timeouts (58 seconds, 754 milliseconds)
[info] - SPARK-37576: Rolling decommissioning (1 minute, 15 seconds)
[info] Run completed in 29 minutes, 15 seconds.
[info] Total number of tests run: 31
[info] Suites: completed 1, aborted 0
[info] Tests: succeeded 31, failed 0, canceled 0, ignored 0, pending 0
[info] All tests passed.
[success] Total time: 2020 s (33:40), completed 2022-4-2 12:35:52

PS. #23546 introduces deleted code and DepsTestsSuite. DepsTestsSuite can check re-write primary resource. This PR can pass DepsTestsSuite, which can prove deletion about renameMainAppResource in baseDriverContainer does not affect the process about re-write primary resource.

@dcoliversun
Copy link
Contributor Author

cc @dongjoon-hyun @skonto
It would be good if you could take a look when you have time, thanks!

@dongjoon-hyun dongjoon-hyun changed the title [SPARK-38770][K8S] Simply steps to re-write primary resource in k8s spark application [SPARK-38770][K8S] Remove renameMainAppResource from baseDriverContainer Apr 3, 2022
Copy link
Member

@dongjoon-hyun dongjoon-hyun left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1, LGTM. Thank you, @dcoliversun and @martin-g .
Merged to master for Apache Spark 3.4.

I also verified K8s IT at the last commit.

[info] KubernetesSuite:
[info] - Run SparkPi with no resources (10 seconds, 81 milliseconds)
[info] - Run SparkPi with no resources & statefulset allocation (7 seconds, 874 milliseconds)
[info] - Run SparkPi with a very long application name. (7 seconds, 810 milliseconds)
[info] - Use SparkLauncher.NO_RESOURCE (7 seconds, 823 milliseconds)
[info] - Run SparkPi with a master URL without a scheme. (7 seconds, 827 milliseconds)
[info] - Run SparkPi with an argument. (7 seconds, 810 milliseconds)
[info] - Run SparkPi with custom labels, annotations, and environment variables. (7 seconds, 843 milliseconds)
[info] - All pods have the same service account by default (7 seconds, 796 milliseconds)
[info] - Run extraJVMOptions check on driver (3 seconds, 705 milliseconds)
[info] - Run SparkRemoteFileTest using a remote data file (7 seconds, 882 milliseconds)
[info] - Verify logging configuration is picked from the provided SPARK_CONF_DIR/log4j2.properties (8 seconds, 221 milliseconds)
[info] - Run SparkPi with env and mount secrets. (15 seconds, 827 milliseconds)
[info] - Run PySpark on simple pi.py example (7 seconds, 956 milliseconds)
[info] - Run PySpark to test a pyfiles example (9 seconds, 856 milliseconds)
[info] - Run PySpark with memory customization (7 seconds, 792 milliseconds)
[info] - Run in client mode. (6 seconds, 281 milliseconds)
[info] - Start pod creation from template (7 seconds, 860 milliseconds)
[info] - SPARK-38398: Schedule pod creation from template (7 seconds, 784 milliseconds)
[info] - Test basic decommissioning (40 seconds, 164 milliseconds)
[info] - Test basic decommissioning with shuffle cleanup (40 seconds, 358 milliseconds)
[info] *** Test still running after 2 minutes, 35 seconds: suite name: KubernetesSuite, test name: Test decommissioning with dynamic allocation & shuffle cleanups.
[info] - Test decommissioning with dynamic allocation & shuffle cleanups (2 minutes, 39 seconds)
[info] - Test decommissioning timeouts (39 seconds, 967 milliseconds)
[info] - SPARK-37576: Rolling decommissioning (1 minute, 5 seconds)
[info] - Run SparkR on simple dataframe.R example (11 seconds, 109 milliseconds)

@dcoliversun
Copy link
Contributor Author

😁 Thanks for your help @dongjoon-hyun @martin-g

@dcoliversun dcoliversun deleted the SPARK-38770 branch April 3, 2022 11:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants