[SPARK-28525][DEPLOY] Allow Launcher to be applied Java options #25265

sarutak · 2019-07-26T17:11:02Z

Launcher is implemented as a Java application and sometimes I'd like to apply Java options.
One situation I have met is the time I try to attach debugger to Launcher.

Launcher is launched from bin/spark-class but there is no room to apply Java options.

build_command() {
  "$RUNNER" -Xmx128m -cp "$LAUNCH_CLASSPATH" org.apache.spark.launcher.Main "$@"
  printf "%d\0" $?
}

Considering that it's not so many times to apply Java options to Launcher, one compromise would just modify spark-class by user like as follows.

build_command() {
  "$RUNNER" -Xmx128m $SPARK_LAUNCHER_OPTS -cp "$LAUNCH_CLASSPATH" org.apache.spark.launcher.Main "$@"
  printf "%d\0" $?
}

But it doesn't work when any text related to Java options is output to standard output because whole output is used as command-string for spark-shell and spark-submit in current implementation.

One example is jdwp. When apply agentlib option to use jdwp for debug, we will get output like as follows.

Listening for transport dt_socket at address: 9876

The output shown above is not a command-string so spark-submit and spark-shell will fail.
To enable Java options for Launcher, we need treat command-string and others.

I changed launcher/Main.java and bin/spark-class to print separator-character and treat it.

How was this patch tested?

Tested manually using Spark Shell with / without LAUNCHER_JAVA_OPTIONS like as follows.

SPARK_LAUNCHER_OPTS="-agentlib:jdwp=transport=dt_socket,suspend=y,address=localhost:9876,server=y" bin/spark-shell

… Java options.

SparkQA · 2019-07-26T21:15:51Z

Test build #108234 has finished for PR 25265 at commit bafd2a6.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-07-27T14:57:29Z

Test build #108250 has finished for PR 25265 at commit 9e8a06a.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

sarutak · 2019-07-27T15:23:52Z

retest this please.

SparkQA · 2019-07-27T19:00:59Z

Test build #108252 has finished for PR 25265 at commit 9e8a06a.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

xuanyuanking · 2019-07-29T05:59:23Z

Is the SPARK_SUBMIT_OPTS parameter cover your requirement?

sarutak · 2019-07-29T08:47:50Z

@xuanyuanking Launcher and SparkSubmit run on different JVM processes so we can't use SPARK_SUBMIT_OPTS for this purpose.

vanzin · 2019-07-29T17:34:36Z

bin/spark-class

-  CMD+=("$ARG")
+DELIM=$'\n'
+while IFS= read -d "$DELIM" -r ARG; do
+  if [ -n "$CMD_START_FLAG" ]; then


Better to initialize CMD_START_FLAG to an empty string before the loop.

Or you could use DELIM for the same effect.

Thanks. I'll initialize the flag.

kiszk · 2019-07-29T18:24:36Z

Do we want to update configuration.md?

sarutak · 2019-07-30T03:04:53Z

@kiszk I think this environment variable is for developers so how about update spark-env.sh.template rather?

SparkQA · 2019-07-30T04:47:38Z

Test build #108364 has finished for PR 25265 at commit f151fbf.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

sarutak · 2019-07-30T04:57:59Z

retest this please.

SparkQA · 2019-07-30T07:05:01Z

Test build #108369 has finished for PR 25265 at commit f151fbf.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

sarutak · 2019-07-30T07:13:27Z

retest this please.

SparkQA · 2019-07-30T09:06:46Z

Test build #108372 has finished for PR 25265 at commit f151fbf.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

sarutak · 2019-07-30T11:46:37Z

Lots of flaky tests...

sarutak · 2019-07-30T12:01:49Z

retest this please.

SparkQA · 2019-07-30T14:41:35Z

Test build #108383 has finished for PR 25265 at commit f151fbf.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

vanzin · 2019-07-30T19:45:09Z

Merging to master. (Hopefully nobody will ask for this on Windows...)

maropu · 2019-07-31T03:49:06Z

I fetched this commit and my spark-shell seemed to be broken... do I miss something?

(py27) maropu@~/Repositories/spark/spark-master: (master $%>)$./bin/spark-shell
NOTE: SPARK_PREPEND_CLASSES is set, placing locally compiled Spark classes ahead of assembly.
/Users/maropu/Repositories/spark/spark-master/bin/spark-class: line 96: CMD: bad array subscript
head: illegal line count -- -1

kiszk · 2019-07-31T04:05:05Z

In my environment (Ubuntu 18.04), it works well.

$ /usr/bin/env bash --version
GNU bash, version 4.4.19(1)-release (powerpc64le-unknown-linux-gnu)
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>

This is free software; you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

$ git log | head -1
commit dba4375359a2dfed1f009edc3b1bcf6b3253fe02
$ ./bin/spark-shell 
19/07/31 13:01:44 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
Spark context Web UI available at http://foo.trl.ibm.com:4040
Spark context available as 'sc' (master = local[*], app id = local-1564545717751).
Spark session available as 'spark'.
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 3.0.0-SNAPSHOT
      /_/
         
Using Scala version 2.12.8 (OpenJDK 64-Bit Server VM, Java 1.8.0_212)
Type in expressions to have them evaluated.
Type :help for more information.

scala>

maropu · 2019-07-31T04:08:23Z

oh, I see. I'll check again later, thx!

maropu · 2019-07-31T04:26:05Z

It seems my bash is too old;

$bash --version
GNU bash, version 3.2.57(1)-release (x86_64-apple-darwin16)
Copyright (C) 2007 Free Software Foundation, Inc.

sarutak · 2019-07-31T05:46:59Z

It's strange. I use bash of same version for you.

$ bash --version
GNU bash, version 3.2.57(1)-release (x86_64-apple-darwin15)
Copyright (C) 2007 Free Software Foundation, Inc.

But the version of macOS is different. Is this the cause?

vanzin · 2019-07-31T16:34:01Z

/Users/maropu/Repositories/spark/spark-master/bin/spark-class: line 96: CMD: bad array subscript

That's also interesting because that line hasn't changed.

vanzin · 2019-07-31T17:59:18Z

Takeshi, I can see that happening if you have the new script with an old build of the launcher jar. Otherwise, the logic seems right (and works for me).

maropu · 2019-08-01T00:11:55Z

That's it! You're right; I rebuilt the package, then the problem has gone. Thx!

alfozan · 2019-10-25T00:28:41Z

I also had the same issue as @maropu

/Users/maropu/Repositories/spark/spark-master/bin/spark-class: line 96: CMD: bad array subscript

Solution: rebuild

 ./build/mvn -Phive -Phive-thriftserver -DskipTests clean package

jahidhasanlinix · 2021-11-24T21:57:26Z

I have this problem when trying to run spark shell, my java env path seems ok, but its still giving me this error>:
echo $JAVA_HOME
/usr/bin/java:/usr/lib/jvm/java-11-openjdk-amd64

Here's the error while trying to run the shell
#./spark-shell
/usr/local/spark/spark-3.2.0-bin-hadoop3.2/bin/spark-class: line 71: /usr/bin/java:/usr/lib/jvm/java-11-openjdk-amd64/bin/java: No such file or directory
/usr/local/spark/spark-3.2.0-bin-hadoop3.2/bin/spark-class: line 96: CMD: bad array subscript

sarutak · 2021-11-25T00:44:41Z

@jahidhasanlinix
Maybe, your JAVA_HOME is wrong.
Please set /usr/lib/jvm/java-11-openjdk-amd64 to JAVA_HOME.

Fix launcher/Main and bin/spark-class to allow Launcher to be applied…

bafd2a6

… Java options.

dongjoon-hyun added the DEPLOY label Jul 26, 2019

Replace \001 as a separator-character with NULL

9e8a06a

vanzin reviewed Jul 29, 2019

View reviewed changes

Add document for SPARK_LAUNCHER_OPTS in spark-env.sh.template

f151fbf

kiszk approved these changes Jul 30, 2019

View reviewed changes

vanzin closed this in 121f933 Jul 30, 2019

[SPARK-28525][DEPLOY] Allow Launcher to be applied Java options #25265

[SPARK-28525][DEPLOY] Allow Launcher to be applied Java options #25265

Uh oh!

Conversation

sarutak commented Jul 26, 2019

How was this patch tested?

Uh oh!

SparkQA commented Jul 26, 2019

Uh oh!

SparkQA commented Jul 27, 2019

Uh oh!

sarutak commented Jul 27, 2019

Uh oh!

SparkQA commented Jul 27, 2019

Uh oh!

xuanyuanking commented Jul 29, 2019

Uh oh!

sarutak commented Jul 29, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

vanzin Jul 29, 2019

Choose a reason for hiding this comment

Uh oh!

sarutak Jul 30, 2019

Choose a reason for hiding this comment

Uh oh!

kiszk commented Jul 29, 2019

Uh oh!

sarutak commented Jul 30, 2019

Uh oh!

SparkQA commented Jul 30, 2019

Uh oh!

sarutak commented Jul 30, 2019

Uh oh!

SparkQA commented Jul 30, 2019

Uh oh!

sarutak commented Jul 30, 2019

Uh oh!

SparkQA commented Jul 30, 2019

Uh oh!

sarutak commented Jul 30, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sarutak commented Jul 30, 2019

Uh oh!

SparkQA commented Jul 30, 2019

Uh oh!

vanzin commented Jul 30, 2019

Uh oh!

maropu commented Jul 31, 2019

Uh oh!

kiszk commented Jul 31, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

maropu commented Jul 31, 2019

Uh oh!

maropu commented Jul 31, 2019

Uh oh!

sarutak commented Jul 31, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

vanzin commented Jul 31, 2019

Uh oh!

vanzin commented Jul 31, 2019

Uh oh!

maropu commented Aug 1, 2019

Uh oh!

alfozan commented Oct 25, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jahidhasanlinix commented Nov 24, 2021

Uh oh!

sarutak commented Nov 25, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

sarutak commented Jul 29, 2019 •

edited

Loading

sarutak commented Jul 30, 2019 •

edited

Loading

kiszk commented Jul 31, 2019 •

edited

Loading

sarutak commented Jul 31, 2019 •

edited

Loading

alfozan commented Oct 25, 2019 •

edited

Loading