How to support Spark 2.x

[qiuzhanghua]

when I run Spark 2.2.1 pre-build with Hadoop 2.7 , it failed.

java.lang.IllegalStateException: Failed to execute CommandLineRunner
	at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:735) [spring-boot-1.5.8.RELEASE.jar!/:1.5.8.RELEASE]
	at org.springframework.boot.SpringApplication.callRunners(SpringApplication.java:716) [spring-boot-1.5.8.RELEASE.jar!/:1.5.8.RELEASE]
	at org.springframework.boot.SpringApplication.afterRefresh(SpringApplication.java:703) [spring-boot-1.5.8.RELEASE.jar!/:1.5.8.RELEASE]
	at org.springframework.boot.SpringApplication.run(SpringApplication.java:304) [spring-boot-1.5.8.RELEASE.jar!/:1.5.8.RELEASE]
	at org.springframework.boot.SpringApplication.run(SpringApplication.java:1118) [spring-boot-1.5.8.RELEASE.jar!/:1.5.8.RELEASE]
	at org.springframework.boot.SpringApplication.run(SpringApplication.java:1107) [spring-boot-1.5.8.RELEASE.jar!/:1.5.8.RELEASE]
	at org.springframework.cloud.task.app.spark.client.SparkClientTaskApplication.main(SparkClientTaskApplication.java:29) [classes!/:1.3.0.RELEASE]
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_162]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_162]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_162]
	at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_162]
	at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:48) [spark-client-task-1.3.0.RELEASE.jar:1.3.0.RELEASE]
	at org.springframework.boot.loader.Launcher.launch(Launcher.java:87) [spark-client-task-1.3.0.RELEASE.jar:1.3.0.RELEASE]
	at org.springframework.boot.loader.Launcher.launch(Launcher.java:50) [spark-client-task-1.3.0.RELEASE.jar:1.3.0.RELEASE]
	at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:51) [spark-client-task-1.3.0.RELEASE.jar:1.3.0.RELEASE]
Caused by: java.lang.RuntimeException: Spark Application failed
	at org.springframework.cloud.task.app.spark.client.SparkClientTaskConfiguration$SparkAppClientRunner.run(SparkClientTaskConfiguration.java:90) ~[spring-cloud-starter-task-spark-client-1.3.0.RELEASE.jar!/:1.3.0.RELEASE]
	at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:732) [spring-boot-1.5.8.RELEASE.jar!/:1.5.8.RELEASE]
	... 14 common frames omitted
Caused by: java.lang.NoClassDefFoundError: org/apache/spark/sql/SparkSession
	at org.apache.spark.examples.JavaSparkPi.main(JavaSparkPi.java:36) ~[spark-examples_2.11-2.1.1.jar:2.1.1]
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_162]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_162]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_162]
	at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_162]
	at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:731) ~[spark-core_2.10-1.6.3.jar!/:1.6.3]
	at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181) ~[spark-core_2.10-1.6.3.jar!/:1.6.3]
	at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206) ~[spark-core_2.10-1.6.3.jar!/:1.6.3]
	at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121) ~[spark-core_2.10-1.6.3.jar!/:1.6.3]
	at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) ~[spark-core_2.10-1.6.3.jar!/:1.6.3]
	at org.springframework.cloud.task.app.spark.client.SparkClientTaskConfiguration$SparkAppClientRunner.run(SparkClientTaskConfiguration.java:87) ~[spring-cloud-starter-task-spark-client-1.3.0.RELEASE.jar!/:1.3.0.RELEASE]
	... 15 common frames omitted
Caused by: java.lang.ClassNotFoundException: org.apache.spark.sql.SparkSession
	at java.net.URLClassLoader.findClass(URLClassLoader.java:381) ~[na:1.8.0_162]
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424) ~[na:1.8.0_162]
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ~[na:1.8.0_162]
	... 26 common frames omitted

[qiuzhanghua]

java -jar ~/.m2/repository/org/springframework/cloud/task/app/spark-client-task/1.3.0.RELEASE/spark-client-task-1.3.0.RELEASE.jar --spark.app-class=org.apache.spark.examples.JavaSparkPi --spark.app-jar=examples/jars/spark-examples_2.11-2.1.1.jar --spark.app-args=10

[sabbyanandan]

Hi, @qiuzhanghua. The current release is compatible with spark-core_2.10. If you'd like to upgrade it, you can change the dependency and follow the build procedure to build it locally.

Also, it'd be great if you can contribute it back to the project. Thanks for your interest!

[qiuzhanghua]

Hi, @sabbyanandan .Please see https://github.com/qiuzhanghua/spark-client , I try spark 2.2.1 successfully.
Sorry I don't have more time merge it to your project structure.

[sabbyanandan]

Thanks! We will take a look.