Skip to content

[SPARK-6392][SQL]Minor fix ClassNotFound exception when use spark cli to add jar #5079

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
jeanlyn wants to merge 3 commits into from
Closed

[SPARK-6392][SQL]Minor fix ClassNotFound exception when use spark cli to add jar #5079

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
ca95849
use spark physical plan when add jar
jeanlyn Mar 18, 2015
ca78d72
add test
jeanlyn Mar 18, 2015
2de3945
rename udf
jeanlyn Mar 19, 2015
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 4 additions & 2 deletions ...hriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/SparkSQLCLIDriver.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ import org.apache.hadoop.hive.common.{HiveInterruptCallback, HiveInterruptUtils,
import org.apache.hadoop.hive.conf.HiveConf
import org.apache.hadoop.hive.ql.Driver
import org.apache.hadoop.hive.ql.exec.Utilities
import org.apache.hadoop.hive.ql.processors.{SetProcessor, CommandProcessor, CommandProcessorFactory}
import org.apache.hadoop.hive.ql.processors.{AddResourceProcessor, SetProcessor, CommandProcessor, CommandProcessorFactory}
import org.apache.hadoop.hive.ql.session.SessionState
import org.apache.hadoop.hive.shims.ShimLoader
import org.apache.thrift.transport.TSocket
Expand Down Expand Up @@ -264,7 +264,9 @@ private[hive] class SparkSQLCLIDriver extends CliDriver with Logging {
val proc: CommandProcessor = HiveShim.getCommandProcessor(Array(tokens(0)), hconf)

if (proc != null) {
if (proc.isInstanceOf[Driver] || proc.isInstanceOf[SetProcessor]) {
if (proc.isInstanceOf[Driver]
|| proc.isInstanceOf[SetProcessor]
|| proc.isInstanceOf[AddResourceProcessor]) {
val driver = new SparkSQLDriver

driver.init()
Expand Down
Binary file added sql/hive-thriftserver/src/test/resources/jar/TestUdf.jar
View file
Open in desktop
Binary file not shown.
24 changes: 24 additions & 0 deletions sql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/CliSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -123,4 +123,28 @@ class CliSuite extends FunSuite with BeforeAndAfterAll with Logging {
test("Single command with -e") {
runCliWithin(1.minute, Seq("-e", "SHOW DATABASES;"))("" -> "OK")
}

test("SPARK-6392:add jar") {
val dataFilePath =
Thread.currentThread().getContextClassLoader.getResource("data/files/small_kv.txt")
val jarPath =
Thread.currentThread().getContextClassLoader.getResource("jar/TestUdf.jar").toString.split(":")(1)

runCliWithin(3.minute)(
s"""
|CREATE TABLE hive_test(key INT, val STRING);
|LOAD DATA LOCAL INPATH '$dataFilePath' OVERWRITE INTO TABLE hive_test;
|""".stripMargin
-> "OK",
s"""
|add jar $jarPath ;
|create temporary function hello as 'hello';
|""".stripMargin
-> "Time taken: ",
"select hello(count(*)) from hive_test;"
-> "hello 5",
"DROP TABLE hive_test;"
-> "Time taken: "
)
}
}