[SPARK-18910][SQL]Resolve faile to use UDF that jar file in hdfs.

[shenh062326]

What changes were proposed in this pull request?

When I create a UDF that jar file in hdfs, I can't use the UDF.

spark-sql> create function trans_array as 'com.test.udf.TransArray' using jar 'hdfs://host1:9000/spark/dev/share/libs/spark-proxy-server-biz-service-impl-1.0.0.jar';
spark-sql> describe function trans_array;
Function: test_db.trans_array
Class: com.alipay.spark.proxy.server.biz.service.impl.udf.TransArray
Usage: N/A.
Time taken: 0.127 seconds, Fetched 3 row(s)
spark-sql> select trans_array(1, '\|', id, position) as (id0, position0) from test_spark limit 10;
Error in query: Undefined function: 'trans_array'. This function is neither a registered temporary function nor a permanent function registered in the database 'test_db'.; line 1 pos 7

The reason is when org.apache.spark.sql.internal.SessionState.FunctionResourceLoader.loadResource, the uri.toURL throw exception with " failed unknown protocol: hdfs"

def addJar(path: String): Unit = {
sparkSession.sparkContext.addJar(path)
val uri = new Path(path).toUri
val jarURL = if (uri.getScheme == null) {
// path is a local file path without a URL scheme
new File(path).toURI.toURL
} else {
// path is a URL with a scheme
{color:red}uri.toURL{color}
}
jarClassLoader.addURL(jarURL)
Thread.currentThread().setContextClassLoader(jarClassLoader)
}

I think we should setURLStreamHandlerFactory method on URL with an instance of FsUrlStreamHandlerFactory, just like:

static {
URL.setURLStreamHandlerFactory(new FsUrlStreamHandlerFactory());
}

How was this patch tested?

I have test it in my cluster.

[gatorsmile]

SparkContext might not be the good place. object SharedState might be better.

In addition, you also need to add comments to explain why you did it before this line.

[shenh062326]

Thanks for you comment, I will add it to SharedState.

[gatorsmile]

Please add the description in the JIRA to the PR description. FYI, you still can edit the description after you create the PR.

[SparkQA]

Test build #70296 has finished for PR 16324 at commit a91b08f.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[gatorsmile]

This method can only be called once per JVM, so it is typically executed in a static block. This limitation means that if some other part of your program—perhaps a third-party component outside your control— sets a URLStreamHandlerFactory, you won’t be able to use this approach for reading data from Hadoop.

[gatorsmile]

This might not be the right solution to do it, as explained above.

[shenh062326]

Should we download the UDF jar from hdfs.

[gatorsmile]

First, I am not sure whether we should support reading UDF jar from HDFS. cc @rxin

Second, if we want to support it, the best reviewers are @zsxwing @tdas They added the file HDFSMetadataLog.scala recently

[shenh062326]

Currently，we can create a UDF with jar in HDFS, but failed to use it.
Spark driver won't download the jar from HDFS, it only add the path to the classLoader.
If we don't support reading UDF jar from HDFS, we should download the UDF jar.
I think support reading UDF jar from HDFS is better.

[rxin]

This is to allow using jars defined using HDFS-API, not just HDFS right? In that case it sounds like a good idea too ... but we need a test case for it.

[shenh062326]

I‘m sorry, @rxin, I don't understand what you mean.

[rxin]

I was saying we need to create a test case for this change.

[HyukjinKwon]

(gentle ping @shenh062326 )

[melin]

in spark 2.1.0
0: jdbc:hive2://localhost:10000> add jar hdfs:///user/datacompute/datacompute-udf-1.0-
Error: java.net.MalformedURLException: unknown protocol: hdfs (state=,code=0)
0: jdbc:hive2://localhost:10000>