[SPARK-16253][SQL] make spark sql compatible with hive sql that using…

[zenglinxi0615]

What changes were proposed in this pull request?

make spark sql compatible with hive sql that using python script transform like using 'xxx.py'

How was this patch tested?

manual tests

[AmplabJenkins]

Can one of the admins verify this patch?

[hvanhovell]

Doesn't this create a huge security hole? Especially because we are not resetting this to its old state.

[zenglinxi0615]

Have you tried it on spark 1.6.2?

[yongliangliu]

I know how to deal with it.
i used the fllow code like this

add file /temp/test.py;
select transform(var1) using '/temp/test.py' as (var1_new)  from test_table;

the error is "Error: /bin/bash: test.py: command not found"
but when i use this and do not change the source code ,it's work,i was so stupid.

add file /temp/test.py;
select transform(var1) using 'python /temp/test.py' as (var1_new)  from test_table;

so there is not bug and no need to change source code

[zenglinxi0615]

sorry for long time no response.
yes, you are right, when you can change the sql from using '/temp/test.py' to using 'python /temp/test.py', there's no need for changing the spark source code.
However, this patch is used for the case when there are already many hive sql which using '/temp/test.py', it cost too much time for modifing these hive sql, so we want to make spark sql compatible with hive sql that using python script transform like using 'xxx.py'.

[HyukjinKwon]

@zenglinxi0615 Is it still active? I wonder if there is any argument against #14686 (comment). Otherwise, I would rather like to suggest to close this.