[SPARK-32105][SQL]Refactor current ScriptTransformationExec code #27983

AngersZhuuuu · 2020-03-23T06:06:33Z

What changes were proposed in this pull request?

Renamed hive transform scrip class hive/execution/ScriptTransformationExec to hive/execution/HiveScriptTransformationExec (don't rename file)
Extract class BaseScriptTransformationExec about common code used across SparkScriptTransformationExec(next pr add this) and HiveScriptTransformationExec
Extract class BaseScriptTransformationWriterThread of writing data thread across SparkScriptTransformationWriterThread(added next for support transform in sql/core ) and HiveScriptTransformationWriterThread ,
HiveScriptTransformationWriterThread additionally supports Hive serde format
Rename current Script strategies in hive module to HiveScript, in next pr will add SparkScript strategies for support transform in sql/core.

Todo List;

Support transform in sql/core base on BaseScriptTransformationExec, which would run script operator in SQL mode (without Hive).
The output of script would be read as a string and column values are extracted by using a delimiter (default : tab character)
For Hive, by default only serde's must be used, and without hive we can run without serde
Cleanup past hacks that are observed (and people suggest / report)， such as
- Solve string value error about Date/Timestamp in ScriptTransform
- support use transform with aggregation
- support array/map as transform's input
Use code-gen projection to serialize rows to output stream()

Why are the changes needed?

Support run transform in SQL mode without hive

Does this PR introduce any user-facing change?

Yes

How was this patch tested?

Added UT

AngersZhuuuu · 2020-03-23T06:07:26Z

...ore/src/test/scala/org/apache/spark/sql/execution/script/ScriptTrnasformationExecSuite.scala

+        (3, "3", 3.0, BigDecimal(3.0), new Timestamp(3), Date.valueOf("2015-05-23"))
+      ).toDF("a", "b", "c", "d", "e", "f") // Note column d's data type is Decimal(38, 18)
+      df.createTempView("v")
+


Here add test case of Timestamp and DateType

AngersZhuuuu · 2020-03-23T06:10:03Z

@gatorsmile @tejasapatil @wangyum

I have restart implement ScriptTransform in sql/core now,

Rewrite origin pr in current code. Hope for your review and suggestion

dongjoon-hyun · 2020-04-01T22:22:53Z

Hi, @AngersZhuuuu . This is too big as a follow-up. Please create a new JIRA issue for this.

dongjoon-hyun · 2020-04-01T22:22:59Z

ok to test

SparkQA · 2020-04-02T02:42:32Z

Test build #120696 has finished for PR 27983 at commit 0d9c437.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

AngersZhuuuu · 2020-04-02T02:57:17Z

Hi, @AngersZhuuuu . This is too big as a follow-up. Please create a new JIRA issue for this.

Seems not need to create new jira? just delete follow-up?

AngersZhuuuu · 2020-04-02T03:23:40Z

@dongjoon-hyun
In latest test,

org.apache.spark.sql.execution.script.ScriptTransformationSuite.SPARK-14400 script transformation should fail for bad script command

Seem sometimes it won't catch the subprocess's exception.
I tried a lot of times, sometimes failed, sometimes success. May have problem about catch subproc's exception.

SparkQA · 2020-05-02T06:41:30Z

Test build #122194 has finished for PR 27983 at commit 6fcb1af.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-05-02T16:26:03Z

Test build #122214 has finished for PR 27983 at commit 1e7c3df.

This patch fails from timeout after a configured wait of 400m.
This patch merges cleanly.
This patch adds no public classes.

AngersZhuuuu · 2020-05-15T06:10:09Z

cc @cloud-fan Can you take a look?

SparkQA · 2020-05-15T06:19:42Z

Test build #122637 has finished for PR 27983 at commit aea128b.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

AngersZhuuuu · 2020-05-15T06:26:23Z

Test build #122637 has finished for PR 27983 at commit aea128b.

This patch fails Spark unit tests.

This patch merges cleanly.

This patch adds no public classes.

@dongjoon-hyun For the latest failed test, Did I miss something?

AngersZhuuuu · 2020-06-04T03:48:40Z

Retest this please

SparkQA · 2020-06-04T07:05:02Z

Test build #123517 has finished for PR 27983 at commit aea128b.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

AngersZhuuuu · 2020-06-04T07:50:22Z

retest this please

SparkQA · 2020-06-04T12:23:17Z

Test build #123525 has finished for PR 27983 at commit aea128b.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-07-08T07:05:05Z

Test build #125305 has finished for PR 27983 at commit f52f376.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

AngersZhuuuu · 2020-07-08T07:25:21Z

retest this please

SparkQA · 2020-07-08T11:07:05Z

Test build #125324 has finished for PR 27983 at commit f52f376.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

maropu · 2020-07-08T12:32:39Z

sql/core/src/main/scala/org/apache/spark/sql/execution/BaseScriptTransform.scala

+import org.apache.spark.sql.types.DataType
+import org.apache.spark.util.{CircularBuffer, SerializableConfiguration, Utils}
+
+trait BaseScriptTransform extends UnaryExecNode {


Ah, my bad. Probably, BaseScriptTransformationExec is better.. #27983 (comment)

Ah, my bad. Probably, BaseScriptTransformationExec is better.. #27983 (comment)

Yea, done

maropu · 2020-07-08T12:55:44Z

Could you update the PR description, too?

AngersZhuuuu · 2020-07-08T23:58:39Z

Could you update the PR description, too?

Done, sorry for always forgot to do that....

SparkQA · 2020-07-09T00:07:23Z

Test build #125424 has finished for PR 27983 at commit 28cb9ae.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

maropu · 2020-07-09T01:07:08Z

Could you check the comment, too? #27983 (comment)

maropu · 2020-07-09T01:08:02Z

Could you update the PR description, too?

Done, sorry for always forgot to do that....

Renamed hive transform scrip class hive/execution/ScriptTransformationExec to hive/execution/script/HiveScriptTransformationExec (don't rename file)

/script/?

SparkQA · 2020-07-09T07:05:02Z

Test build #125431 has finished for PR 27983 at commit fcb0957.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-07-09T07:05:02Z

Test build #125428 has finished for PR 27983 at commit e5cd3d8.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

cloud-fan · 2020-07-09T07:44:20Z

retest this please

SparkQA · 2020-07-09T10:42:38Z

Test build #125452 has finished for PR 27983 at commit fcb0957.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

maropu · 2020-07-09T10:46:07Z

retest this please

SparkQA · 2020-07-09T14:45:39Z

Test build #125461 has finished for PR 27983 at commit fcb0957.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

cloud-fan · 2020-07-09T15:42:17Z

retest this please

SparkQA · 2020-07-09T15:45:13Z

Test build #125492 has started for PR 27983 at commit fcb0957.

maropu · 2020-07-10T01:37:39Z

retest this please

SparkQA · 2020-07-10T06:41:19Z

Test build #125537 has finished for PR 27983 at commit fcb0957.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

cloud-fan · 2020-07-10T06:52:28Z

retest this please

SparkQA · 2020-07-10T07:05:03Z

Test build #125567 has finished for PR 27983 at commit fcb0957.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

maropu · 2020-07-10T08:33:37Z

retest this please

SparkQA · 2020-07-10T08:36:06Z

Test build #125587 has started for PR 27983 at commit fcb0957.

maropu · 2020-07-10T22:24:53Z

retest this please

SparkQA · 2020-07-11T06:01:33Z

Test build #125645 has finished for PR 27983 at commit fcb0957.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

cloud-fan · 2020-07-13T08:58:18Z

thanks, merging to master!

AngersZhuuuu added 6 commits March 21, 2020 23:12

save

a20ef5f

Update ScriptTransformbase.scala

47f2150

Update ScriptTransformationExec.scala

479bd69

fix error

890b45c

fix time stamp type

2ebc702

save

0d9c437

AngersZhuuuu commented Mar 23, 2020

View reviewed changes

AngersZhuuuu changed the title ~~[SPARK-15694][SQL][FOLLOW-UP] Implement ScriptTransformation in sql/core (part 1)~~ [SPARK-15694][SQL][FOLLOW-UP] Implement ScriptTransformation in sql/core Mar 23, 2020

dongjoon-hyun added the SQL label Apr 1, 2020

AngersZhuuuu changed the title ~~[SPARK-15694][SQL][FOLLOW-UP] Implement ScriptTransformation in sql/core~~ [SPARK-15694][SQL]Implement ScriptTransformation in sql/core Apr 2, 2020

AngersZhuuuu mentioned this pull request Apr 2, 2020

[SPARK-30973][SQL]ScriptTransformationExec should wait for the termination #28098

Closed

Update ScriptTransformbase.scala

6fcb1af

Update ScriptTransformbase.scala

1e7c3df

AngersZhuuuu added 2 commits May 15, 2020 09:38

Merge branch 'master' into follow_spark_15694

7fcccc9

update

aea128b

maropu reviewed Jul 8, 2020

View reviewed changes

follow commnet

e5cd3d8

AngersZhuuuu force-pushed the follow_spark_15694 branch from 185f26c to e5cd3d8 Compare July 9, 2020 02:41

Update BaseScriptTransformationExec.scala

fcb0957

cloud-fan closed this in 6d49964 Jul 13, 2020

[SPARK-32105][SQL]Refactor current ScriptTransformationExec code #27983

[SPARK-32105][SQL]Refactor current ScriptTransformationExec code #27983

Uh oh!

Conversation

AngersZhuuuu commented Mar 23, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

AngersZhuuuu Mar 23, 2020

Choose a reason for hiding this comment

Uh oh!

AngersZhuuuu commented Mar 23, 2020

Uh oh!

dongjoon-hyun commented Apr 1, 2020

Uh oh!

dongjoon-hyun commented Apr 1, 2020

Uh oh!

SparkQA commented Apr 2, 2020

Uh oh!

AngersZhuuuu commented Apr 2, 2020

Uh oh!

AngersZhuuuu commented Apr 2, 2020

Uh oh!

SparkQA commented May 2, 2020

Uh oh!

SparkQA commented May 2, 2020

Uh oh!

AngersZhuuuu commented May 15, 2020

Uh oh!

SparkQA commented May 15, 2020

Uh oh!

AngersZhuuuu commented May 15, 2020

Uh oh!

AngersZhuuuu commented Jun 4, 2020

Uh oh!

SparkQA commented Jun 4, 2020

Uh oh!

AngersZhuuuu commented Jun 4, 2020

Uh oh!

SparkQA commented Jun 4, 2020

Uh oh!

SparkQA commented Jul 8, 2020

Uh oh!

AngersZhuuuu commented Jul 8, 2020

Uh oh!

SparkQA commented Jul 8, 2020

Uh oh!

maropu Jul 8, 2020

Choose a reason for hiding this comment

Uh oh!

AngersZhuuuu Jul 8, 2020

Choose a reason for hiding this comment

Uh oh!

maropu commented Jul 8, 2020

Uh oh!

AngersZhuuuu commented Jul 8, 2020

Uh oh!

SparkQA commented Jul 9, 2020

Uh oh!

maropu commented Jul 9, 2020

Uh oh!

maropu commented Jul 9, 2020

Uh oh!

SparkQA commented Jul 9, 2020

Uh oh!

SparkQA commented Jul 9, 2020

Uh oh!

cloud-fan commented Jul 9, 2020

Uh oh!

SparkQA commented Jul 9, 2020

Uh oh!

maropu commented Jul 9, 2020

Uh oh!

SparkQA commented Jul 9, 2020

Uh oh!

cloud-fan commented Jul 9, 2020

Uh oh!

AngersZhuuuu commented Mar 23, 2020 •

edited

Loading