[SPARK-14353] Dataset Time Window `window` API for Python, and SQL #12136

brkyvz · 2016-04-03T08:42:02Z

What changes were proposed in this pull request?

The window function was added to Dataset with this PR.
This PR adds the Python, and SQL, API for this function.

With this PR, SQL, Java, and Scala will share the same APIs as in users can use:

window(timeColumn, windowDuration)
window(timeColumn, windowDuration, slideDuration)
window(timeColumn, windowDuration, slideDuration, startTime)

In Python, users can access all APIs above, but in addition they can do

In Python:
window(timeColumn, windowDuration, startTime=...)

that is, they can provide the startTime without providing the slideDuration. In this case, we will generate tumbling windows.

How was this patch tested?

Unit tests + manual tests

SparkQA · 2016-04-03T08:46:08Z

Test build #54801 has finished for PR 12136 at commit f589469.

This patch fails R style tests.
This patch merges cleanly.
This patch adds no public classes.

rxin · 2016-04-03T18:33:12Z

Can we leave all the streaming stuff out of R for now? I don't know whether it makes sense to provide this functionality in R, and even if we do, the current API might change a lot, so having it in fewer languages would make it easier to change.

rxin · 2016-04-03T18:33:56Z

Ah OK this "window" thing is not streaming specific. I'd still consider breaking them into two separate prs so we can get the python thing in sooner.

SparkQA · 2016-04-03T18:43:17Z

Test build #54807 has finished for PR 12136 at commit 12e1841.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-04-04T03:59:10Z

Test build #54820 has finished for PR 12136 at commit e2aa616.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

brkyvz · 2016-04-04T22:44:38Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/trees/TreeNode.scala

+    } else {
+      newArgs ++ otherCopyArgs
+    }
+    val defaultCtor = ctors.find { ctor =>


we need a smarter way to pick the constructor, or else the two constructors for TimeWindow cause issues depending on Java version

SparkQA · 2016-04-05T00:02:15Z

Test build #54906 has finished for PR 12136 at commit a59ad58.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

brkyvz · 2016-04-05T00:58:57Z

@davies Addressed your comments

SparkQA · 2016-04-05T01:30:49Z

Test build #54912 has finished for PR 12136 at commit 71fa73f.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-04-05T01:47:22Z

Test build #54913 has finished for PR 12136 at commit c891d75.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-04-05T03:55:48Z

Test build #54933 has finished for PR 12136 at commit 9a3a47f.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-04-05T06:12:43Z

Test build #54952 has finished for PR 12136 at commit 8dec7ca.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-04-05T07:46:56Z

Test build #54961 has finished for PR 12136 at commit 891f448.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-04-05T18:24:08Z

Test build #54999 has finished for PR 12136 at commit 1bd7563.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-04-05T20:08:44Z

Test build #55008 has finished for PR 12136 at commit 68b9de2.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

davies · 2016-04-05T20:18:15Z

LGTM, merging this into master, thanks!

…n Exception of Function Lookup ### What changes were proposed in this pull request? When the exception is an invocation exception during function lookup, we return a useless/confusing error message: For example, ```Scala df.selectExpr("concat_ws()") ``` Below is the error message we got: ``` null; line 1 pos 0 org.apache.spark.sql.AnalysisException: null; line 1 pos 0 ``` To get the meaningful error message, we need to get the cause. The fix is exactly the same as what we did in #12136. After the fix, the message we got is the exception issued in the constuctor of function implementation: ``` requirement failed: concat_ws requires at least one argument.; line 1 pos 0 org.apache.spark.sql.AnalysisException: requirement failed: concat_ws requires at least one argument.; line 1 pos 0 ``` ### How was this patch tested? Added test cases. Author: gatorsmile <gatorsmile@gmail.com> Closes #15878 from gatorsmile/functionNotFound. (cherry picked from commit 86430cc) Signed-off-by: Reynold Xin <rxin@databricks.com>

…n Exception of Function Lookup ### What changes were proposed in this pull request? When the exception is an invocation exception during function lookup, we return a useless/confusing error message: For example, ```Scala df.selectExpr("concat_ws()") ``` Below is the error message we got: ``` null; line 1 pos 0 org.apache.spark.sql.AnalysisException: null; line 1 pos 0 ``` To get the meaningful error message, we need to get the cause. The fix is exactly the same as what we did in #12136. After the fix, the message we got is the exception issued in the constuctor of function implementation: ``` requirement failed: concat_ws requires at least one argument.; line 1 pos 0 org.apache.spark.sql.AnalysisException: requirement failed: concat_ws requires at least one argument.; line 1 pos 0 ``` ### How was this patch tested? Added test cases. Author: gatorsmile <gatorsmile@gmail.com> Closes #15878 from gatorsmile/functionNotFound.

…g an Invocation Exception of Function Lookup ### What changes were proposed in this pull request? This PR is to backport #15878 When the exception is an invocation exception during function lookup, we return a useless/confusing error message: For example, ```Scala df.selectExpr("concat_ws()") ``` Below is the error message we got: ``` null; line 1 pos 0 org.apache.spark.sql.AnalysisException: null; line 1 pos 0 ``` To get the meaningful error message, we need to get the cause. The fix is exactly the same as what we did in #12136. After the fix, the message we got is the exception issued in the constuctor of function implementation: ``` requirement failed: concat_ws requires at least one argument.; line 1 pos 0 org.apache.spark.sql.AnalysisException: requirement failed: concat_ws requires at least one argument.; line 1 pos 0 ``` ### How was this patch tested? Added test cases. Author: gatorsmile <gatorsmile@gmail.com> Closes #15902 from gatorsmile/functionNotFound20.

…n Exception of Function Lookup ### What changes were proposed in this pull request? When the exception is an invocation exception during function lookup, we return a useless/confusing error message: For example, ```Scala df.selectExpr("concat_ws()") ``` Below is the error message we got: ``` null; line 1 pos 0 org.apache.spark.sql.AnalysisException: null; line 1 pos 0 ``` To get the meaningful error message, we need to get the cause. The fix is exactly the same as what we did in apache#12136. After the fix, the message we got is the exception issued in the constuctor of function implementation: ``` requirement failed: concat_ws requires at least one argument.; line 1 pos 0 org.apache.spark.sql.AnalysisException: requirement failed: concat_ws requires at least one argument.; line 1 pos 0 ``` ### How was this patch tested? Added test cases. Author: gatorsmile <gatorsmile@gmail.com> Closes apache#15878 from gatorsmile/functionNotFound.

brkyvz added 3 commits April 2, 2016 20:39

added python api for time windowing

4777b77

everything except R

7b0fb13

added window support for sql, python, and R

f589469

fixed R style

12e1841

remove R changes

e2aa616

brkyvz changed the title ~~[SPARK-14353] Dateset Time Window window API for Python, SQL, and R~~ [SPARK-14353] Dataset Time Window window API for Python, and SQL Apr 4, 2016

smarter reflection

a59ad58

brkyvz reviewed Apr 4, 2016
View reviewed changes

brkyvz added 2 commits April 4, 2016 17:12

use existing util method

71fa73f

address comments

c891d75

should be fixed

9a3a47f

handle null

8dec7ca

fix col name

891f448

fix second line

1bd7563

Update functions.py

68b9de2

asfgit closed this in 9ee5c25 Apr 5, 2016

gatorsmile mentioned this pull request Nov 14, 2016

[SPARK-18430] [SQL] Fixed Exception Messages when Hitting an Invocation Exception of Function Lookup #15878

Closed

gatorsmile mentioned this pull request Nov 16, 2016

[SPARK-18430][SQL][BACKPORT-2.0] Fixed Exception Messages when Hitting an Invocation Exception of Function Lookup #15902

Closed

brkyvz deleted the python-windows branch February 3, 2019 20:55

[SPARK-14353] Dataset Time Window window API for Python, and SQL #12136

[SPARK-14353] Dataset Time Window window API for Python, and SQL #12136

Uh oh!

Conversation

brkyvz commented Apr 3, 2016

What changes were proposed in this pull request?

How was this patch tested?

Uh oh!

SparkQA commented Apr 3, 2016

Uh oh!

rxin commented Apr 3, 2016

Uh oh!

rxin commented Apr 3, 2016

Uh oh!

SparkQA commented Apr 3, 2016

Uh oh!

SparkQA commented Apr 4, 2016

Uh oh!

brkyvz Apr 4, 2016

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Apr 5, 2016

Uh oh!

brkyvz commented Apr 5, 2016

Uh oh!

SparkQA commented Apr 5, 2016

Uh oh!

SparkQA commented Apr 5, 2016

Uh oh!

SparkQA commented Apr 5, 2016

Uh oh!

SparkQA commented Apr 5, 2016

Uh oh!

SparkQA commented Apr 5, 2016

Uh oh!

SparkQA commented Apr 5, 2016

Uh oh!

SparkQA commented Apr 5, 2016

Uh oh!

davies commented Apr 5, 2016

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

[SPARK-14353] Dataset Time Window `window` API for Python, and SQL #12136

[SPARK-14353] Dataset Time Window `window` API for Python, and SQL #12136