Skip to content

[SPARK-7322][SQL] Window functions in DataFrame #6343

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
rxin wants to merge 17 commits into from
Closed

[SPARK-7322][SQL] Window functions in DataFrame #6343

rxin wants to merge 17 commits into from

Conversation

@rxin
Copy link
Contributor

@rxin rxin commented May 22, 2015

This closes #6104.

rxin
rxin reviewed May 22, 2015
View reviewed changes
sql/core/src/main/scala/org/apache/spark/sql/Column.scala
Copy link
Contributor Author

@rxin rxin May 22, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

note: throw an exception if this is a not an aggregate function

@chenghao-intel
Copy link
Contributor

chenghao-intel commented May 22, 2015

The new APIs is much better now, except some small issues, but should be easy to solve.

@SparkQA
Copy link

SparkQA commented May 22, 2015

Test build #33314 has finished for PR 6343 at commit dc448fe.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@rxin
Copy link
Contributor Author

rxin commented May 22, 2015

@davies what do you think the Python API should look like? What's the best way to have a static method on a class (i.e. the ones on Window)

@rxin
Copy link
Contributor Author

rxin commented May 22, 2015

Also Python doesn't have Long.MinValue. What would you recommend?

@SparkQA
Copy link

SparkQA commented May 22, 2015

Test build #33329 has finished for PR 6343 at commit 026d587.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@rxin
Copy link
Contributor Author

rxin commented May 22, 2015

Merging this and I will create a separate ticket for Python.

@asfgit asfgit closed this in f6f2eeb May 22, 2015
asfgit pushed a commit that referenced this pull request May 22, 2015
@chenghao-intel @rxin
[SPARK-7322][SQL] Window functions in DataFrame
bfaf6a0 
This closes #6104.

Author: Cheng Hao <hao.cheng@intel.com>
Author: Reynold Xin <rxin@databricks.com>

Closes #6343 from rxin/window-df and squashes the following commits:

026d587 [Reynold Xin] Address code review feedback.
dc448fe [Reynold Xin] Fixed Hive tests.
9794d9d [Reynold Xin] Moved Java test package.
9331605 [Reynold Xin] Refactored API.
3313e2a [Reynold Xin] Merge pull request #6104 from chenghao-intel/df_window
d625a64 [Cheng Hao] Update the dataframe window API as suggsted
c141fb1 [Cheng Hao] hide all of properties of the WindowFunctionDefinition
3b1865f [Cheng Hao] scaladoc typos
f3fd2d0 [Cheng Hao] polish the unit test
6847825 [Cheng Hao] Add additional analystcs functions
57e3bc0 [Cheng Hao] typos
24a08ec [Cheng Hao] scaladoc
28222ed [Cheng Hao] fix bug of range/row Frame
1d91865 [Cheng Hao] style issue
53f89f2 [Cheng Hao] remove the over from the functions.scala
964c013 [Cheng Hao] add more unit tests and window functions
64e18a7 [Cheng Hao] Add Window Function support for DataFrame

(cherry picked from commit f6f2eeb)
Signed-off-by: Reynold Xin <rxin@databricks.com>
@rxin rxin deleted the window-df branch May 22, 2015 08:21
jeanlyn pushed a commit to jeanlyn/spark that referenced this pull request May 28, 2015
@chenghao-intel @jeanlyn
[SPARK-7322][SQL] Window functions in DataFrame
5ce4ef3 
This closes apache#6104.

Author: Cheng Hao <hao.cheng@intel.com>
Author: Reynold Xin <rxin@databricks.com>

Closes apache#6343 from rxin/window-df and squashes the following commits:

026d587 [Reynold Xin] Address code review feedback.
dc448fe [Reynold Xin] Fixed Hive tests.
9794d9d [Reynold Xin] Moved Java test package.
9331605 [Reynold Xin] Refactored API.
3313e2a [Reynold Xin] Merge pull request apache#6104 from chenghao-intel/df_window
d625a64 [Cheng Hao] Update the dataframe window API as suggsted
c141fb1 [Cheng Hao] hide all of properties of the WindowFunctionDefinition
3b1865f [Cheng Hao] scaladoc typos
f3fd2d0 [Cheng Hao] polish the unit test
6847825 [Cheng Hao] Add additional analystcs functions
57e3bc0 [Cheng Hao] typos
24a08ec [Cheng Hao] scaladoc
28222ed [Cheng Hao] fix bug of range/row Frame
1d91865 [Cheng Hao] style issue
53f89f2 [Cheng Hao] remove the over from the functions.scala
964c013 [Cheng Hao] add more unit tests and window functions
64e18a7 [Cheng Hao] Add Window Function support for DataFrame
jeanlyn pushed a commit to jeanlyn/spark that referenced this pull request Jun 12, 2015
@chenghao-intel @jeanlyn
[SPARK-7322][SQL] Window functions in DataFrame
4b5a00c 
This closes apache#6104.

Author: Cheng Hao <hao.cheng@intel.com>
Author: Reynold Xin <rxin@databricks.com>

Closes apache#6343 from rxin/window-df and squashes the following commits:

026d587 [Reynold Xin] Address code review feedback.
dc448fe [Reynold Xin] Fixed Hive tests.
9794d9d [Reynold Xin] Moved Java test package.
9331605 [Reynold Xin] Refactored API.
3313e2a [Reynold Xin] Merge pull request apache#6104 from chenghao-intel/df_window
d625a64 [Cheng Hao] Update the dataframe window API as suggsted
c141fb1 [Cheng Hao] hide all of properties of the WindowFunctionDefinition
3b1865f [Cheng Hao] scaladoc typos
f3fd2d0 [Cheng Hao] polish the unit test
6847825 [Cheng Hao] Add additional analystcs functions
57e3bc0 [Cheng Hao] typos
24a08ec [Cheng Hao] scaladoc
28222ed [Cheng Hao] fix bug of range/row Frame
1d91865 [Cheng Hao] style issue
53f89f2 [Cheng Hao] remove the over from the functions.scala
964c013 [Cheng Hao] add more unit tests and window functions
64e18a7 [Cheng Hao] Add Window Function support for DataFrame
nemccarthy pushed a commit to nemccarthy/spark that referenced this pull request Jun 19, 2015
@chenghao-intel @nemccarthy
[SPARK-7322][SQL] Window functions in DataFrame
1bc0d0c 
This closes apache#6104.

Author: Cheng Hao <hao.cheng@intel.com>
Author: Reynold Xin <rxin@databricks.com>

Closes apache#6343 from rxin/window-df and squashes the following commits:

026d587 [Reynold Xin] Address code review feedback.
dc448fe [Reynold Xin] Fixed Hive tests.
9794d9d [Reynold Xin] Moved Java test package.
9331605 [Reynold Xin] Refactored API.
3313e2a [Reynold Xin] Merge pull request apache#6104 from chenghao-intel/df_window
d625a64 [Cheng Hao] Update the dataframe window API as suggsted
c141fb1 [Cheng Hao] hide all of properties of the WindowFunctionDefinition
3b1865f [Cheng Hao] scaladoc typos
f3fd2d0 [Cheng Hao] polish the unit test
6847825 [Cheng Hao] Add additional analystcs functions
57e3bc0 [Cheng Hao] typos
24a08ec [Cheng Hao] scaladoc
28222ed [Cheng Hao] fix bug of range/row Frame
1d91865 [Cheng Hao] style issue
53f89f2 [Cheng Hao] remove the over from the functions.scala
964c013 [Cheng Hao] add more unit tests and window functions
64e18a7 [Cheng Hao] Add Window Function support for DataFrame
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@rxin @chenghao-intel @SparkQA