Skip to content

[SPARK-7320] [SQL] Add Cube / Rollup for dataframe #6304

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
chenghao-intel wants to merge 8 commits into from
Closed

[SPARK-7320] [SQL] Add Cube / Rollup for dataframe #6304

chenghao-intel wants to merge 8 commits into from

Conversation

@chenghao-intel
Copy link
Contributor

@chenghao-intel chenghao-intel commented May 21, 2015

This is a follow up for #6257, which broke the maven test.

Add cube & rollup for DataFrame
For example:

testData.rollup($"a" + $"b", $"b").agg(sum($"a" - $"b"))
testData.cube($"a" + $"b", $"b").agg(sum($"a" - $"b"))

@chenghao-intel chenghao-intel mentioned this pull request May 21, 2015
Closed
@chenghao-intel
Copy link
Contributor Author

chenghao-intel commented May 21, 2015

/cc @yhuai @liancheng

Thanks for the suggestion to move the table registration into beforeAll in test suite.

Let's see what jenkins will say.

@yhuai
Copy link
Contributor

yhuai commented May 21, 2015

I manually tested HiveDataFrameAnalyticsSuite in maven. It is good.

@SparkQA
Copy link

SparkQA commented May 21, 2015

Test build #33191 has finished for PR 6304 at commit 04bb1de.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):
    • class GroupedData protected[sql](
    • public class TaskMemoryManager

@yhuai
Copy link
Contributor

yhuai commented May 21, 2015

Thanks for fixing the test! I am merging it to master and branch 1.4.

asfgit pushed a commit that referenced this pull request May 21, 2015
@chenghao-intel @yhuai
[SPARK-7320] [SQL] Add Cube / Rollup for dataframe
4fd6743 
This is a follow up for #6257, which broke the maven test.

Add cube & rollup for DataFrame
For example:
```scala
testData.rollup($"a" + $"b", $"b").agg(sum($"a" - $"b"))
testData.cube($"a" + $"b", $"b").agg(sum($"a" - $"b"))
```

Author: Cheng Hao <hao.cheng@intel.com>

Closes #6304 from chenghao-intel/rollup and squashes the following commits:

04bb1de [Cheng Hao] move the table register/unregister into beforeAll/afterAll
a6069f1 [Cheng Hao] cancel the implicit keyword
ced4b8f [Cheng Hao] remove the unnecessary code changes
9959dfa [Cheng Hao] update the code as comments
e1d88aa [Cheng Hao] update the code as suggested
03bc3d9 [Cheng Hao] Remove the CubedData & RollupedData
5fd62d0 [Cheng Hao] hiden the CubedData & RollupedData
5ffb196 [Cheng Hao] Add Cube / Rollup for dataframe

(cherry picked from commit 42c592a)
Signed-off-by: Yin Huai <yhuai@databricks.com>
@asfgit asfgit closed this in 42c592a May 21, 2015
rxin
rxin reviewed May 21, 2015
View reviewed changes
sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveDataFrameAnalyticsSuite.scala
Copy link
Contributor

@rxin rxin May 21, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

actually we should avoid setting up the test resource in the ctor. Either mark this as lazy or move it into beforeAll.

The reason is that once this fails, it will skip the entire suite and not have any "failed" tests reported.

@chenghao-intel
Copy link
Contributor Author

chenghao-intel commented May 21, 2015

Ok, yes, I will update that.

jeanlyn pushed a commit to jeanlyn/spark that referenced this pull request May 28, 2015
@chenghao-intel @jeanlyn
[SPARK-7320] [SQL] Add Cube / Rollup for dataframe
c2ef8a8 
This is a follow up for apache#6257, which broke the maven test.

Add cube & rollup for DataFrame
For example:
```scala
testData.rollup($"a" + $"b", $"b").agg(sum($"a" - $"b"))
testData.cube($"a" + $"b", $"b").agg(sum($"a" - $"b"))
```

Author: Cheng Hao <hao.cheng@intel.com>

Closes apache#6304 from chenghao-intel/rollup and squashes the following commits:

04bb1de [Cheng Hao] move the table register/unregister into beforeAll/afterAll
a6069f1 [Cheng Hao] cancel the implicit keyword
ced4b8f [Cheng Hao] remove the unnecessary code changes
9959dfa [Cheng Hao] update the code as comments
e1d88aa [Cheng Hao] update the code as suggested
03bc3d9 [Cheng Hao] Remove the CubedData & RollupedData
5fd62d0 [Cheng Hao] hiden the CubedData & RollupedData
5ffb196 [Cheng Hao] Add Cube / Rollup for dataframe
jeanlyn pushed a commit to jeanlyn/spark that referenced this pull request Jun 12, 2015
@chenghao-intel @jeanlyn
[SPARK-7320] [SQL] Add Cube / Rollup for dataframe
38c6628 
This is a follow up for apache#6257, which broke the maven test.

Add cube & rollup for DataFrame
For example:
```scala
testData.rollup($"a" + $"b", $"b").agg(sum($"a" - $"b"))
testData.cube($"a" + $"b", $"b").agg(sum($"a" - $"b"))
```

Author: Cheng Hao <hao.cheng@intel.com>

Closes apache#6304 from chenghao-intel/rollup and squashes the following commits:

04bb1de [Cheng Hao] move the table register/unregister into beforeAll/afterAll
a6069f1 [Cheng Hao] cancel the implicit keyword
ced4b8f [Cheng Hao] remove the unnecessary code changes
9959dfa [Cheng Hao] update the code as comments
e1d88aa [Cheng Hao] update the code as suggested
03bc3d9 [Cheng Hao] Remove the CubedData & RollupedData
5fd62d0 [Cheng Hao] hiden the CubedData & RollupedData
5ffb196 [Cheng Hao] Add Cube / Rollup for dataframe
nemccarthy pushed a commit to nemccarthy/spark that referenced this pull request Jun 19, 2015
@chenghao-intel @nemccarthy
[SPARK-7320] [SQL] Add Cube / Rollup for dataframe
0c192a6 
This is a follow up for apache#6257, which broke the maven test.

Add cube & rollup for DataFrame
For example:
```scala
testData.rollup($"a" + $"b", $"b").agg(sum($"a" - $"b"))
testData.cube($"a" + $"b", $"b").agg(sum($"a" - $"b"))
```

Author: Cheng Hao <hao.cheng@intel.com>

Closes apache#6304 from chenghao-intel/rollup and squashes the following commits:

04bb1de [Cheng Hao] move the table register/unregister into beforeAll/afterAll
a6069f1 [Cheng Hao] cancel the implicit keyword
ced4b8f [Cheng Hao] remove the unnecessary code changes
9959dfa [Cheng Hao] update the code as comments
e1d88aa [Cheng Hao] update the code as suggested
03bc3d9 [Cheng Hao] Remove the CubedData & RollupedData
5fd62d0 [Cheng Hao] hiden the CubedData & RollupedData
5ffb196 [Cheng Hao] Add Cube / Rollup for dataframe
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@chenghao-intel @yhuai @SparkQA @rxin