Skip to content

[SPARK-25479][TEST] Refactor DatasetBenchmark to use main method #22488

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wangyum wants to merge 9 commits into from
Closed

[SPARK-25479][TEST] Refactor DatasetBenchmark to use main method #22488

wangyum wants to merge 9 commits into from

Conversation

@wangyum
Copy link
Member

@wangyum wangyum commented Sep 20, 2018

What changes were proposed in this pull request?

Refactor DatasetBenchmark to use main method.
Generate benchmark result:

SPARK_GENERATE_BENCHMARK_FILES=1 build/sbt "sql/test:runMain org.apache.spark.sql.DatasetBenchmark"

How was this patch tested?

manual tests

@SparkQA
Copy link

SparkQA commented Sep 20, 2018

Test build #96342 has finished for PR 22488 at commit 21b623a.

  • This patch fails to generate documentation.
  • This patch merges cleanly.
  • This patch adds no public classes.

@wangyum
Copy link
Member Author

wangyum commented Sep 20, 2018

retest this please

@SparkQA
Copy link

SparkQA commented Sep 20, 2018

Test build #96344 has finished for PR 22488 at commit 21b623a.

  • This patch fails to generate documentation.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Sep 20, 2018

Test build #96354 has finished for PR 22488 at commit 152c549.

  • This patch fails to generate documentation.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Sep 20, 2018

Test build #96375 has finished for PR 22488 at commit 51ae87e.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@wangyum
Merge remote-tracking branch 'upstream/master' into SPARK-25479
71dfe03 
# Conflicts:
#	sql/core/src/test/scala/org/apache/spark/sql/DatasetBenchmark.scala
@SparkQA
Copy link

SparkQA commented Sep 21, 2018

Test build #96440 has finished for PR 22488 at commit 71dfe03.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@wangyum
Copy link
Member Author

wangyum commented Sep 21, 2018

retest this please

@SparkQA
Copy link

SparkQA commented Sep 22, 2018

Test build #96461 has finished for PR 22488 at commit 71dfe03.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@wangyum
Copy link
Member Author

wangyum commented Sep 22, 2018

retest this please

@SparkQA
Copy link

SparkQA commented Sep 22, 2018

Test build #96464 has finished for PR 22488 at commit 71dfe03.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@wangyum
Copy link
Member Author

wangyum commented Sep 24, 2018

@dongjoon-hyun I think this refactor is ready to go. Thanks.

dongjoon-hyun
dongjoon-hyun reviewed Sep 24, 2018
View reviewed changes
sql/core/src/test/scala/org/apache/spark/sql/DatasetBenchmark.scala Outdated
val spark = SparkSession.builder
.master("local[*]")
.appName("Dataset benchmark")
.getOrCreate()
Copy link
Member

@dongjoon-hyun dongjoon-hyun Sep 24, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we move this SparkSession building part into benchmark() function and before runBenchmark("Dataset Benchmark")?

@SparkQA
Copy link

SparkQA commented Oct 1, 2018

Test build #96819 has finished for PR 22488 at commit 7990e13.

  • This patch fails PySpark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@wangyum
Copy link
Member Author

wangyum commented Oct 1, 2018

retest this please

wangyum
wangyum commented Oct 1, 2018
View reviewed changes
sql/core/src/test/scala/org/apache/spark/sql/DatasetBenchmark.scala

def main(args: Array[String]): Unit = {
val spark = SparkSession.builder
override def getSparkSession: SparkSession = {
Copy link
Member Author

@wangyum wangyum Oct 1, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Need override default SparkSession as default SparkSession is:

    SparkSession.builder()
      .master("local[1]")
      .appName(this.getClass.getCanonicalName)
      .config(SQLConf.SHUFFLE_PARTITIONS.key, 1)
      .config(SQLConf.AUTO_BROADCASTJOIN_THRESHOLD.key, 1)
      .getOrCreate()

@SparkQA
Copy link

SparkQA commented Oct 2, 2018

Test build #96832 has finished for PR 22488 at commit 7990e13.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@dongjoon-hyun
Copy link
Member

dongjoon-hyun commented Oct 2, 2018

Retest this please.

@SparkQA
Copy link

SparkQA commented Oct 2, 2018

Test build #96864 has finished for PR 22488 at commit 7990e13.

  • This patch fails to build.
  • This patch merges cleanly.
  • This patch adds no public classes.

@dongjoon-hyun
Copy link
Member

dongjoon-hyun commented Oct 2, 2018

@wangyum . Could you review and merge wangyum#13 ?

dongjoon-hyun
dongjoon-hyun approved these changes Oct 2, 2018
View reviewed changes
Copy link
Member

@dongjoon-hyun dongjoon-hyun left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1, LGTM (pending Jenkins)

@dongjoon-hyun
Copy link
Member

dongjoon-hyun commented Oct 3, 2018

Hi, @jiangxb1987 .
Could you review (and merge) this PR?

@wangyum
Copy link
Member Author

wangyum commented Oct 3, 2018

Congratulation, @jiangxb1987

@SparkQA
Copy link

SparkQA commented Oct 3, 2018

Test build #96879 has finished for PR 22488 at commit d2d0a3e.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Oct 3, 2018

Test build #96880 has finished for PR 22488 at commit 27c6493.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@dongjoon-hyun
Copy link
Member

dongjoon-hyun commented Oct 4, 2018

Merged to master.

@asfgit asfgit closed this in 95ae209 Oct 5, 2018
@wangyum wangyum deleted the SPARK-25479 branch October 5, 2018 01:40
jackylee-ch pushed a commit to jackylee-ch/spark that referenced this pull request Feb 18, 2019
@wangyum @dongjoon-hyun
[SPARK-25479][TEST] Refactor DatasetBenchmark to use main method
c8859e2 
## What changes were proposed in this pull request?

Refactor `DatasetBenchmark` to use main method.
Generate benchmark result:
```sh
SPARK_GENERATE_BENCHMARK_FILES=1 build/sbt "sql/test:runMain org.apache.spark.sql.DatasetBenchmark"
```

## How was this patch tested?

manual tests

Closes apache#22488 from wangyum/SPARK-25479.

Lead-authored-by: Yuming Wang <yumwang@ebay.com>
Co-authored-by: Dongjoon Hyun <dongjoon@apache.org>
Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@dongjoon-hyun dongjoon-hyun dongjoon-hyun approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@wangyum @SparkQA @dongjoon-hyun