[SPARK-25663][SPARK-25661][SQL][TEST] Refactor BuiltInDataSourceWriteBenchmark, DataSourceWriteBenchmark and AvroWriteBenchmark to use main method #22861

yucai · 2018-10-27T15:51:29Z

What changes were proposed in this pull request?

Refactor BuiltInDataSourceWriteBenchmark, DataSourceWriteBenchmark and AvroWriteBenchmark to use main method.

SPARK_GENERATE_BENCHMARK_FILES=1 build/sbt "sql/test:runMain org.apache.spark.sql.execution.benchmark.BuiltInDataSourceWriteBenchmark"

SPARK_GENERATE_BENCHMARK_FILES=1 build/sbt "avro/test:runMain org.apache.spark.sql.execution.benchmark.AvroWriteBenchmark"

How was this patch tested?

manual tests

yucai · 2018-10-27T15:54:45Z

@gengliangwang @wangyum @dongjoon-hyun help review.

SparkQA · 2018-10-27T19:56:02Z

Test build #98133 has finished for PR 22861 at commit 81fe383.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
trait DataSourceWriteBenchmark extends SqlBasedBenchmark

dongjoon-hyun

@yucai . We (@yucai and @gengliangwang and me) know what is this for, but please don't piggy-back (hide) something like this. This kind of change requires another explicit PR in general.

yucai · 2018-10-29T01:57:25Z

@dongjoon-hyun Originally, I want to do two things in this PR.

Make mainArgs correctly set in BenchmarkBase.
Include an example to use mainArgs: refactor DataSourceWriteBenchmark and BuiltInDataSourceWriteBenchmark to use main method.

But, refactor DataSourceWriteBenchmark will lead to refactor AvroWriteBenchmark.

Any suggestion how to split PR?

dongjoon-hyun · 2018-10-29T04:13:53Z

At least, the following is worth for a separate PR because it's orthogonal Refactor ... to use main method.

1. Make mainArgs correctly set in BenchmarkBase.

One PR had better have one theme. Putting different themes into one PR together is not a good practice.

Please start with the minimal one. If the committer asks some example, then add that later. That's better.

gengliangwang · 2018-10-29T05:49:56Z

Personally I am against accessing the main args in such way. It looks a bit ugly.
But if we have to move everything to BenchmarkBase, then maybe this is the way.

yucai · 2018-10-29T07:15:58Z

Current implementation misses main args, but some suite would need it anyway.
Let's can discuss #22872.

…iteBenchmark

yucai · 2018-10-29T15:36:21Z

@dongjoon-hyun I used #22872 to make main args accessible for BenchmarkBase's subclass, this PR is mainly for refactoring DataSourceWriteBenchmark and BuiltInDataSourceWriteBenchmark. But it will lead to AvroWriteBenchmark refactor, so I put them in one PR, kindly let me know if you have better idea.

wangyum · 2018-10-29T16:03:36Z

external/avro/src/test/scala/org/apache/spark/sql/execution/benchmark/AvroWriteBenchmark.scala

- * 2. with sbt: build/sbt "avro/test:runMain <this class>"
+ * To run this benchmark:
+ * {{{
+ *   1. without sbt: bin/spark-submit --class <this class>


Please add avro:

bin/spark-submit --class <this class> --jars <spark core test jar>,<spark catalyst test jar>,<spark sql test jar> <spark avro test jar>

I hint an exception when run:

bin/spark-submit --class org.apache.spark.sql.execution.benchmark.AvroWriteBenchmark --jars ./core/target/spark-core_2.11-3.0.0-SNAPSHOT-tests.jar,./sql/catalyst/target/spark-catalyst_2.11-3.0.0-SNAPSHOT-tests.jar,./sql/core/target/spark-sql_2.11-3.0.0-SNAPSHOT-tests.jar ./external/avro/target/spark-avro_2.11-3.0.0-SNAPSHOT-tests.jar

Exception in thread "main" org.apache.spark.sql.AnalysisException: Failed to find data source: Avro. Avro is built-in but external data source module since Spark 2.4. Please deploy the application as per the deployment section of "Apache Avro Data Source Guide".; at org.apache.spark.sql.execution.datasources.DataSource$.lookupDataSource(DataSource.scala:647) at org.apache.spark.sql.execution.datasources.DataSource.providingClass$lzycompute(DataSource.scala:94) at org.apache.spark.sql.execution.datasources.DataSource.providingClass(DataSource.scala:93) at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:313) at org.apache.spark.sql.execution.command.CreateDataSourceTableCommand.run(createDataSourceTables.scala:78) at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:70) at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:68) at org.apache.spark.sql.execution.command.ExecutedCommandExec.executeCollect(commands.scala:79) at org.apache.spark.sql.Dataset$$anonfun$6.apply(Dataset.scala:195) at org.apache.spark.sql.Dataset$$anonfun$6.apply(Dataset.scala:195) ......

@wangyum Good catch! I think it needs <spark avro jar>, added.

SparkQA · 2018-10-29T16:21:39Z

Test build #98207 has finished for PR 22861 at commit 59f167d.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
trait UnevaluableAggregate extends DeclarativeAggregate
abstract class UnevaluableBooleanAggBase(arg: Expression)
case class EveryAgg(arg: Expression) extends UnevaluableBooleanAggBase(arg)
case class AnyAgg(arg: Expression) extends UnevaluableBooleanAggBase(arg)
case class SomeAgg(arg: Expression) extends UnevaluableBooleanAggBase(arg)
case class UnresolvedCatalystToExternalMap(

SparkQA · 2018-10-29T17:50:19Z

Test build #98215 has finished for PR 22861 at commit 74f6ef5.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2018-10-30T06:15:44Z

Test build #98247 has finished for PR 22861 at commit 93ca3be.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2018-10-30T06:19:30Z

Test build #98245 has finished for PR 22861 at commit f06b2ac.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

yucai · 2018-10-30T06:47:24Z

@dongjoon-hyun Tests have been passed.

dongjoon-hyun

@yucai . Almost looks good. Please update the PR description according to the PR content and merge yucai#6 .

Fix

- SPARK_GENERATE_BENCHMARK_FILES=1 build/sbt "sql/test:runMain org.apache.spark.sql.execution.benchmark.AvroWriteBenchmark"
+ SPARK_GENERATE_BENCHMARK_FILES=1 build/sbt "avro/test:runMain org.apache.spark.sql.execution.benchmark.AvroWriteBenchmark"

Remove

SPARK_GENERATE_BENCHMARK_FILES=1 build/sbt "sql/test:runMain org.apache.spark.sql.execution.benchmark.BuiltInDataSourceWriteBenchmark Parquet ORC"

@dongjoon-hyun

Thanks @dongjoon-hyun , merged.

dongjoon-hyun · 2018-10-31T10:01:33Z

Thank you, @yucai , @gengliangwang and @wangyum .
Merged to master. (My PR is just for test output update. So there is no need to wait Jenkins)

SparkQA · 2018-10-31T14:38:52Z

Test build #98310 has finished for PR 22861 at commit 3ca0cf2.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

…Benchmark, DataSourceWriteBenchmark and AvroWriteBenchmark to use main method ## What changes were proposed in this pull request? Refactor BuiltInDataSourceWriteBenchmark, DataSourceWriteBenchmark and AvroWriteBenchmark to use main method. ``` SPARK_GENERATE_BENCHMARK_FILES=1 build/sbt "sql/test:runMain org.apache.spark.sql.execution.benchmark.BuiltInDataSourceWriteBenchmark" SPARK_GENERATE_BENCHMARK_FILES=1 build/sbt "avro/test:runMain org.apache.spark.sql.execution.benchmark.AvroWriteBenchmark" ``` ## How was this patch tested? manual tests Closes apache#22861 from yucai/BuiltInDataSourceWriteBenchmark. Lead-authored-by: yucai <yyu1@ebay.com> Co-authored-by: Yucai Yu <yucai.yu@foxmail.com> Co-authored-by: Dongjoon Hyun <dongjoon@apache.org> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>

BuiltInDataSourceWriteBenchmark

81fe383

dongjoon-hyun requested changes Oct 27, 2018

View reviewed changes

yucai added 2 commits October 29, 2018 13:46

monir

155a872

[SPARK-25864][SQL][TEST] Make main args set correctly in BenchmarkBase

22b4c50

minor

83d21c2

yucai added 2 commits October 29, 2018 20:12

Merge remote-tracking branch 'origin/master' into BuiltInDataSourceWr…

59f167d

…iteBenchmark

minor

74f6ef5

wangyum reviewed Oct 29, 2018

View reviewed changes

yucai added 2 commits October 30, 2018 10:36

address comments

f06b2ac

minor

93ca3be

Update result

cf8dbea

dongjoon-hyun reviewed Oct 31, 2018

View reviewed changes

Merge pull request #6 from dongjoon-hyun/PR-22861

3ca0cf2

Thanks @dongjoon-hyun , merged.

asfgit closed this in f8484e4 Oct 31, 2018

[SPARK-25663][SPARK-25661][SQL][TEST] Refactor BuiltInDataSourceWriteBenchmark, DataSourceWriteBenchmark and AvroWriteBenchmark to use main method #22861

[SPARK-25663][SPARK-25661][SQL][TEST] Refactor BuiltInDataSourceWriteBenchmark, DataSourceWriteBenchmark and AvroWriteBenchmark to use main method #22861

Uh oh!

Conversation

yucai commented Oct 27, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

How was this patch tested?

Uh oh!

yucai commented Oct 27, 2018

Uh oh!

SparkQA commented Oct 27, 2018

Uh oh!

dongjoon-hyun left a comment

Choose a reason for hiding this comment

Uh oh!

yucai commented Oct 29, 2018

Uh oh!

dongjoon-hyun commented Oct 29, 2018

Uh oh!

gengliangwang commented Oct 29, 2018

Uh oh!

yucai commented Oct 29, 2018

Uh oh!

yucai commented Oct 29, 2018

Uh oh!

wangyum Oct 29, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

wangyum Oct 29, 2018

Choose a reason for hiding this comment

Uh oh!

yucai Oct 30, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Oct 29, 2018

Uh oh!

SparkQA commented Oct 29, 2018

Uh oh!

SparkQA commented Oct 30, 2018

Uh oh!

SparkQA commented Oct 30, 2018

Uh oh!

yucai commented Oct 30, 2018

Uh oh!

dongjoon-hyun left a comment

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun commented Oct 31, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

SparkQA commented Oct 31, 2018

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

yucai commented Oct 27, 2018 •

edited

Loading

wangyum Oct 29, 2018 •

edited

Loading

yucai Oct 30, 2018 •

edited

Loading

dongjoon-hyun commented Oct 31, 2018 •

edited

Loading