[SPARK-14819] [SQL] Improve SET / SET -v command #12583

bomeng · 2016-04-21T20:21:20Z

What changes were proposed in this pull request?

Currently SET and SET -v commands are similar to Hive SET command except the following difference:

The result is not sorted;
When using SET and SET -v, in addition to the Hive related properties, it will also list all the system properties and environment properties, which is very useful in some cases.

This JIRA is trying to make the current SET command more consistent to Hive output.

How was this patch tested?

Current test cases SQLQuerySuite and HiveQuerySuite should have enough test coverage.

SparkQA · 2016-04-21T20:23:53Z

Test build #56582 has finished for PR 12583 at commit 01d3e5a.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

rxin · 2016-04-21T21:28:20Z

You might need to rebase this in a bit because i'm refactoring all of these quite heavily for Spark 2.0.

rxin · 2016-04-21T21:28:42Z

sql/core/src/test/scala/org/apache/spark/sql/execution/command/DDLSuite.scala

  }

+  test("set / set -v") {
+    checkExistence(sql("set"), true, "env:", "system:")


can we add a test case that checks the value of SPARK_TESTING?

SPARK_TESTING is not in sys.env nor sys.props while running the test cases. Any suggestions to add the test cases for that?

Found out that current test cases SQLQuerySuite and HiveQuerySuite have intensive test coverage for SET command.

SparkQA · 2016-04-21T21:30:49Z

Test build #56585 has finished for PR 12583 at commit 790cfda.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-04-22T00:32:47Z

Test build #56603 has finished for PR 12583 at commit 2a077bd.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-04-22T03:57:36Z

Test build #56636 has finished for PR 12583 at commit e3ba716.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-04-22T05:09:12Z

Test build #56651 has finished for PR 12583 at commit c6805e4.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-04-22T06:51:03Z

Test build #56653 has finished for PR 12583 at commit 99ff8a5.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

rxin · 2016-04-22T06:56:55Z

@bomeng while you are at this, do you mind implementing https://issues.apache.org/jira/browse/SPARK-14806 together? Should be pretty trivial. Thanks.

SparkQA · 2016-04-22T10:54:34Z

Test build #56688 has finished for PR 12583 at commit 6c7276b.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun · 2017-01-05T00:13:13Z

Hi, @bomeng .
I hit the same issue. Could you update the PR?

viirya · 2017-01-05T02:36:43Z

sql/core/src/main/scala/org/apache/spark/sql/execution/command/SetCommand.scala

      (keyValueOutput, runFunc)

    // Queries all properties along with their default values and docs that are defined in the
    // SQLConf of the sqlContext.


We should update the comment regarding the system environment properties.

viirya · 2017-01-05T02:36:53Z

sql/core/src/main/scala/org/apache/spark/sql/execution/command/SetCommand.scala

      (keyValueOutput, runFunc)

    // (In Hive, "SET" returns all changed properties while "SET -v" returns all properties.)
    // Queries all key-value pairs that are set in the SQLConf of the sqlContext.


viirya · 2017-01-05T02:37:51Z

sql/core/src/main/scala/org/apache/spark/sql/execution/command/SetCommand.scala

  override def run(sqlContext: SQLContext): Seq[Row] = runFunc(sqlContext)

+  /**
+   * get the system environment properties as a sequence


Nit: "get" -> "Gets", "as a sequence" -> "as a sequence of rows".

dongjoon-hyun · 2017-01-13T18:54:48Z

Hmm. This PR seems to be stale. Actually, what I wanted was a sorted result of SET and SET -v.

dongjoon-hyun · 2017-01-13T19:08:34Z

I made another #16579 for SET for a sorted result.

HyukjinKwon · 2017-02-03T12:44:59Z

@dongjoon-hyun, then, do you mind if I ask if this PR is not useful in favour of #16579?

dongjoon-hyun · 2017-02-03T17:02:52Z

Hi, @bomeng , do you have something to refresh here?
I think we can close this for now.

## What changes were proposed in this pull request? This PR proposes to close stale PRs. What I mean by "stale" here includes that there are some review comments by reviewers but the author looks inactive without any answer to them more than a month. I left some comments roughly a week ago to ping and the author looks still inactive in these PR below These below includes some PR suggested to be closed and a PR against another branch which seems obviously inappropriate. Given the comments in the last three PRs below, they are probably worth being taken over by anyone who is interested in it. Closes apache#7963 Closes apache#8374 Closes apache#11192 Closes apache#11374 Closes apache#11692 Closes apache#12243 Closes apache#12583 Closes apache#12620 Closes apache#12675 Closes apache#12697 Closes apache#12800 Closes apache#13715 Closes apache#14266 Closes apache#15053 Closes apache#15159 Closes apache#15209 Closes apache#15264 Closes apache#15267 Closes apache#15871 Closes apache#15861 Closes apache#16319 Closes apache#16324 Closes apache#16890 Closes apache#12398 Closes apache#12933 Closes apache#14517 ## How was this patch tested? N/A Author: hyukjinkwon <gurwls223@gmail.com> Closes apache#16937 from HyukjinKwon/stale-prs-close.

improve SET / SET -v command

01d3e5a

remove unused imports

790cfda

rxin reviewed Apr 21, 2016
View reviewed changes

bomeng added 3 commits April 21, 2016 14:53

Merge remote-tracking branch 'upstream/master' into SPARK-14819

a0de01e

Merge remote-tracking branch 'upstream/master' into SPARK-14819

392327a

improve the codes and fix test failure

2a077bd

fix the test failure

e3ba716

bomeng added 3 commits April 21, 2016 22:05

remove unnecessary test case

84243a0

roll back

c6805e4

roll back

99ff8a5

merge upstream

6c7276b

viirya reviewed Jan 5, 2017

View reviewed changes

HyukjinKwon mentioned this pull request Feb 15, 2017

[BUILD] Close stale PRs #16937

Closed

asfgit closed this in ed338f7 Feb 17, 2017

[SPARK-14819] [SQL] Improve SET / SET -v command #12583

[SPARK-14819] [SQL] Improve SET / SET -v command #12583

Uh oh!

Conversation

bomeng commented Apr 21, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

How was this patch tested?

Uh oh!

SparkQA commented Apr 21, 2016

Uh oh!

rxin commented Apr 21, 2016

Uh oh!

rxin Apr 21, 2016

Choose a reason for hiding this comment

Uh oh!

bomeng Apr 21, 2016

Choose a reason for hiding this comment

Uh oh!

bomeng Apr 22, 2016

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Apr 21, 2016

Uh oh!

SparkQA commented Apr 22, 2016

Uh oh!

SparkQA commented Apr 22, 2016

Uh oh!

SparkQA commented Apr 22, 2016

Uh oh!

SparkQA commented Apr 22, 2016

Uh oh!

rxin commented Apr 22, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

SparkQA commented Apr 22, 2016

Uh oh!

dongjoon-hyun commented Jan 5, 2017

Uh oh!

viirya Jan 5, 2017

Choose a reason for hiding this comment

Uh oh!

viirya Jan 5, 2017

Choose a reason for hiding this comment

Uh oh!

viirya Jan 5, 2017

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun commented Jan 13, 2017

Uh oh!

dongjoon-hyun commented Jan 13, 2017

Uh oh!

HyukjinKwon commented Feb 3, 2017

Uh oh!

dongjoon-hyun commented Feb 3, 2017

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

bomeng commented Apr 21, 2016 •

edited

Loading

rxin commented Apr 22, 2016 •

edited

Loading