[SPARK-36597][DOCS] Fix issues in SQL function docs #33847

gengliangwang · 2021-08-26T07:44:27Z

What changes were proposed in this pull request?

the functions make_dt_interval and make_ym_interval should make it clear that some of the fields are optional
remove the | symbol from the doc of bit_get https://dist.apache.org/repos/dist/dev/spark/v3.2.0-rc1-docs/_site/api/sql/index.html#bit_get
Address one missing comment in [SPARK-36457][DOCS] Review and fix issues in Scala/Java API docs #33824 (comment)

Why are the changes needed?

Improve the documentation.

Does this PR introduce any user-facing change?

No

How was this patch tested?

Build doc and preview:

gengliangwang · 2021-08-26T07:44:39Z

project/SparkBuild.scala

      .map(_.filterNot(_.getCanonicalPath.contains("org/apache/spark/sql/internal")))
      .map(_.filterNot(_.getCanonicalPath.contains("org/apache/spark/sql/hive")))
      .map(_.filterNot(_.getCanonicalPath.contains("org/apache/spark/sql/catalog/v2/utils")))
+      .map(_.filterNot(_.getCanonicalPath.contains("org.apache.spark.errors")))


cc @cloud-fan

gengliangwang · 2021-08-26T07:44:46Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/bitwiseExpressions.scala

-            |  The positions are numbered from right to left, starting at zero.
-            |  The position argument cannot be negative.
-          """,
+    _FUNC_(expr, pos) - Returns the value of the bit (0 or 1) at the specified position.


cc @beliefer

gengliangwang · 2021-08-26T07:44:55Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/intervalExpressions.scala

 // scalastyle:off line.size.limit
 @ExpressionDescription(
-  usage = "_FUNC_(days, hours, mins, secs) - Make DayTimeIntervalType duration from days, hours, mins and secs.",
+  usage = "_FUNC_(days[, hours[, mins[, secs]]]) - Make DayTimeIntervalType duration from days, hours, mins and secs.",


cc @MaxGekk

days is optional

gengliangwang · 2021-08-26T07:45:07Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/intervalExpressions.scala


 @ExpressionDescription(
-  usage = "_FUNC_(years, months) - Make year-month interval from years, months.",
+  usage = "_FUNC_(years[, months]) - Make year-month interval from years, months.",


cc @MaxGekk

years is optional

hmmm, does it really make sense to make it optional?

nvm, it follows the old make_interval.

beliefer · 2021-08-26T08:12:47Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/bitwiseExpressions.scala

-            |  The positions are numbered from right to left, starting at zero.
-            |  The position argument cannot be negative.
-          """,
+    _FUNC_(expr, pos) - Returns the value of the bit (0 or 1) at the specified position.


MaxGekk

FYI

spark/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/intervalExpressions.scala

Line 230 in 39002cb

    
           usage = "_FUNC_(years, months, weeks, days, hours, mins, secs) - Make interval from years, months, weeks, days, hours, mins and secs.",

MaxGekk · 2021-08-26T08:13:38Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/intervalExpressions.scala

 // scalastyle:off line.size.limit
 @ExpressionDescription(
-  usage = "_FUNC_(days, hours, mins, secs) - Make DayTimeIntervalType duration from days, hours, mins and secs.",
+  usage = "_FUNC_(days[, hours[, mins[, secs]]]) - Make DayTimeIntervalType duration from days, hours, mins and secs.",


MaxGekk · 2021-08-26T08:14:14Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/intervalExpressions.scala


 @ExpressionDescription(
-  usage = "_FUNC_(years, months) - Make year-month interval from years, months.",
+  usage = "_FUNC_(years[, months]) - Make year-month interval from years, months.",


SparkQA · 2021-08-26T08:32:37Z

Kubernetes integration test unable to build dist.

exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47295/

gengliangwang · 2021-08-26T09:36:51Z

@MaxGekk I have addressed your comment on "make_interval"

MaxGekk · 2021-08-26T09:43:17Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/intervalExpressions.scala

 // scalastyle:off line.size.limit
 @ExpressionDescription(
-  usage = "_FUNC_(years, months, weeks, days, hours, mins, secs) - Make interval from years, months, weeks, days, hours, mins and secs.",
+  usage = "_FUNC_(years[, months[, weeks[, days[, hours[, mins[, secs]]]]]]) - Make interval from years, months, weeks, days, hours, mins and secs.",


years is optional too, see

spark/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/intervalExpressions.scala

Line 298 in 39002cb

def this() = this(Literal(0))

SparkQA · 2021-08-26T10:50:59Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47308/

SparkQA · 2021-08-26T10:57:57Z

Test build #142795 has finished for PR 33847 at commit ef063eb.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-08-26T11:35:16Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47308/

SparkQA · 2021-08-26T12:41:20Z

Test build #142807 has finished for PR 33847 at commit 02f7d4e.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-08-26T12:51:35Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47316/

SparkQA · 2021-08-26T13:35:41Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47316/

SparkQA · 2021-08-26T14:36:12Z

Test build #142814 has finished for PR 33847 at commit dddc1fd.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-08-26T16:45:55Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47323/

SparkQA · 2021-08-26T17:44:20Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47323/

SparkQA · 2021-08-26T18:29:47Z

Test build #142823 has finished for PR 33847 at commit 85e4d98.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

gengliangwang · 2021-08-27T05:28:37Z

Merging to master
I will create another PR against 3.2

* the functions make_dt_interval and make_ym_interval should make it clear that some of the fields are optional * remove the `|` symbol from the doc of `bit_get` https://dist.apache.org/repos/dist/dev/spark/v3.2.0-rc1-docs/_site/api/sql/index.html#bit_get * Address one missing comment in apache#33824 (comment) Improve the documentation. No Build doc and preview: ![image](https://user-images.githubusercontent.com/1097932/130996918-8c1fff88-ef5a-434b-8445-df7140bad3ba.png) ![image](https://user-images.githubusercontent.com/1097932/130996954-0ced28e7-fb90-4fcc-857e-6ccc31dc3c09.png) ![image](https://user-images.githubusercontent.com/1097932/130955106-5ae32dfc-6e89-4e28-bb8a-6c1b5213051c.png) ![image](https://user-images.githubusercontent.com/1097932/130922351-2f0f262d-5624-4d08-ba83-dfa3ed0b646b.png) Closes apache#33847 from gengliangwang/auditSQLDoc. Authored-by: Gengliang Wang <gengliang@apache.org> Signed-off-by: Gengliang Wang <gengliang@apache.org>

gengliangwang added 2 commits August 26, 2021 14:57

audit

a8b979e

address comments

ef063eb

github-actions bot added BUILD CORE SQL labels Aug 26, 2021

gengliangwang commented Aug 26, 2021

View reviewed changes

beliefer approved these changes Aug 26, 2021

View reviewed changes

MaxGekk reviewed Aug 26, 2021

View reviewed changes

HyukjinKwon approved these changes Aug 26, 2021

View reviewed changes

cloud-fan approved these changes Aug 26, 2021

View reviewed changes

address comment

02f7d4e

MaxGekk reviewed Aug 26, 2021

View reviewed changes

address

dddc1fd

MaxGekk approved these changes Aug 26, 2021

View reviewed changes

update

85e4d98

gengliangwang closed this in e650d06 Aug 27, 2021

gengliangwang mentioned this pull request Aug 27, 2021

[SPARK-36597][DOCS][3.2] Fix issues in SQL function docs #33857

Closed

[SPARK-36597][DOCS] Fix issues in SQL function docs #33847

[SPARK-36597][DOCS] Fix issues in SQL function docs #33847

Uh oh!

Conversation

gengliangwang commented Aug 26, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

MaxGekk left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Aug 26, 2021

Uh oh!

gengliangwang commented Aug 26, 2021

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Aug 26, 2021

Uh oh!

SparkQA commented Aug 26, 2021

Uh oh!

SparkQA commented Aug 26, 2021

Uh oh!

SparkQA commented Aug 26, 2021

Uh oh!

SparkQA commented Aug 26, 2021

Uh oh!

SparkQA commented Aug 26, 2021

Uh oh!

SparkQA commented Aug 26, 2021

Uh oh!

SparkQA commented Aug 26, 2021

Uh oh!

SparkQA commented Aug 26, 2021

Uh oh!

SparkQA commented Aug 26, 2021

Uh oh!

gengliangwang commented Aug 27, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

gengliangwang commented Aug 26, 2021 •

edited

Loading