[SPARK-36457][DOCS] Review and fix issues in Scala/Java API docs #33824

gengliangwang · 2021-08-24T14:16:32Z

What changes were proposed in this pull request?

Compare the 3.2.0 API doc with the latest release version 3.1.2. Fix the following issues:

Add missing Since annotation for new APIs
Remove the leaking class/object in API doc

Why are the changes needed?

Improve API docs

Does this PR introduce any user-facing change?

No

How was this patch tested?

Existing UT

gengliangwang · 2021-08-24T14:19:30Z

core/src/main/scala/org/apache/spark/SparkException.scala

 * Arithmetic exception thrown from Spark with an error class.
 */
-class SparkArithmeticException(errorClass: String, messageParameters: Array[String])
+private[spark] class SparkArithmeticException(errorClass: String, messageParameters: Array[String])


cc @karenfeng

gengliangwang · 2021-08-24T14:19:35Z

core/src/main/scala/org/apache/spark/errors/SparkCoreErrors.scala

 * Object for grouping error messages from (most) exceptions thrown during query execution.
 */
-object SparkCoreErrors {
+private[spark] object SparkCoreErrors {


cc @karenfeng

shall we mark org.apache.spark.errors as a private package as well?

gengliangwang · 2021-08-24T14:20:05Z

core/src/main/scala/org/apache/spark/io/MutableCheckedOutputStream.scala

 * change the checksum calculator at runtime.
 */
-class MutableCheckedOutputStream(out: OutputStream) extends OutputStream {
+private[spark] class MutableCheckedOutputStream(out: OutputStream) extends OutputStream {


cc @Ngone51

LGTM, thanks!

gengliangwang · 2021-08-24T14:26:12Z

core/src/main/scala/org/apache/spark/util/DependencyUtils.scala

 import org.apache.spark.internal.config._

-case class IvyProperties(
+private[spark] case class IvyProperties(


cc @AngersZhuuuu

gengliangwang · 2021-08-24T14:31:23Z

sql/catalyst/src/main/java/org/apache/spark/sql/connector/catalog/FunctionCatalog.java

+ *
+ * @since 3.2.0
 */
+@Evolving


cc @rdblue @cloud-fan

gengliangwang · 2021-08-24T14:32:18Z

...talyst/src/main/java/org/apache/spark/sql/connector/catalog/functions/AggregateFunction.java

+ *
+ * @since 3.2.0
 */
+@Evolving


cc @rdblue @cloud-fan

Also please take a look at the following DSV2 changes, thanks!

gengliangwang · 2021-08-24T14:33:42Z

sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryExecutionErrors.scala

 * grouped into [[QueryCompilationErrors]].
 */
-object QueryExecutionErrors {
+private[spark] object QueryExecutionErrors {


cc @karenfeng

shall we mark the entire org.apache.spark.sql.errors as private?

See https://github.com/apache/spark/blob/master/project/SparkBuild.scala#L944-L973

SGTM. @karenfeng will there be any new public API under org.apache.spark.sql.errors?

cc @allisonwang-db

I think that the only useful API to expose related to this may be SparkThrowable, which contains errorClass and sqlState. But that is under org.apache.spark.

ok, let me mark org.apache.spark.sql.errors as private

gengliangwang · 2021-08-24T14:33:59Z

sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryParsingErrors.scala

 * Currently it includes all ParseException.
 */
-object QueryParsingErrors {
+private[spark] object QueryParsingErrors {


cc @karenfeng

cc @allisonwang-db

SparkQA · 2021-08-24T15:41:40Z

Kubernetes integration test unable to build dist.

exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47231/

SparkQA · 2021-08-24T17:50:54Z

Test build #142731 has finished for PR 33824 at commit df29e32.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-08-25T04:43:40Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47251/

SparkQA · 2021-08-25T05:21:56Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47251/

SparkQA · 2021-08-25T06:28:02Z

Test build #142752 has finished for PR 33824 at commit 3769688.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

gengliangwang · 2021-08-26T04:58:37Z

Merging to master/3.2

Compare the 3.2.0 API doc with the latest release version 3.1.2. Fix the following issues: - Add missing `Since` annotation for new APIs - Remove the leaking class/object in API doc Improve API docs No Existing UT Closes apache#33824 from gengliangwang/auditDoc. Authored-by: Gengliang Wang <gengliang@apache.org> Signed-off-by: Gengliang Wang <gengliang@apache.org>

* the functions make_dt_interval and make_ym_interval should make it clear that some of the fields are optional * remove the `|` symbol from the doc of `bit_get` https://dist.apache.org/repos/dist/dev/spark/v3.2.0-rc1-docs/_site/api/sql/index.html#bit_get * Address one missing comment in apache#33824 (comment) Improve the documentation. No Build doc and preview: ![image](https://user-images.githubusercontent.com/1097932/130996918-8c1fff88-ef5a-434b-8445-df7140bad3ba.png) ![image](https://user-images.githubusercontent.com/1097932/130996954-0ced28e7-fb90-4fcc-857e-6ccc31dc3c09.png) ![image](https://user-images.githubusercontent.com/1097932/130955106-5ae32dfc-6e89-4e28-bb8a-6c1b5213051c.png) ![image](https://user-images.githubusercontent.com/1097932/130922351-2f0f262d-5624-4d08-ba83-dfa3ed0b646b.png) Closes apache#33847 from gengliangwang/auditSQLDoc. Authored-by: Gengliang Wang <gengliang@apache.org> Signed-off-by: Gengliang Wang <gengliang@apache.org>

### What changes were proposed in this pull request? * the functions make_dt_interval and make_ym_interval should make it clear that some of the fields are optional * remove the `|` symbol from the doc of `bit_get` https://dist.apache.org/repos/dist/dev/spark/v3.2.0-rc1-docs/_site/api/sql/index.html#bit_get * Address one missing comment in #33824 (comment) ### Why are the changes needed? Improve the documentation. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Build doc and preview: ![image](https://user-images.githubusercontent.com/1097932/130996918-8c1fff88-ef5a-434b-8445-df7140bad3ba.png) ![image](https://user-images.githubusercontent.com/1097932/130996954-0ced28e7-fb90-4fcc-857e-6ccc31dc3c09.png) ![image](https://user-images.githubusercontent.com/1097932/130955106-5ae32dfc-6e89-4e28-bb8a-6c1b5213051c.png) ![image](https://user-images.githubusercontent.com/1097932/130922351-2f0f262d-5624-4d08-ba83-dfa3ed0b646b.png) Closes #33857 from gengliangwang/SPARK-36597-3.2. Authored-by: Gengliang Wang <gengliang@apache.org> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>

audit

f7b3ebb

github-actions bot added CORE SQL STRUCTURED STREAMING labels Aug 24, 2021

private

df29e32

gengliangwang changed the title ~~[SPARK-36457][Doc] Review and fix issues in API docs~~ [SPARK-36457][Docs] Review and fix issues in API docs Aug 24, 2021

gengliangwang commented Aug 24, 2021

View reviewed changes

HyukjinKwon changed the title ~~[SPARK-36457][Docs] Review and fix issues in API docs~~ [SPARK-36457][DOCS] Review and fix issues in API docs Aug 25, 2021

HyukjinKwon approved these changes Aug 25, 2021

View reviewed changes

ignore org.apache.spark.sql.errors doc

3769688

github-actions bot added the BUILD label Aug 25, 2021

gengliangwang changed the title ~~[SPARK-36457][DOCS] Review and fix issues in API docs~~ [SPARK-36457][DOCS] Review and fix issues in Scala/Java API docs Aug 26, 2021

gengliangwang closed this in 1a42aa5 Aug 26, 2021

This was referenced Aug 26, 2021

[SPARK-36457][DOCS][3.2] Review and fix issues in Scala/Java API docs #33845

Closed

[SPARK-36597][DOCS] Fix issues in SQL function docs #33847

Closed

[SPARK-36336][SQL][FOLLOWUP] Mark exception private[spark] #33856

Closed

gengliangwang mentioned this pull request Aug 27, 2021

[SPARK-36597][DOCS][3.2] Fix issues in SQL function docs #33857

Closed

[SPARK-36457][DOCS] Review and fix issues in Scala/Java API docs #33824

[SPARK-36457][DOCS] Review and fix issues in Scala/Java API docs #33824

Uh oh!

Conversation

gengliangwang commented Aug 24, 2021

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Aug 24, 2021

Uh oh!

SparkQA commented Aug 24, 2021

Uh oh!

SparkQA commented Aug 25, 2021

Uh oh!

SparkQA commented Aug 25, 2021

Uh oh!

SparkQA commented Aug 25, 2021

Uh oh!

gengliangwang commented Aug 26, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants