Skip to content

[SPARK-12046] [DOC] [1.6] ScalaDoc and JavaDoc fixes #10039

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
liancheng wants to merge 1 commit into from
Closed

[SPARK-12046] [DOC] [1.6] ScalaDoc and JavaDoc fixes #10039

liancheng wants to merge 1 commit into from

Conversation

@liancheng
Copy link
Contributor

@liancheng liancheng commented Nov 30, 2015

This PR fixes several issues in our Scala and Java API doc.

  • Hides spark.memory package

  • Hides spark.examples package

  • Fixes ScalaDoc unordered list format through out the code base

    Unordoreed list in ScalaDoc requires an extra space before - or *, i.e.:

    /**
  * A list:
  *
  *  - Item 1
  *  - Item 2
  */

    rather than

    /**
  * A list:
  *
  * - Item 1
  * - Item 2
  */

    Otherwise the list won't be properly rendered.

  • Some other minor formatting issues

Remaining issues that I haven't figured out a proper fix:

  • :: DeveloperApi :: badge is not rendered properly on my local machine, but not sure whether it's an environmental issue.

  • akka package still appears in ScalaDoc root package page, even though it's already filtered in SparkBuild.scala.

  • Seems that most stuff in spark.util shouldn't be public, e.g.

    • RpcUtils
    • InnerClosureFinder

    But they've been there for a long time. Not sure whether they are already considered as part of the public API?

  • KinesisUtilsPythonHelper shows up in JavaDoc even though it's marked as private

@liancheng
Copy link
Contributor Author

liancheng commented Nov 30, 2015

cc @rxin

@srowen
Copy link
Member

srowen commented Nov 30, 2015

LGTM

@rxin
Copy link
Contributor

rxin commented Nov 30, 2015

LGTM too

@SparkQA
Copy link

SparkQA commented Nov 30, 2015

Test build #46878 has finished for PR 10039 at commit 81d973b.

  • This patch fails PySpark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@yhuai
Copy link
Contributor

yhuai commented Nov 30, 2015

test this please

@SparkQA
Copy link

SparkQA commented Nov 30, 2015

Test build #46896 has finished for PR 10039 at commit 81d973b.

  • This patch fails PySpark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@zsxwing zsxwing mentioned this pull request Nov 30, 2015
Closed
@zsxwing
Copy link
Member

zsxwing commented Dec 1, 2015

retest this please

asfgit pushed a commit that referenced this pull request Dec 1, 2015
@zsxwing
[SPARK-12058][HOTFIX] Disable KinesisStreamTests
edb26e7 
KinesisStreamTests in test.py is broken because of #9403. See https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46896/testReport/(root)/KinesisStreamTests/test_kinesis_stream/

Because Streaming Python didn’t work when merging #9403, the PR build didn’t report the Python test failure actually.

This PR just disabled the test to unblock #10039

Author: Shixiong Zhu <shixiong@databricks.com>

Closes #10047 from zsxwing/disable-python-kinesis-test.
@liancheng
Copy link
Contributor Author

liancheng commented Dec 1, 2015

retest this please

@yhuai
Copy link
Contributor

yhuai commented Dec 1, 2015

test this please

@yhuai
Copy link
Contributor

yhuai commented Dec 1, 2015

should we also update master?

@JoshRosen
Copy link
Contributor

JoshRosen commented Dec 1, 2015

Jenkins, retest this please.

@SparkQA
Copy link

SparkQA commented Dec 1, 2015

Test build #46938 has finished for PR 10039 at commit 81d973b.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@yhuai
Copy link
Contributor

yhuai commented Dec 1, 2015

I think we need to delete the hive-exec jar from amp-jenkins-worker-03. The failure of https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46938/consoleFull was caused by a bad hive jar in the maven/ivy cache.

@JoshRosen
Copy link
Contributor

JoshRosen commented Dec 1, 2015

@yhuai which JAR specifically? I can log in to that machine and remove it.

@liancheng liancheng mentioned this pull request Dec 1, 2015
Closed
@liancheng
Copy link
Contributor Author

liancheng commented Dec 1, 2015

@JoshRosen I guess it's safe to simply remove all Hive 1.2.1 and 1.2.1.spark jars in .m2 and .ivy2.

@liancheng
Copy link
Contributor Author

liancheng commented Dec 1, 2015

@yhuai Sent PR #10063 to backport this one to master.

asfgit pushed a commit that referenced this pull request Dec 1, 2015
@liancheng @marmbrus
[SPARK-12046][DOC] Fixes various ScalaDoc/JavaDoc issues
69dbe6b 
This PR backports PR #10039 to master

Author: Cheng Lian <lian@databricks.com>

Closes #10063 from liancheng/spark-12046.doc-fix.master.
@marmbrus
Copy link
Contributor

marmbrus commented Dec 1, 2015

Can you close this now?

asfgit pushed a commit that referenced this pull request Dec 1, 2015
@liancheng @yhuai
[SPARK-12046][DOC] Fixes various ScalaDoc/JavaDoc issues
843a31a 
This PR backports PR #10039 to master

Author: Cheng Lian <lian@databricks.com>

Closes #10063 from liancheng/spark-12046.doc-fix.master.

(cherry picked from commit 69dbe6b)
Signed-off-by: Yin Huai <yhuai@databricks.com>
@yhuai
Copy link
Contributor

yhuai commented Dec 1, 2015

#10063 is in both master and branch 1.6. We can close this.

@liancheng liancheng closed this Dec 2, 2015
@liancheng liancheng deleted the spark-12046.doc-fix branch December 2, 2015 07:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

8 participants

@liancheng @srowen @rxin @SparkQA @yhuai @zsxwing @JoshRosen @marmbrus