Skip to content

Conversation

@HyukjinKwon
Copy link
Member

@HyukjinKwon HyukjinKwon commented Jul 6, 2016

What changes were proposed in this pull request?

This PR fixes wrongly formatted examples in PySpark documentation as below:

  • SparkSession
    • Before

      2016-07-06 11 34 41

    • After

      2016-07-06 11 33 56

  • Builder
    • Before
      2016-07-06 11 34 44
    • After
      2016-07-06 1 26 37

This PR also fixes several similar instances across the documentation in sql PySpark module.

How was this patch tested?

N/A

@HyukjinKwon HyukjinKwon changed the title [MINOR][PySpark][DOC] Fix SparkSession and Builder API docum [MINOR][PySpark][DOC] Fix code examples of SparkSession and Builder in PySpark documentation Jul 6, 2016
@SparkQA
Copy link

SparkQA commented Jul 6, 2016

Test build #61804 has finished for PR 14063 at commit a8968bf.

  • This patch fails PySpark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@HyukjinKwon
Copy link
Member Author

Oh, I will fix them up soon.

@SparkQA
Copy link

SparkQA commented Jul 6, 2016

Test build #61808 has finished for PR 14063 at commit 1cacb89.

  • This patch fails PySpark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Jul 6, 2016

Test build #61810 has finished for PR 14063 at commit 9edca4f.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@rxin
Copy link
Contributor

rxin commented Jul 6, 2016

Can you verify the output after the all the follow-up commits?

@HyukjinKwon
Copy link
Member Author

HyukjinKwon commented Jul 6, 2016

Sure, It seems the output (representation) for nested class is varied by the version of Python.

First, I updated this to <pyspark.sql.session.SparkSession.Builder object at ...> and got this error below with pypy:

Expected:
    <pyspark.sql.session.SparkSession.Builder object at ...>
Got:
    <pyspark.sql.session.Builder object at 0x000000000163e4b8>

then I updated this to <pyspark.sql.session.Builder object at ...> and got this error below with python 3.4:

Expected:
    <pyspark.sql.session.Builder object at ...>
Got:
    <pyspark.sql.session.SparkSession.Builder object at 0x7f8c5657d9e8>

I tested this with Python 2.7 and 3.4 with this module below:

  • test.py

    class SparkSession(object):
        class Builder(object):
            pass
        builder = Builder()
  • Python 2.7 shell

    >>> from test import SparkSession
    >>> SparkSession.builder
    <test.Builder object at 0x7fa9224fa650>
  • Python 3.4 shell

    >>> from test import SparkSession
    >>> SparkSession.builder
    <test.SparkSession.Builder object at 0x7f11e79e18d0>

@rxin
Copy link
Contributor

rxin commented Jul 6, 2016

Output is fine -- I just want to make sure the doc is correct.

@HyukjinKwon
Copy link
Member Author

Ah, yes it seems fine.

2016-07-06 3 49 58

@rxin
Copy link
Contributor

rxin commented Jul 6, 2016

Can you scan the Python docs and see if there are more instances?

http://people.apache.org/~pwendell/spark-releases/spark-2.0.0-rc2-docs/api/python/pyspark.sql.html

I found at least one ...

@HyukjinKwon
Copy link
Member Author

Sure!

@HyukjinKwon HyukjinKwon changed the title [MINOR][PySpark][DOC] Fix code examples of SparkSession and Builder in PySpark documentation [MINOR][PySpark][DOC] Fix wrongly formatted examples in PySpark documentation Jul 6, 2016
@HyukjinKwon HyukjinKwon changed the title [MINOR][PySpark][DOC] Fix wrongly formatted examples in PySpark documentation [MINOR][PySpark][DOC] Fix wrongly formatted examples in SQL module at PySpark documentation Jul 6, 2016
@HyukjinKwon
Copy link
Member Author

HyukjinKwon commented Jul 6, 2016

I just double-checked the built documentation after the last commit. Do you want me to scan other modules (maybe in another PR) as well?

@rxin
Copy link
Contributor

rxin commented Jul 6, 2016

If you can please check other modules too. Thanks!

@HyukjinKwon
Copy link
Member Author

Sure! Thanks.

@SparkQA
Copy link

SparkQA commented Jul 6, 2016

Test build #61825 has finished for PR 14063 at commit d0672ae.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Jul 6, 2016

Test build #61826 has finished for PR 14063 at commit 607e442.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@HyukjinKwon
Copy link
Member Author

HyukjinKwon commented Jul 6, 2016

@rxin It seems these are all instances related with this (I can't believe!). I just simply corrected indentation for a doc-test in mllib.

@HyukjinKwon HyukjinKwon changed the title [MINOR][PySpark][DOC] Fix wrongly formatted examples in SQL module at PySpark documentation [MINOR][PySpark][DOC] Fix wrongly formatted examples in PySpark documentation Jul 6, 2016
@SparkQA
Copy link

SparkQA commented Jul 6, 2016

Test build #61833 has finished for PR 14063 at commit 673150d.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@rxin
Copy link
Contributor

rxin commented Jul 6, 2016

Merging in master/2.0. Thanks.

@asfgit asfgit closed this in 4e14199 Jul 6, 2016
asfgit pushed a commit that referenced this pull request Jul 6, 2016
…entation

## What changes were proposed in this pull request?

This PR fixes wrongly formatted examples in PySpark documentation as below:

- **`SparkSession`**

  - **Before**

    ![2016-07-06 11 34 41](https://cloud.githubusercontent.com/assets/6477701/16605847/ae939526-436d-11e6-8ab8-6ad578362425.png)

  - **After**

    ![2016-07-06 11 33 56](https://cloud.githubusercontent.com/assets/6477701/16605845/ace9ee78-436d-11e6-8923-b76d4fc3e7c3.png)

- **`Builder`**

  - **Before**
    ![2016-07-06 11 34 44](https://cloud.githubusercontent.com/assets/6477701/16605844/aba60dbc-436d-11e6-990a-c87bc0281c6b.png)

  - **After**
    ![2016-07-06 1 26 37](https://cloud.githubusercontent.com/assets/6477701/16607562/586704c0-437d-11e6-9483-e0af93d8f74e.png)

This PR also fixes several similar instances across the documentation in `sql` PySpark module.

## How was this patch tested?

N/A

Author: hyukjinkwon <gurwls223@gmail.com>

Closes #14063 from HyukjinKwon/minor-pyspark-builder.

(cherry picked from commit 4e14199)
Signed-off-by: Reynold Xin <rxin@databricks.com>
@HyukjinKwon HyukjinKwon deleted the minor-pyspark-builder branch January 2, 2018 03:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants