Skip to content

[SPARK-23619][DOCS] Add output description for some generator expressions / functions #23748

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
jashgala wants to merge 8 commits into from
Closed

[SPARK-23619][DOCS] Add output description for some generator expressions / functions #23748

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
82e217b
[SPARK-23619][SQL] Added default column names for explode & posexplod…
jashgala Feb 8, 2019
e7a34eb
Added more comments for explode_outer and posexplode_outer
jashgala Feb 9, 2019
4ed035b
Added default column names in ExpressionDescription for explode and p…
jashgala Feb 9, 2019
e5751ed
Added default column names for inline and stack in comments
jashgala Feb 22, 2019
80c0649
Added scala style line-size exception to prevent failure due to comment
jashgala Feb 22, 2019
d029d3c
Fixed typo in scala style exceptions in comments
jashgala Feb 22, 2019
0313b59
Added default column names in comments for explode, etc. functions in R
jashgala Feb 22, 2019
d0dea9c
Added default column names in comments for explode, etc. functions in…
jashgala Feb 22, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 10 additions & 2 deletions R/pkg/R/functions.R
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3568,6 +3568,8 @@ setMethod("element_at",

#' @details
#' \code{explode}: Creates a new row for each element in the given array or map column.
#' Uses the default column name \code{col} for elements in the array and
#' \code{key} and \code{value} for elements in the map unless specified otherwise.
#'
#' @rdname column_collection_functions
#' @aliases explode explode,Column-method
Expand Down Expand Up @@ -3628,7 +3630,9 @@ setMethod("sort_array",

#' @details
#' \code{posexplode}: Creates a new row for each element with position in the given array
#' or map column.
#' or map column. Uses the default column name \code{pos} for position, and \code{col}
#' for elements in the array and \code{key} and \code{value} for elements in the map
#' unless specified otherwise.
#'
#' @rdname column_collection_functions
#' @aliases posexplode posexplode,Column-method
Expand Down Expand Up @@ -3769,7 +3773,8 @@ setMethod("repeat_string",
#' \code{explode}: Creates a new row for each element in the given array or map column.
#' Unlike \code{explode}, if the array/map is \code{null} or empty
#' then \code{null} is produced.
#'
#' Uses the default column name \code{col} for elements in the array and
#' \code{key} and \code{value} for elements in the map unless specified otherwise.
#'
#' @rdname column_collection_functions
#' @aliases explode_outer explode_outer,Column-method
Expand All @@ -3794,6 +3799,9 @@ setMethod("explode_outer",
#' \code{posexplode_outer}: Creates a new row for each element with position in the given
#' array or map column. Unlike \code{posexplode}, if the array/map is \code{null} or empty
#' then the row (\code{null}, \code{null}) is produced.
#' Uses the default column name \code{pos} for position, and \code{col}
#' for elements in the array and \code{key} and \code{value} for elements in the map
#' unless specified otherwise.
#'
#' @rdname column_collection_functions
#' @aliases posexplode_outer posexplode_outer,Column-method
Expand Down
20 changes: 16 additions & 4 deletions python/pyspark/sql/functions.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2088,7 +2088,10 @@ def array_except(col1, col2):

@since(1.4)
def explode(col):
"""Returns a new row for each element in the given array or map.
"""
Returns a new row for each element in the given array or map.
Uses the default column name `col` for elements in the array and
`key` and `value` for elements in the map unless specified otherwise.

>>> from pyspark.sql import Row
>>> eDF = spark.createDataFrame([Row(a=1, intlist=[1,2,3], mapfield={"a": "b"})])
Expand All @@ -2109,7 +2112,10 @@ def explode(col):

@since(2.1)
def posexplode(col):
"""Returns a new row for each element with position in the given array or map.
"""
Returns a new row for each element with position in the given array or map.
Uses the default column name `pos` for position, and `col` for elements in the
array and `key` and `value` for elements in the map unless specified otherwise.

>>> from pyspark.sql import Row
>>> eDF = spark.createDataFrame([Row(a=1, intlist=[1,2,3], mapfield={"a": "b"})])
Expand All @@ -2130,8 +2136,11 @@ def posexplode(col):

@since(2.3)
def explode_outer(col):
"""Returns a new row for each element in the given array or map.
"""
Returns a new row for each element in the given array or map.
Unlike explode, if the array/map is null or empty then null is produced.
Uses the default column name `col` for elements in the array and
`key` and `value` for elements in the map unless specified otherwise.

>>> df = spark.createDataFrame(
... [(1, ["foo", "bar"], {"x": 1.0}), (2, [], {}), (3, None, None)],
Expand Down Expand Up @@ -2163,8 +2172,11 @@ def explode_outer(col):

@since(2.3)
def posexplode_outer(col):
"""Returns a new row for each element with position in the given array or map.
"""
Returns a new row for each element with position in the given array or map.
Unlike posexplode, if the array/map is null or empty then the row (null, null) is produced.
Uses the default column name `pos` for position, and `col` for elements in the
array and `key` and `value` for elements in the map unless specified otherwise.

>>> df = spark.createDataFrame(
... [(1, ["foo", "bar"], {"x": 1.0}), (2, [], {}), (3, None, None)],
Expand Down
12 changes: 8 additions & 4 deletions sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/generators.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -127,14 +127,16 @@ case class UserDefinedGenerator(
* 3 NULL
* }}}
*/
// scalastyle:off line.size.limit
@ExpressionDescription(
usage = "_FUNC_(n, expr1, ..., exprk) - Separates `expr1`, ..., `exprk` into `n` rows.",
usage = "_FUNC_(n, expr1, ..., exprk) - Separates `expr1`, ..., `exprk` into `n` rows. Uses column names col0, col1, etc. by default unless specified otherwise.",
examples = """
Examples:
> SELECT _FUNC_(2, 1, 2, 3);
1 2
3 NULL
""")
// scalastyle:on line.size.limit
case class Stack(children: Seq[Expression]) extends Generator {

private lazy val numRows = children.head.eval().asInstanceOf[Int]
Expand Down Expand Up @@ -352,7 +354,7 @@ abstract class ExplodeBase extends UnaryExpression with CollectionGenerator with
*/
// scalastyle:off line.size.limit
@ExpressionDescription(
usage = "_FUNC_(expr) - Separates the elements of array `expr` into multiple rows, or the elements of map `expr` into multiple rows and columns.",
usage = "_FUNC_(expr) - Separates the elements of array `expr` into multiple rows, or the elements of map `expr` into multiple rows and columns. Unless specified otherwise, uses the default column name `col` for elements of the array or `key` and `value` for the elements of the map.",
examples = """
Examples:
> SELECT _FUNC_(array(10, 20));
Expand All @@ -375,7 +377,7 @@ case class Explode(child: Expression) extends ExplodeBase {
*/
// scalastyle:off line.size.limit
@ExpressionDescription(
usage = "_FUNC_(expr) - Separates the elements of array `expr` into multiple rows with positions, or the elements of map `expr` into multiple rows and columns with positions.",
usage = "_FUNC_(expr) - Separates the elements of array `expr` into multiple rows with positions, or the elements of map `expr` into multiple rows and columns with positions. Unless specified otherwise, uses the column name `pos` for position, `col` for elements of the array or `key` and `value` for elements of the map.",
examples = """
Examples:
> SELECT _FUNC_(array(10,20));
Expand All @@ -390,14 +392,16 @@ case class PosExplode(child: Expression) extends ExplodeBase {
/**
* Explodes an array of structs into a table.
*/
// scalastyle:off line.size.limit
@ExpressionDescription(
usage = "_FUNC_(expr) - Explodes an array of structs into a table.",
usage = "_FUNC_(expr) - Explodes an array of structs into a table. Uses column names col1, col2, etc. by default unless specified otherwise.",
examples = """
Examples:
> SELECT _FUNC_(array(struct(1, 'a'), struct(2, 'b')));
1 a
2 b
""")
// scalastyle:on line.size.limit
case class Inline(child: Expression) extends UnaryExpression with CollectionGenerator {
override val inline: Boolean = true
override val position: Boolean = false
Expand Down
8 changes: 8 additions & 0 deletions sql/core/src/main/scala/org/apache/spark/sql/functions.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3305,6 +3305,8 @@ object functions {

/**
* Creates a new row for each element in the given array or map column.
* Uses the default column name `col` for elements in the array and
* `key` and `value` for elements in the map unless specified otherwise.
Copy link
Member

@HyukjinKwon HyukjinKwon Feb 11, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's also update Python doc and R doc while we're here. Take a look for functions.py and functions.R

Copy link
Member

@maropu maropu Feb 12, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oh... I forgot the viewpoint...thanks.

*
* @group collection_funcs
* @since 1.3.0
Expand All @@ -3313,6 +3315,8 @@ object functions {

/**
* Creates a new row for each element in the given array or map column.
* Uses the default column name `col` for elements in the array and
* `key` and `value` for elements in the map unless specified otherwise.
* Unlike explode, if the array/map is null or empty then null is produced.
*
* @group collection_funcs
Expand All @@ -3322,6 +3326,8 @@ object functions {

/**
* Creates a new row for each element with position in the given array or map column.
* Uses the default column name `pos` for position, and `col` for elements in the array
* and `key` and `value` for elements in the map unless specified otherwise.
*
* @group collection_funcs
* @since 2.1.0
Expand All @@ -3330,6 +3336,8 @@ object functions {

/**
* Creates a new row for each element with position in the given array or map column.
* Uses the default column name `pos` for position, and `col` for elements in the array
* and `key` and `value` for elements in the map unless specified otherwise.
* Unlike posexplode, if the array/map is null or empty then the row (null, null) is produced.
*
* @group collection_funcs
Expand Down