Skip to content

[SPARK-31171][SQL] size(null) should return null under ansi mode #27936

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
cloud-fan wants to merge 2 commits into from
Closed

[SPARK-31171][SQL] size(null) should return null under ansi mode #27936

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
a1b68cd
size(null) should return null under ansi mode
cloud-fan Mar 17, 2020
8459d63
update conf doc
cloud-fan Mar 17, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 6 additions & 3 deletions sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2154,8 +2154,8 @@ object SQLConf {

val LEGACY_SIZE_OF_NULL = buildConf("spark.sql.legacy.sizeOfNull")
.internal()
.doc("If it is set to true, size of null returns -1. This behavior was inherited from Hive. " +
"The size function returns null for null input if the flag is disabled.")
.doc(s"If it is set to false, or ${ANSI_ENABLED.key} is true, then size of null returns " +
"null. Otherwise, it returns -1, which was inherited from Hive.")
.version("2.4.0")
.booleanConf
.createWithDefault(true)
Expand Down Expand Up @@ -3014,7 +3014,10 @@ class SQLConf extends Serializable with Logging {

def csvColumnPruning: Boolean = getConf(SQLConf.CSV_PARSER_COLUMN_PRUNING)

def legacySizeOfNull: Boolean = getConf(SQLConf.LEGACY_SIZE_OF_NULL)
def legacySizeOfNull: Boolean = {
// size(null) should return null under ansi mode.
getConf(SQLConf.LEGACY_SIZE_OF_NULL) && !getConf(ANSI_ENABLED)
Copy link
Member

@dongjoon-hyun dongjoon-hyun Mar 17, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, ansi conf overrides legacy conf ? Do we have another legacy conf like this? Is it consistent with the other legacy? Could you update the migration guide and conf document to be consistent, too?

cc @gatorsmile

Copy link
Contributor Author

@cloud-fan cloud-fan Mar 17, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the first one, and I don't think we need a migration guide as the behavior doesn't change by default.

I'll update the config doc.

Copy link
Member

@dongjoon-hyun dongjoon-hyun Mar 17, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you!

}

def isReplEagerEvalEnabled: Boolean = getConf(SQLConf.REPL_EAGER_EVAL_ENABLED)

Expand Down
6 changes: 6 additions & 0 deletions ...src/test/scala/org/apache/spark/sql/catalyst/expressions/CollectionExpressionsSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -74,6 +74,12 @@ class CollectionExpressionsSuite extends SparkFunSuite with ExpressionEvalHelper
withSQLConf(SQLConf.LEGACY_SIZE_OF_NULL.key -> "false") {
testSize(sizeOfNull = null)
}
// size(null) should return null under ansi mode.
withSQLConf(
SQLConf.LEGACY_SIZE_OF_NULL.key -> "true",
SQLConf.ANSI_ENABLED.key -> "true") {
testSize(sizeOfNull = null)
}
}

test("MapKeys/MapValues") {
Expand Down
12 changes: 12 additions & 0 deletions sql/core/src/test/scala/org/apache/spark/sql/DataFrameFunctionsSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -490,6 +490,12 @@ class DataFrameFunctionsSuite extends QueryTest with SharedSparkSession {
withSQLConf(SQLConf.LEGACY_SIZE_OF_NULL.key -> "false") {
testSizeOfArray(sizeOfNull = null)
}
// size(null) should return null under ansi mode.
withSQLConf(
SQLConf.LEGACY_SIZE_OF_NULL.key -> "true",
SQLConf.ANSI_ENABLED.key -> "true") {
testSizeOfArray(sizeOfNull = null)
}
}

test("dataframe arrays_zip function") {
Expand Down Expand Up @@ -569,6 +575,12 @@ class DataFrameFunctionsSuite extends QueryTest with SharedSparkSession {
withSQLConf(SQLConf.LEGACY_SIZE_OF_NULL.key -> "false") {
testSizeOfMap(sizeOfNull = null)
}
// size(null) should return null under ansi mode.
withSQLConf(
SQLConf.LEGACY_SIZE_OF_NULL.key -> "true",
SQLConf.ANSI_ENABLED.key -> "true") {
testSizeOfMap(sizeOfNull = null)
}
}

test("map_keys/map_values function") {
Expand Down