Skip to content

[SPARK-7277][SQL] Throw exception if the property mapred.reduce.tasks is set to -1 #5811

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
viirya wants to merge 4 commits into from
Closed

[SPARK-7277][SQL] Throw exception if the property mapred.reduce.tasks is set to -1 #5811

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
68a1c70
Show warning message if mapred.reduce.tasks is set to -1.
viirya Apr 30, 2015
4ede705
Throw exception instead of warning message.
viirya May 1, 2015
fd9c817
Merge remote-tracking branch 'upstream/master' into no_neg_reduce_tasks
viirya May 4, 2015
e518f96
Consider other wrong setting values.
viirya May 4, 2015
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 8 additions & 2 deletions sql/core/src/main/scala/org/apache/spark/sql/execution/commands.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -84,8 +84,14 @@ case class SetCommand(
logWarning(
s"Property ${SQLConf.Deprecated.MAPRED_REDUCE_TASKS} is deprecated, " +
s"automatically converted to ${SQLConf.SHUFFLE_PARTITIONS} instead.")
sqlContext.setConf(SQLConf.SHUFFLE_PARTITIONS, value)
Seq(Row(s"${SQLConf.SHUFFLE_PARTITIONS}=$value"))
if (value.toInt < 1) {
val msg = s"Setting negative ${SQLConf.Deprecated.MAPRED_REDUCE_TASKS} for automatically " +
"determining the number of reducers is not supported."
throw new IllegalArgumentException(msg)
} else {
sqlContext.setConf(SQLConf.SHUFFLE_PARTITIONS, value)
Seq(Row(s"${SQLConf.SHUFFLE_PARTITIONS}=$value"))
}

// Configures a single property.
case Some((key, Some(value))) =>
Expand Down
10 changes: 10 additions & 0 deletions sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -862,6 +862,16 @@ class SQLQuerySuite extends QueryTest with BeforeAndAfterAll {
conf.clear()
}

test("SET commands with illegal or inappropriate argument") {
conf.clear()
// Set negative mapred.reduce.tasks for automatically determing
// the number of reducers is not supported
intercept[IllegalArgumentException](sql(s"SET mapred.reduce.tasks=-1"))
intercept[IllegalArgumentException](sql(s"SET mapred.reduce.tasks=-01"))
intercept[IllegalArgumentException](sql(s"SET mapred.reduce.tasks=-2"))
conf.clear()
}

test("apply schema") {
val schema1 = StructType(
StructField("f1", IntegerType, false) ::
Expand Down