Skip to content

[SPARK-29203][SQL][TESTS][2.4] Reduce shuffle partitions in SQLQueryTestSuite #25938

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wangyum wants to merge 1 commit into from
Closed

[SPARK-29203][SQL][TESTS][2.4] Reduce shuffle partitions in SQLQueryTestSuite #25938

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion sql/core/src/test/resources/sql-tests/inputs/subquery/in-subquery/in-joins.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -83,7 +83,7 @@ GROUP BY t1a,
t3a,
t3b,
t3c
ORDER BY t1a DESC, t3b DESC;
ORDER BY t1a DESC, t3b DESC, t3c ASC;

-- TC 01.03
SELECT Count(DISTINCT(t1a))
Expand Down
2 changes: 1 addition & 1 deletion sql/core/src/test/resources/sql-tests/inputs/subquery/in-subquery/not-in-joins.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -128,7 +128,7 @@ GROUP BY t1b,
HAVING t1d NOT IN (SELECT t2d
FROM t2
WHERE t1d = t2d)
ORDER BY t1b DESC;
ORDER BY t1b DESC, t1d ASC;

-- TC 01.05
SELECT COUNT(DISTINCT(t1a)),
Expand Down
2 changes: 1 addition & 1 deletion sql/core/src/test/resources/sql-tests/results/subquery/in-subquery/in-joins.sql.out
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -102,7 +102,7 @@ GROUP BY t1a,
t3a,
t3b,
t3c
ORDER BY t1a DESC, t3b DESC
ORDER BY t1a DESC, t3b DESC, t3c ASC
-- !query 4 schema
struct<count(DISTINCT t1a):bigint,t1b:smallint,t3a:string,t3b:smallint,t3c:int>
-- !query 4 output
Expand Down
2 changes: 1 addition & 1 deletion sql/core/src/test/resources/sql-tests/results/subquery/in-subquery/not-in-joins.sql.out
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -169,7 +169,7 @@ GROUP BY t1b,
HAVING t1d NOT IN (SELECT t2d
FROM t2
WHERE t1d = t2d)
ORDER BY t1b DESC
ORDER BY t1b DESC, t1d ASC
-- !query 6 schema
struct<count(DISTINCT t1a):bigint,t1b:smallint,t1c:int,t1d:bigint>
-- !query 6 output
Expand Down
6 changes: 6 additions & 0 deletions sql/core/src/test/scala/org/apache/spark/sql/SQLQueryTestSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,11 +22,13 @@ import java.util.{Locale, TimeZone}

import scala.util.control.NonFatal

import org.apache.spark.SparkConf
import org.apache.spark.sql.catalyst.planning.PhysicalOperation
import org.apache.spark.sql.catalyst.plans.logical._
import org.apache.spark.sql.catalyst.rules.RuleExecutor
import org.apache.spark.sql.catalyst.util.{fileToString, stringToFile}
import org.apache.spark.sql.execution.command.{DescribeColumnCommand, DescribeTableCommand}
import org.apache.spark.sql.internal.SQLConf
import org.apache.spark.sql.test.SharedSQLContext
import org.apache.spark.sql.types.StructType

Expand Down Expand Up @@ -98,6 +100,10 @@ class SQLQueryTestSuite extends QueryTest with SharedSQLContext {
private val inputFilePath = new File(baseResourcePath, "inputs").getAbsolutePath
private val goldenFilePath = new File(baseResourcePath, "results").getAbsolutePath

override def sparkConf: SparkConf = super.sparkConf
Copy link
Member

@gatorsmile gatorsmile Oct 4, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should be protected, right? In the parent trait SharedSparkSession, it is protected.

protected override def sparkConf ....

Copy link
Member

@xuanyuanking xuanyuanking Oct 4, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in #26019.

Copy link
Member

@HyukjinKwon HyukjinKwon Oct 4, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, seems adding a protected modifier is correct, though, we don't usually make separate PRs to address minor nits. Since the PR is open, I guess it's fine.

Copy link
Member

@xuanyuanking xuanyuanking Oct 4, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, copy that.

// Fewer shuffle partitions to speed up testing.
.set(SQLConf.SHUFFLE_PARTITIONS, 4)

/** List of test cases to ignore, in lower cases. */
private val blackList = Set(
"blacklist.sql", // Do NOT remove this one. It is here to test the blacklist functionality.
Expand Down