Skip to content

Commit aaa3dcc

Browse files
AngersZhuuuumaropu
authored andcommitted
[SPARK-34012][SQL][3.0] Keep behavior consistent when conf spark.sqllegacy.parser.havingWithoutGroupByAsWhere is true with migration guide
### What changes were proposed in this pull request? In #22696 we support HAVING without GROUP BY means global aggregate But since we treat having as Filter before, in this way will cause a lot of analyze error, after #28294 we use `UnresolvedHaving` to instead `Filter` to solve such problem, but break origin logical about treat `SELECT 1 FROM range(10) HAVING true` as `SELECT 1 FROM range(10) WHERE true` . This PR fix this issue and add UT. NOTE: This backport comes from #31039 ### Why are the changes needed? Keep consistent behavior of migration guide. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? added UT Closes #31049 from AngersZhuuuu/SPARK-34012-3.0. Authored-by: angerszhu <angers.zhu@gmail.com> Signed-off-by: Takeshi Yamamuro <yamamuro@apache.org>
1 parent 403bca4 commit aaa3dcc

File tree

3 files changed

+77
-2
lines changed

3 files changed

+77
-2
lines changed

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -723,7 +723,11 @@ class AstBuilder(conf: SQLConf) extends SqlBaseBaseVisitor[AnyRef] with Logging
723723
val withProject = if (aggregationClause == null && havingClause != null) {
724724
if (conf.getConf(SQLConf.LEGACY_HAVING_WITHOUT_GROUP_BY_AS_WHERE)) {
725725
// If the legacy conf is set, treat HAVING without GROUP BY as WHERE.
726-
withHavingClause(havingClause, createProject())
726+
val predicate = expression(havingClause.booleanExpression) match {
727+
case p: Predicate => p
728+
case e => Cast(e, BooleanType)
729+
}
730+
Filter(predicate, createProject())
727731
} else {
728732
// According to SQL standard, HAVING without GROUP BY means global aggregate.
729733
withHavingClause(havingClause, Aggregate(Nil, namedExpressions, withFilter))

sql/core/src/test/resources/sql-tests/inputs/group-by.sql

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -86,6 +86,16 @@ SELECT 1 FROM range(10) HAVING MAX(id) > 0;
8686

8787
SELECT id FROM range(10) HAVING id > 0;
8888

89+
SET spark.sql.legacy.parser.havingWithoutGroupByAsWhere=true;
90+
91+
SELECT 1 FROM range(10) HAVING true;
92+
93+
SELECT 1 FROM range(10) HAVING MAX(id) > 0;
94+
95+
SELECT id FROM range(10) HAVING id > 0;
96+
97+
SET spark.sql.legacy.parser.havingWithoutGroupByAsWhere=false;
98+
8999
-- Test data
90100
CREATE OR REPLACE TEMPORARY VIEW test_agg AS SELECT * FROM VALUES
91101
(1, true), (1, false),

sql/core/src/test/resources/sql-tests/results/group-by.sql.out

Lines changed: 62 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
-- Automatically generated by SQLQueryTestSuite
2-
-- Number of queries: 56
2+
-- Number of queries: 61
33

44

55
-- !query
@@ -277,6 +277,67 @@ org.apache.spark.sql.AnalysisException
277277
grouping expressions sequence is empty, and '`id`' is not an aggregate function. Wrap '()' in windowing function(s) or wrap '`id`' in first() (or first_value) if you don't care which value you get.;
278278

279279

280+
-- !query
281+
SET spark.sql.legacy.parser.havingWithoutGroupByAsWhere=true
282+
-- !query schema
283+
struct<key:string,value:string>
284+
-- !query output
285+
spark.sql.legacy.parser.havingWithoutGroupByAsWhere true
286+
287+
288+
-- !query
289+
SELECT 1 FROM range(10) HAVING true
290+
-- !query schema
291+
struct<1:int>
292+
-- !query output
293+
1
294+
1
295+
1
296+
1
297+
1
298+
1
299+
1
300+
1
301+
1
302+
1
303+
304+
305+
-- !query
306+
SELECT 1 FROM range(10) HAVING MAX(id) > 0
307+
-- !query schema
308+
struct<>
309+
-- !query output
310+
org.apache.spark.sql.AnalysisException
311+
312+
Aggregate/Window/Generate expressions are not valid in where clause of the query.
313+
Expression in where clause: [(max(`id`) > CAST(0 AS BIGINT))]
314+
Invalid expressions: [max(`id`)];
315+
316+
317+
-- !query
318+
SELECT id FROM range(10) HAVING id > 0
319+
-- !query schema
320+
struct<id:bigint>
321+
-- !query output
322+
1
323+
2
324+
3
325+
4
326+
5
327+
6
328+
7
329+
8
330+
9
331+
332+
333+
-- !query
334+
SET spark.sql.legacy.parser.havingWithoutGroupByAsWhere=false
335+
-- !query schema
336+
struct<key:string,value:string>
337+
-- !query output
338+
spark.sql.legacy.parser.havingWithoutGroupByAsWhere false
339+
340+
280341
-- !query
281342
CREATE OR REPLACE TEMPORARY VIEW test_agg AS SELECT * FROM VALUES
282343
(1, true), (1, false),

0 commit comments

Comments
 (0)