SQL: Failing Group By queries due to different ExpressionIds #43072

emasab · 2019-06-10T22:46:48Z

Fix an issue that arises from the use of ExpressionIds as keys in a lookup map
that helps the QueryTranslator to identify the grouping columns. The issue is
that the same expression in different parts of the query (SELECT clause and GROUP BY clause)
ends up with different ExpressionIds so the lookup fails. So, instead of ExpressionIds
use the hashCode() of NamedExpression.

Fixes: #41159
Fixes: #40001
Fixes: #40240
Fixes: #33361
Fixes: #46316
Fixes: #36074
Fixes: #34543
Fixes: #37044

Fixes: #42041

elasticmachine · 2019-06-11T06:13:03Z

Pinging @elastic/es-search

matriv

Hello @emasab!

First of all apologies for the delay on picking up this PR and many thanks for your effort here.
The issue with the ExpressionIds is quite complicated and there are plans to solve it overall through a substantial refactoring. Your changes though, solve a substantial subset of the issues
so we decided to go ahead and proceed with this fix. I've changed the title and description of the PR and link to the issues that it fixes.

For some cases (without aliases or numeric refs) queries still fail, e.g.:
SELECT PI() * emp_no FROM test_emp GROUP BY PI() * emp_no ORDER BY PI() * emp_no LIMIT 10

Moreover I'd like to ask from you to add some integration tests for the issues that this fix addresses:
In agg.sql-spec file: https://github.com/emasab/elasticsearch/blob/bugfix/42041-failing-group-by-queries/x-pack/plugin/sql/qa/src/main/resources/agg.sql-spec#L52 the following:

groupByCastScalar
SELECT CAST(ABS(EXTRACT(YEAR FROM "birth_date")) AS BIGINT) FROM test_emp GROUP BY CAST(ABS(EXTRACT(YEAR FROM "birth_date")) AS BIGINT) ORDER BY CAST(ABS(EXTRACT(YEAR FROM "birth_date")) AS BIGINT) NULLS FIRST;
groupByCastScalarWithAlias
SELECT CAST(ABS(EXTRACT(YEAR FROM "birth_date")) AS BIGINT) as "cast" FROM test_emp GROUP BY "cast" ORDER BY "cast" NULLS FIRST;

and in agg.csv-spec file: https://github.com/emasab/elasticsearch/blob/bugfix/42041-failing-group-by-queries/x-pack/plugin/sql/qa/src/main/resources/agg.csv-spec#L220
the following:

// Fails for H2
groupByCastScalarWithNumericRef
SELECT CAST(ABS(EXTRACT(YEAR FROM "birth_date")) AS BIGINT) FROM test_emp GROUP BY 1 ORDER BY 1 NULLS FIRST;

CAST(ABS(EXTRACT(YEAR FROM "birth_date")) AS BIGINT):l
------------------------------------------------------
null
1952
1953
1954
1955
1956
1957
1958
1959
1960
1961
1962
1963
1964
1965
;

groupByConvertScalar
SELECT CONVERT(ABS(EXTRACT(YEAR FROM "birth_date")), SQL_BIGINT) FROM test_emp GROUP BY CONVERT(ABS(EXTRACT(YEAR FROM "birth_date")), SQL_BIGINT) ORDER BY CONVERT(ABS(EXTRACT(YEAR FROM "birth_date")), SQL_BIGINT) NULLS FIRST;


CONVERT(ABS(EXTRACT(YEAR FROM "birth_date")), SQL_BIGINT):l
-----------------------------------------------------------
null
1952
1953
1954
1955
1956
1957
1958
1959
1960
1961
1962
1963
1964
1965
;


groupByConvertScalarWithAlias
SELECT CONVERT(ABS(EXTRACT(YEAR FROM "birth_date")), SQL_BIGINT) as "convert" FROM test_emp GROUP BY "convert" ORDER BY "convert" NULLS FIRST;

convert:l
---------
null
1952
1953
1954
1955
1956
1957
1958
1959
1960
1961
1962
1963
1964
1965
;

groupByConvertScalarWithNumericRef
SELECT CONVERT(ABS(EXTRACT(YEAR FROM "birth_date")), SQL_BIGINT) FROM test_emp GROUP BY 1 ORDER BY 1 NULLS FIRST;

CONVERT(ABS(EXTRACT(YEAR FROM "birth_date")), SQL_BIGINT):l
-----------------------------------------------------------
null
1952
1953
1954
1955
1956
1957
1958
1959
1960
1961
1962
1963
1964
1965
;

// AwaitsFix https://github.com/elastic/elasticsearch/issues/36074
//groupByConstantScalar
//SELECT PI() * emp_no FROM test_emp GROUP BY PI() * emp_no ORDER BY PI() * emp_no LIMIT 10;

//PI() * emp_no:d
//---------------
//31419.0681285515
//31422.2097212051
//31425.3513138587
//31428.4929065123
//31431.6344991659
//31434.7760918195
//31437.9176844731
//31441.0592771266
//31444.2008697802
//31447.3424624338
//;

groupByConstantScalarWithAlias
SELECT PI() * emp_no AS "value" FROM test_emp GROUP BY value ORDER BY value LIMIT 10;

value:d
-------
31419.0681285515
31422.2097212051
31425.3513138587
31428.4929065123
31431.6344991659
31434.7760918195
31437.9176844731
31441.0592771266
31444.2008697802
31447.3424624338
;

groupByConstantScalarWithNumericRef
SELECT PI() * emp_no FROM test_emp GROUP BY 1 ORDER BY 1 LIMIT 10;

PI() * emp_no:d
---------------
31419.0681285515
31422.2097212051
31425.3513138587
31428.4929065123
31431.6344991659
31434.7760918195
31437.9176844731
31441.0592771266
31444.2008697802
31447.3424624338
;

matriv · 2019-07-29T13:16:26Z

x-pack/plugin/sql/src/test/java/org/elasticsearch/xpack/sql/planner/QueryTranslatorTests.java

@@ -779,6 +779,85 @@ public void testAllCountVariantsWithHavingGenerateCorrectAggregations() {
                        + "\"gap_policy\":\"skip\"}}}}}"));
    }

+    public void testGroupSelection() {


It would be great to split this up to more methods with more descriptive names and add more tests so that all 3 cases (no alias, alias, numeric reference) are covered, similarly to the integration tests in my comment: #43072 (review)
Of course some of the cases, as mentioned, should be skipped as they won't work even with this fix.

emasab · 2019-07-30T14:48:47Z

Hi @matriv , happy to hear from you. I understand, the checks to do are a lot before giving an ok. I'm going to include these integration tests, change the unit tests and see if there is a simple fix for the failing query that you've highlighted.

emasab · 2019-08-03T16:52:50Z

Fixed the groupByConstantScalar case too, mainly unifying the semantic of NamedExpression.equals to that of NamedExpression.hashCode . The ExpressionId is still there but is used only as a unique identifier for the object instance. Changed the semantic of Failure.equals because now two NamedExpression are equal if they have the same tree even if the position in the source is different so Failure takes into account the source position not the parsed node.

emasab · 2019-09-07T17:43:48Z

Sorry, the latest commit

4902015

even if more correct theorically because of the difference between hashCode and equals, solves the specified case and passes the unit tests, causes a lot of failures with the integration test org.elasticsearch.xpack.sql.qa.single_node.JdbcSqlSpecIT. The previous one

8720563

is not affected

emasab · 2019-09-10T22:30:33Z

With the 9b7878b I've solved the integration tests failures while keeping the equals and hashCode methods consistent. The order by constant scalar is still excluded. With my previous commit there was an incomplete change to ResolveMissingRefs that made it work but I've reverted it because this other query failed:

SELECT last_name FROM "test_emp" ORDER BY birth_date::TIME, emp_no LIMIT 5

emasab · 2019-09-19T20:06:06Z

In the latest commit these is a fix for groupByConstantScalar case with ORDER BY, while keeping the equals consistent with the hashCode, and the integration tests successful

The result of the ordering is still not correct in the DESC case or when multiplied by -1 but the query seems to be translated correctly before it's processed by the Querier

emasab · 2019-09-20T18:21:16Z

rebased on current master
changed expected result according to the behavior enforced by testGeoShapeInWhereClause

emasab · 2019-10-09T22:07:53Z

In the latest commit there is a fix for the results ordering when there is a grouped ScalarFunctionAttribute in the ORDER BY clause, like:
SELECT PI() * emp_no FROM test_emp GROUP BY PI() * emp_no ORDER BY PI() * emp_no DESC LIMIT 10;

That should be the last change for this PR, unless other issues come out, you request some changes or rebases to master

x-pack/plugin/sql/qa/src/main/resources/agg.csv-spec

matriv · 2019-10-29T16:25:41Z

@elasticmachine test this please

matriv

Hello @emasab,

Once again thanks a lot for your effort here.
There has been delay in getting back to you, as there were other features and bugs in our plate, and as stated before there is an ongoing effort of complete refactoring in the area of NamedExpression and ExpressionIds.

The state of the PR though looks good overall to me and would be nice to fix those issues independently of the refactoring that might take more time to be ready.

I left another round of comments and I'd like to do some more testing myself using your branch.

x-pack/plugin/sql/src/main/java/org/elasticsearch/xpack/sql/expression/Expressions.java

x-pack/plugin/sql/src/main/java/org/elasticsearch/xpack/sql/expression/gen/script/Param.java

.../plugin/sql/src/main/java/org/elasticsearch/xpack/sql/querydsl/container/QueryContainer.java

x-pack/plugin/sql/src/main/java/org/elasticsearch/xpack/sql/querydsl/agg/Aggs.java

x-pack/plugin/sql/src/main/java/org/elasticsearch/xpack/sql/expression/gen/script/Params.java

x-pack/plugin/sql/src/main/java/org/elasticsearch/xpack/sql/plan/logical/Pivot.java

matriv · 2019-10-29T16:24:25Z

x-pack/plugin/sql/src/main/java/org/elasticsearch/xpack/sql/expression/Expressions.java

+    public static AttributeSet referencesExcept(Set<? extends Expression> exps,
+                                                AttributeSet except) {
+        AttributeSet ret = new AttributeSet();
+        while(exps.size()>0){


I don't get the logic here, why do you need the while loop and revisit the exps which have become filteredExps?

It's a breadth-first search where some nodes are removed during the traversal. At each loop the exps represents the previous list of tree nodes and filteredExps the expressions that aren't in the except set. At the end of the loop exps is replaced with the children of each node. When a leaf is found, its references are added to ret.

code style Co-Authored-By: Marios Trivyzas <matriv@gmail.com>

x-pack/plugin/sql/src/main/java/org/elasticsearch/xpack/sql/plan/logical/Pivot.java

matriv

Thx for addressing the comments @emasab
I left a few more, minor, regarding code formatting.

x-pack/plugin/sql/src/main/java/org/elasticsearch/xpack/sql/expression/Expressions.java

code style Co-Authored-By: Marios Trivyzas <matriv@gmail.com>

emasab · 2019-10-30T20:20:31Z

Thx for addressing the comments @emasab
I left a few more, minor, regarding code formatting.

Now I've set up the save action and the eclipseformat :)

matriv

LGTM. Thank you @emasab for your work, your patience and the effort and time spent on addressing the comments.

matriv · 2019-10-30T22:06:23Z

@elasticmachine test this please

Fix an issue that arises from the use of ExpressionIds as keys in a lookup map that helps the QueryTranslator to identify the grouping columns. The issue is that the same expression in different parts of the query (SELECT clause and GROUP BY clause) ends up with different ExpressionIds so the lookup fails. So, instead of ExpressionIds use the hashCode() of NamedExpression. Fixes: #41159 Fixes: #40001 Fixes: #40240 Fixes: #33361 Fixes: #46316 Fixes: #36074 Fixes: #34543 Fixes: #37044 Fixes: #42041 (cherry picked from commit 3c38ea5)

emasab changed the title ~~bugfix for https://github.com/elastic/elasticsearch/issues/42041~~ SQL: Failing Group By queries due to different ExpressionIds Jun 10, 2019

alpar-t added the :Analytics/SQL SQL querying label Jun 11, 2019

javanna requested a review from matriv July 15, 2019 08:59

matriv added >bug v6.8.3 v7.3.1 v7.4.0 v8.0.0 labels Jul 29, 2019

matriv reviewed Jul 29, 2019

View reviewed changes

jakelandis added v7.3.2 and removed v7.3.1 labels Aug 22, 2019

pcsanwald added v6.8.4 and removed v6.8.3 labels Aug 29, 2019

colings86 added v7.5.0 and removed v7.4.0 labels Aug 30, 2019

$@polyfractal$ polyfractal added v7.3.3 and removed v7.3.2 labels Sep 6, 2019

emasab force-pushed the bugfix/42041-failing-group-by-queries branch from e904df4 to 4111876 Compare September 20, 2019 18:20

emasab force-pushed the bugfix/42041-failing-group-by-queries branch from 4111876 to 4a57627 Compare September 25, 2019 22:48

emasab force-pushed the bugfix/42041-failing-group-by-queries branch from 4a57627 to 2ccb219 Compare October 9, 2019 21:42

tomcallahan added the v6.8.5 label Oct 22, 2019

musteaf mentioned this pull request Oct 29, 2019

SQL: Extend DATE_TRUNC to also operate on intervals(elastic - #46632 ) #47720

Merged

matriv reviewed Oct 29, 2019

View reviewed changes

x-pack/plugin/sql/qa/src/main/resources/agg.csv-spec Outdated Show resolved Hide resolved

matriv reviewed Oct 29, 2019

View reviewed changes

matriv requested a review from costin October 29, 2019 16:34

matriv added v7.4.2 v7.6.0 and removed v7.3.3 labels Oct 29, 2019

emasab and others added 2 commits October 29, 2019 23:27

Apply suggestions from code review

c98df56

code style Co-Authored-By: Marios Trivyzas <matriv@gmail.com>

changes suggested by @matriv

bafec27

matriv reviewed Oct 30, 2019

View reviewed changes

x-pack/plugin/sql/src/main/java/org/elasticsearch/xpack/sql/plan/logical/Pivot.java Outdated Show resolved Hide resolved

matriv reviewed Oct 30, 2019

View reviewed changes

x-pack/plugin/sql/src/main/java/org/elasticsearch/xpack/sql/plan/logical/Pivot.java Outdated Show resolved Hide resolved

matriv reviewed Oct 30, 2019

View reviewed changes

x-pack/plugin/sql/src/main/java/org/elasticsearch/xpack/sql/plan/logical/Pivot.java Outdated Show resolved Hide resolved

matriv reviewed Oct 30, 2019

View reviewed changes

emasab and others added 2 commits October 30, 2019 18:57

Apply suggestions from code review

5c53d91

code style Co-Authored-By: Marios Trivyzas <matriv@gmail.com>

change test case name

b68db36

referencesExcept changed to filterReferences

b442b6a

matriv approved these changes Oct 30, 2019

View reviewed changes

matriv merged commit 3c38ea5 into elastic:master Oct 31, 2019

This was referenced Feb 3, 2020

[meta] 7.6 release elastic/elasticsearch-net#4340

Closed

[meta] 7.6 release elastic/elasticsearch-net#4341

Closed

jakelandis added v8.0.0-alpha1 and removed v8.0.0 labels Jul 26, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

SQL: Failing Group By queries due to different ExpressionIds #43072

SQL: Failing Group By queries due to different ExpressionIds #43072

emasab commented Jun 10, 2019 •

edited by matriv

Loading

elasticmachine commented Jun 11, 2019

matriv left a comment

matriv Jul 29, 2019

emasab commented Jul 30, 2019 •

edited

Loading

emasab commented Aug 3, 2019 •

edited

Loading

emasab commented Sep 7, 2019

emasab commented Sep 10, 2019 •

edited

Loading

emasab commented Sep 19, 2019 •

edited

Loading

emasab commented Sep 20, 2019 •

edited

Loading

emasab commented Oct 9, 2019 •

edited

Loading

matriv commented Oct 29, 2019

matriv left a comment

matriv Oct 29, 2019

emasab Oct 29, 2019

matriv left a comment

emasab commented Oct 30, 2019 •

edited

Loading

matriv left a comment •

edited

Loading

matriv commented Oct 30, 2019

SQL: Failing Group By queries due to different ExpressionIds #43072

SQL: Failing Group By queries due to different ExpressionIds #43072

Conversation

emasab commented Jun 10, 2019 • edited by matriv Loading

elasticmachine commented Jun 11, 2019

matriv left a comment

Choose a reason for hiding this comment

matriv Jul 29, 2019

Choose a reason for hiding this comment

emasab commented Jul 30, 2019 • edited Loading

emasab commented Aug 3, 2019 • edited Loading

emasab commented Sep 7, 2019

emasab commented Sep 10, 2019 • edited Loading

emasab commented Sep 19, 2019 • edited Loading

emasab commented Sep 20, 2019 • edited Loading

emasab commented Oct 9, 2019 • edited Loading

matriv commented Oct 29, 2019

matriv left a comment

Choose a reason for hiding this comment

matriv Oct 29, 2019

Choose a reason for hiding this comment

emasab Oct 29, 2019

Choose a reason for hiding this comment

matriv left a comment

Choose a reason for hiding this comment

emasab commented Oct 30, 2019 • edited Loading

matriv left a comment • edited Loading

Choose a reason for hiding this comment

matriv commented Oct 30, 2019

emasab commented Jun 10, 2019 •

edited by matriv

Loading

emasab commented Jul 30, 2019 •

edited

Loading

emasab commented Aug 3, 2019 •

edited

Loading

emasab commented Sep 10, 2019 •

edited

Loading

emasab commented Sep 19, 2019 •

edited

Loading

emasab commented Sep 20, 2019 •

edited

Loading

emasab commented Oct 9, 2019 •

edited

Loading

emasab commented Oct 30, 2019 •

edited

Loading

matriv left a comment •

edited

Loading