CALCITE-6804 Anti-join with WHERE NOT EXISTS syntax has corrupted condition

[antonkw]

I don't expect that it can be merged as is.
In the first order, I want to demonstrate the issue with anti joins: https://issues.apache.org/jira/browse/CALCITE-6804

For any anti-join with input (left-side) more complex than single table scan, rex sub query can't be converted to SQL.
Reason: Filter's node input is not considered as something that can be referenced (via alias) in condition.
But anti-join is that exact case.

Example of logical plan:

LogicalProject(product_id=[$0])
  LogicalFilter(condition=[NOT(EXISTS({
LogicalFilter(condition=[=($cor0.product_id, $1)])
  JdbcTableScan(table=[[foodmart, product]])
}))], variablesSet=[[$cor0]])
    LogicalProject(product_id=[$1])
      LogicalJoin(condition=[=($1, $16)], joinType=[left])
        JdbcTableScan(table=[[foodmart, product]])
        JdbcTableScan(table=[[foodmart, product]])

$cor0 should eventually refer to whole input aliased as t.
Without proposed change $cor0 is being converted into JdbcTableScan(table=[[foodmart, product]])

[mihaibudiu]

why are these the only conditions that require a reset alias?

[suibianwanwank]

I suspect this doesn't cover all cases. e.g.
e2."product_id" = selected."product_id" and e2."product_id" > 10
How about resetting on parseCorrelTable?

+++ b/core/src/main/java/org/apache/calcite/rel/rel2sql/RelToSqlConverter.java
@@ -1406,7 +1406,7 @@ public List<SqlNode> createAsFullOperands(RelDataType rowType, SqlNode leftOpera
 
   private void parseCorrelTable(RelNode relNode, Result x) {
     for (CorrelationId id : relNode.getVariablesSet()) {
-      correlTableMap.put(id, x.qualifiedContext());
+      correlTableMap.put(id, x.resetAlias().qualifiedContext());
     }
   }

[antonkw]

@mihaibudiu

why are these the only conditions that require a reset alias?

My take here is that typically predicates can't access the input via aliases. At least, for many SQL engines user ought to move the input into CTE to refer to it in filtering conditions. And JOINs with EXISTS/NOT EXISTS syntax are exceptions in that regard.

I thought about explicit function with name like canAccessInputViaAlias that processes condition. But it seems too nit-pick'y given the fact that I am sure only about two greenlighting keywords

[antonkw]

@suibianwanwank
Can you please elaborate on what you suspected?

I added the predicate you mentioned:

  @Test void testAntiJoinWithComplexInput() {
    final String sql = "SELECT * FROM "
        + "(select * from ("
        + "select e1.\"product_id\" FROM \"foodmart\".\"product\" e1 "
        + "LEFT JOIN \"foodmart\".\"product\" e3 "
        + "on e1.\"product_id\" = e3.\"product_id\""
        + ")"
        + ") selected where not exists\n"
        + "(select 1 from \"foodmart\".\"product\" e2 "
        + "where e2.\"product_id\" = selected.\"product_id\" and e2.\"product_id\" > 10)";
    final String expected =
        "SELECT *\nFROM (SELECT \"product\".\"product_id\"\nFROM \"foodmart\".\"product\"\n"
            + "LEFT JOIN \"foodmart\".\"product\" AS \"product0\" "
            + "ON \"product\".\"product_id\" = \"product0\".\"product_id\") AS \"t\"\n"
            + "WHERE NOT EXISTS ("
            + "SELECT *\nFROM \"foodmart\".\"product\"\n"
            + "WHERE \"product_id\" = \"t\".\"product_id\" AND \"product_id\" > 10"
            + ")";
    sql(sql).ok(expected);
  }

Without escaping mess:

SELECT * FROM 
  foodmart.product 
WHERE 
  product_id = t.product_id AND product_id > 10

Seems good to me.

How about resetting on parseCorrelTable?

Feels unsafe to me. Particularly due to fact that such aliases are not legitimate in all cases(why we should represent and maintain state that isn't reflecting possible query).
I can try your suggestion to see what tests think about it :)

[suibianwanwank]

Apologies, you are right. I misunderstood the meaning here.

[mihaibudiu]

Please use a commit message and a PR title that match exactly the issue in JIRA.
They are used to keep track of issues.

[suibianwanwank]

@antonkw Maybe I’m overthinking it, but I still believe that only considering NOT and EXISTS doesn't cover all possible cases for subqueries. Could you try the example with

    final String sql = "SELECT * FROM "
        + "(select * from ("
        + "select e1.\"product_id\" FROM \"foodmart\".\"product\" e1 "
        + "LEFT JOIN \"foodmart\".\"product\" e3 "
        + "on e1.\"product_id\" = e3.\"product_id\""
        + ")"
        + ") selected where 1 in\n"
        + "(select \"gross_weight\" from \"foodmart\".\"product\" e2 "
        + "where e2.\"product_id\" = selected.\"product_id\" and e2.\"product_id\" > 10)";

and see if it works?

[antonkw]

@suibianwanwank

\"product_id\" = \"product\".\"product_class_id\"

here we go again :)

Added IN match and test has repaired.

  @Test void testFilterWithSubQuery() {
    final String sql = "SELECT * FROM "
        + "(select * from ("
        + "select e1.\"product_id\" FROM \"foodmart\".\"product\" e1 "
        + "LEFT JOIN \"foodmart\".\"product\" e3 "
        + "on e1.\"product_id\" = e3.\"product_id\""
        + ")"
        + ") selected where 1 in\n"
        + "(select \"gross_weight\" from \"foodmart\".\"product\" e2 "
        + "where e2.\"product_id\" = selected.\"product_id\" and e2.\"product_id\" > 10)";

    final String expected =
        "SELECT *\nFROM (SELECT \"product\".\"product_id\"\nFROM \"foodmart\".\"product\"\n"
            + "LEFT JOIN \"foodmart\".\"product\" AS \"product0\" "
            + "ON \"product\".\"product_id\" = \"product0\".\"product_id\") AS \"t\"\n"
            + "WHERE CAST(1 AS DOUBLE) IN ("
            + "SELECT \"gross_weight\"\nFROM \"foodmart\".\"product\"\n"
            + "WHERE \"product_id\" = \"t\".\"product_id\" AND \"product_id\" > 10)";

    sql(sql).ok(expected);
  }

Waiting for CI to see if everything IN-related is good

[sonarqubecloud]

Quality Gate passed

Issues
2 New issues
0 Accepted issues

Measures
0 Security Hotspots
100.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[antonkw]

@suibianwanwank
I added the test with IN clause.
CI is green after catching IN sql-kind of conditions.