[FLINK-35855][table] Upgrade Calcite to 1.36.0 #26827

snuyanzin · 2025-07-23T22:01:05Z

The PR bumps Calcite to 1.36.0

Brief change log

Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): (yes )
The public API, i.e., is any changed class annotated with @Public(Evolving): (no)
The serializers: ( no)
The runtime per-record code paths (performance sensitive): ( no)
Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (no)
The S3 file system connector: ( no)

Documentation

Does this pull request introduce a new feature? ( no)
If yes, how is the feature documented? (not applicable)

flinkbot · 2025-07-23T22:08:29Z

CI report:

4835d9d Azure: SUCCESS

Bot commands

The @flinkbot bot supports the following commands:

@flinkbot run azure re-run the last Azure build

snuyanzin · 2025-07-24T01:16:44Z

flink-table/flink-sql-parser/src/main/codegen/templates/Parser.jj

+    SqlNode e;
+ }
+ {
+     <CONTAINS_SUBSTR> { s = span(); }


checked with Flink: currently fails on validation

snuyanzin · 2025-07-24T01:16:58Z

flink-table/flink-sql-parser/src/main/codegen/templates/Parser.jj

+    |   <PERCENTILE_CONT>
+    |   <PERCENTILE_DISC>


checked with Flink: currently fail on validation

snuyanzin · 2025-07-24T01:17:36Z

flink-table/flink-sql-parser/src/main/java/org/apache/calcite/sql/SqlMapTypeNameSpec.java

+        // Default is not null.
+        if (Boolean.TRUE.equals(keyType.getNullable())) {
+            // BEGIN FLINK MODIFICATION
+            // writer.keyword("NULL");


so far we don't have NULL type

snuyanzin · 2025-07-24T01:19:49Z

flink-table/flink-table-planner/src/main/java/org/apache/calcite/runtime/SqlFunctions.java

+ */
+@SuppressWarnings("UnnecessaryUnboxing")
+@Deterministic
+public class SqlFunctions {


There is a bug in Calcite breaking shading
https://issues.apache.org/jira/browse/CALCITE-6393
this is the reason we have this class here (no Flink changes so far)

...link-table-planner/src/main/java/org/apache/calcite/sql/util/ReflectiveSqlOperatorTable.java

snuyanzin · 2025-07-24T01:23:09Z

...r/src/test/resources/org/apache/flink/table/planner/plan/batch/sql/PartitionableSinkTest.xml

@@ -101,14 +101,14 @@ Sink(table=[default_catalog.default_database.sink], fields=[a, EXPR$1, EXPR$2])
    <Resource name="ast">
      <![CDATA[
 LogicalSink(table=[default_catalog.default_database.sink], fields=[a, b, c])
-+- LogicalProject(a=[5:BIGINT], b=[1:BIGINT], c=[1:BIGINT])
+- LogicalProject(a=[CAST(5:BIGINT):BIGINT], b=[CAST(1:BIGINT):BIGINT], c=[CAST(1:BIGINT):BIGINT])


side effect of WITH functionality.

It only impacts PartitionableSink cases

snuyanzin · 2025-07-24T01:24:06Z

flink-table/flink-table-planner/pom.xml

+		<dependency>
+			<groupId>commons-codec</groupId>
+			<artifactId>commons-codec</artifactId>
+			<version>1.15</version>
+			<optional>${flink.markBundledAsOptional}</optional>
+		</dependency>
+		<dependency>
+			<groupId>org.apache.commons</groupId>
+			<artifactId>commons-lang3</artifactId>
+			<version>3.12.0</version>
+			<optional>${flink.markBundledAsOptional}</optional>
+		</dependency>


required to compile SqlFunctions

twalthr

Thank you very much @snuyanzin. Looks mostly good to me.

twalthr · 2025-07-25T10:41:23Z

flink-table/flink-table-planner/src/main/java/org/apache/calcite/sql2rel/SqlToRelConverter.java

@@ -2511,6 +2529,11 @@ protected void convertFrom(
                // ----- FLINK MODIFICATION END -----
                return;

+            case WITH_ITEM_TABLE_REF:


Can we add a test here? Even if it is just error behavior.

Added RecursiveWithTest currently it fails while planning

...link-table-planner/src/main/java/org/apache/calcite/sql/util/ReflectiveSqlOperatorTable.java

flink-table/flink-table-planner/pom.xml

...table/flink-sql-parser/src/test/java/org/apache/flink/sql/parser/FlinkSqlParserImplTest.java

twalthr · 2025-07-25T11:16:30Z

flink-table/flink-sql-client/src/test/resources/sql/table.q

   +- Values(type=[RecordType(INTEGER ZERO)], tuples=[[{ 0 }]])

 == Optimized Execution Plan ==
 Sink(table=[default_catalog.default_database.daily_orders], fields=[user, product, amount, dt])
-+- Calc(select=[123 AS user, 'toothpick' AS product, 1 AS amount, '2022-06-12' AS dt])
+- Calc(select=[CAST(123 AS BIGINT) AS user, CAST('toothpick' AS VARCHAR(2147483647)) AS product, CAST(1 AS INTEGER) AS amount, CAST('2022-06-12' AS VARCHAR(2147483647)) AS dt])


Would be great to get a bit more explanation on the root cause of this change.

this line is the reason

flink/flink-table/flink-table-planner/src/main/java/org/apache/calcite/rex/RexUtil.java

Line 147 in 4835d9d

castExps.add(rexBuilder.makeCast(lhsType, rhsExp, true, false));

before that it was

flink/flink-table/flink-table-planner/src/main/java/org/apache/calcite/rex/RexUtil.java

Line 145 in 9fb8a2e

castExps.add(rexBuilder.makeCast(lhsType, rhsExp));

after that it takes nullability into account, and here is cast because of different nullability

snuyanzin · 2025-07-28T08:22:03Z

...flink-sql-parser/src/main/java/org/apache/flink/sql/parser/validate/FlinkSqlConformance.java

@@ -114,7 +119,7 @@ public boolean allowAliasUnnestItems() {

    @Override
    public boolean allowNiladicParentheses() {
-        return false;
+        return true;


Allow niladic parentheses
from javadoc

* Whether to allow parentheses to be specified in calls to niladic functions * and procedures (that is, functions and procedures with no parameters). * * <p>For example, {@code CURRENT_DATE} is a niladic system function. In * standard SQL it must be invoked without parentheses: * * <blockquote><code>VALUES CURRENT_DATE</code></blockquote> * * <p>If {@code allowNiladicParentheses}, the following syntax is also valid:

seems it matches out case
https://github.com/apache/calcite/blob/3719447e125982aaa47f0a7b2564f94b9900539e/core/src/main/java/org/apache/calcite/sql/validate/SqlConformance.java#L366-L390

snuyanzin · 2025-07-28T08:23:17Z

...src/test/resources/org/apache/flink/table/planner/plan/batch/sql/DynamicFunctionPlanTest.xml

@@ -77,7 +77,7 @@ GROUP BY cat, hh]]>
 LogicalAggregate(group=[{0, 1}], EXPR$2=[SUM($2)], EXPR$3=[COUNT()])
 +- LogicalProject(cat=[$2], hh=[$6], cnt=[$4])
   +- LogicalFilter(condition=[=(SUBSTR(CAST($5):VARCHAR(2147483647) CHARACTER SET "UTF-16LE", 1, 2), $6)])
-      +- LogicalProject(a=[$0], b=[$1], cat=[$2], gmt_date=[$3], cnt=[$4], ts=[$5], hh=[SUBSTR(CAST(LOCALTIME()):VARCHAR(2147483647) CHARACTER SET "UTF-16LE" NOT NULL, 1, 2)])
+      +- LogicalProject(a=[$0], b=[$1], cat=[$2], gmt_date=[$3], cnt=[$4], ts=[$5], hh=[SUBSTR(CAST(LOCALTIME):VARCHAR(2147483647) CHARACTER SET "UTF-16LE" NOT NULL, 1, 2)])


result of enabling allowNiladicParentheses
in FlinkSqlConformance

twalthr

LGTM. Thanks @snuyanzin!

This closes apache#26827.

snuyanzin mentioned this pull request Jul 24, 2025

[FLINK-35855][table] Upgrade Calcite version to 1.36.0 #26824

Closed

snuyanzin force-pushed the flink35855 branch from 6dfb4cc to 2869497 Compare July 24, 2025 00:43

snuyanzin added 2 commits July 24, 2025 03:14

[FLINK-35855][table] poms and NOTICE

9fb8a2e

[FLINK-35855][table] Update Calcite classes

c5ef326

snuyanzin force-pushed the flink35855 branch from 2869497 to b8f3d1e Compare July 24, 2025 01:14

snuyanzin commented Jul 24, 2025

View reviewed changes

...link-table-planner/src/main/java/org/apache/calcite/sql/util/ReflectiveSqlOperatorTable.java Outdated Show resolved Hide resolved

snuyanzin commented Jul 24, 2025

View reviewed changes

snuyanzin added 3 commits July 24, 2025 03:33

[FLINK-35855][table] Update parsers tests

7b40cc1

[FLINK-35855][table] Update Flink classes and partial insert tests

ef516f5

[FLINK-35855][table] parser.tdd

60906f9

snuyanzin force-pushed the flink35855 branch from b8f3d1e to 60906f9 Compare July 24, 2025 01:33

[FLINK-35855][table] Add more comments

64ce65d

twalthr reviewed Jul 25, 2025

View reviewed changes

snuyanzin added 2 commits July 25, 2025 16:14

Address feedback

6a68866

Address feedback

0b2b171

snuyanzin force-pushed the flink35855 branch from d00f01b to 0b2b171 Compare July 27, 2025 17:08

snuyanzin added 2 commits July 27, 2025 23:16

[FLINK-35855][table] Add test for recursive with

db7c140

[FLINK-35855][table] Do not use deprecated RexBuilder#makeCast

4835d9d

snuyanzin commented Jul 28, 2025

View reviewed changes

twalthr approved these changes Jul 28, 2025

View reviewed changes

snuyanzin added a commit to snuyanzin/flink that referenced this pull request Jul 28, 2025

[FLINK-35855][table] Do not use deprecated RexBuilder#makeCast

bd48e8c

This closes apache#26827.

snuyanzin added a commit to snuyanzin/flink that referenced this pull request Jul 28, 2025

[FLINK-35855][table] Upgrade Calcite to 1.36.0

8c2851e

This closes apache#26827.

snuyanzin added a commit to snuyanzin/flink that referenced this pull request Jul 28, 2025

[FLINK-35855][table] Do not use deprecated RexBuilder#makeCast

6774be6

This closes apache#26827.

snuyanzin added a commit to snuyanzin/flink that referenced this pull request Jul 28, 2025

[FLINK-35855][table] Do not use deprecated RexBuilder#makeCast

c9a2ef7

This closes apache#26827.

snuyanzin closed this in 06726f4 Jul 28, 2025

snuyanzin deleted the flink35855 branch July 29, 2025 07:49

[FLINK-35855][table] Upgrade Calcite to 1.36.0 #26827

[FLINK-35855][table] Upgrade Calcite to 1.36.0 #26827

Conversation

snuyanzin commented Jul 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Brief change log

Does this pull request potentially affect one of the following parts:

Documentation

Uh oh!

flinkbot commented Jul 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

CI report:

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

twalthr left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

snuyanzin Jul 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

twalthr left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

snuyanzin commented Jul 23, 2025 •

edited

Loading

flinkbot commented Jul 23, 2025 •

edited

Loading

snuyanzin Jul 27, 2025 •

edited

Loading