Support casting date literal to timestamp #3831

yuancu · 2025-06-30T11:19:55Z

Description

Before this PR, only the following casts are supported

	date	time	timestamp
date str	✅	❌	❌
time str	❌	✅	❌
timestamp str	✅	✅	✅

With this PR:

	date	time	timestamp
date str	✅	❌	✅
time str	❌	✅	❌
timestamp str	✅	✅	✅

The castings that remain impossible are in place to avoid swallowing errors silently.

An example case it solves
... | where date_time > '1950-10-11' | fields date_time

Before this PR

calcite

EnumerableCalc(expr#0=[{inputs}], expr#1=['1950-10-11':VARCHAR], expr#2=[TIMESTAMP($t1)], expr#3=[>($t0, $t2)], date_time=[$t0], $condition=[$t3])
  CalciteEnumerableIndexScan(table=[[OpenSearch, dates]], PushDownContext=[[PROJECT->[date_time]], OpenSearchRequestBuilder(sourceBuilder={"from":0,"timeout":"1m","_source":{"includes":["date_time"],"excludes":[]}}, requestedTotalSize=2147483647, pageSize=null, startFrom=0)])

Run into follow exception when executed

{
  "error": {
    "reason": "There was internal problem at backend",
    "details": "java.sql.SQLException: exception while executing query: timestamp:1950-10-11 in unsupported format, please use 'yyyy-MM-dd HH:mm:ss[.SSSSSSSSS]'",
    "type": "RuntimeException"
  },
  "status": 500
}

v2

{
 "error": {
   "reason": "Invalid Query",
   "details": "timestamp:1950-10-11 in unsupported format, please use 'yyyy-MM-dd HH:mm:ss[.SSSSSSSSS]'",
   "type": "SemanticCheckException"
 },
 "status": 400
}

After this PR

calcite

CalciteEnumerableIndexScan(table=[[OpenSearch, dates]], PushDownContext=[[PROJECT->[date_time], FILTER->>($0, '1950-10-11 00:00:00')], OpenSearchRequestBuilder(sourceBuilder={"from":0,"timeout":"1m","query":{"range":{"date_time":{"from":"1950-10-11T00:00:00.000Z","to":null,"include_lower":false,"include_upper":true,"boost":1.0}}},"_source":{"includes":["date_time"],"excludes":[]},"sort":[{"_doc":{"order":"asc"}}]}, requestedTotalSize=2147483647, pageSize=null, startFrom=0)])

v2

{
    "root": {
        "name": "ProjectOperator",
        "description": {
            "fields": "[date_time]"
        },
        "children": [
            {
                "name": "OpenSearchIndexScan",
                "description": {
                    "request": "OpenSearchQueryRequest(indexName=dates, sourceBuilder={\"from\":0,\"size\":10000,\"timeout\":\"1m\",\"query\":{\"range\":{\"date_time\":{\"from\":\"1950-10-11T00:00:00.000Z\",\"to\":null,\"include_lower\":false,\"include_upper\":true,\"boost\":1.0}}},\"_source\":{\"includes\":[\"date_time\"],\"excludes\":[]},\"sort\":[{\"_doc\":{\"order\":\"asc\"}}]}, needClean=true, searchDone=false, pitId=*, cursorKeepAlive=1m, searchAfter=null, searchResponse=null)"
                },
                "children": []
            }
        ]
    }
}

Callout: The exception thrown when failed to parse a date/time string is changed to ExpressionEvaluationException (from SemanticCheckException). Users relying on the exception type will be affected.

Implementation Notes

With Calcite, the original way was just to make literals with types like EXPR_TIMESTAMP. The values inside are not validated: https://github.com/apache/calcite/blob/a6973bde8107a7c2a605a82c779a64fe28bdb5d1/core/src/main/java/org/apache/calcite/rex/RexBuilder.java#L835
Now we explicitly call TIMESTAMP, TIME, or DATE functions to cast string values.

Concerns

Calling UDFs for casting may render it impossible to e.g. pushdown
Calling UDFs for casting may make the execution slower

Related Issues

Resolves #3728

Check List

New functionality includes testing.
New functionality has been documented.
New functionality has javadoc added.
New functionality has a user manual doc added.
API changes companion pull request created.
Commits are signed per the DCO using --signoff.
Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

date str -> timestamp timestamp str -> date timestamp str -> time Signed-off-by: Yuanchun Shen <yuanchu@amazon.com>

Signed-off-by: Yuanchun Shen <yuanchu@amazon.com>

LantaoJin · 2025-07-01T04:53:48Z

integ-test/src/test/java/org/opensearch/sql/calcite/remote/CalcitePPLCastFunctionIT.java

+        rows("New York"),
+        rows("Ontario"),
+        rows("Quebec"),
+        rows((Object) null),


Good job to support passing null in verifyDataRows

LantaoJin · 2025-07-01T04:55:42Z

integ-test/src/test/java/org/opensearch/sql/calcite/remote/CalcitePPLCastFunctionIT.java

+                        "source=%s | eval a = cast('09:07:42' as DATE) | fields a",
+                        TEST_INDEX_DATE_FORMATS)));
+
+    verifyErrorMessageContains(t, "date:09:07:42 in unsupported format, please use 'yyyy-MM-dd'");


seems we are not aligning with Spark, so this is the behviour of v2 or mysql?

This was the behavior of v2.

In MySQL, cast a time string to date will return NULL; in some other databases like Postgres, it raises an error.

Signed-off-by: Yuanchun Shen <yuanchu@amazon.com>

yuancu · 2025-07-04T07:13:33Z

Is this still a concern if expression pushdown is supported?

It should not be a issue when script pushdown is supported. Besides, if it is a literal that has to be cast, it can be pushed-down as a range query with #3798. I added an example with physical plan in the PR description to demonstrate this.

Probably we can double check and see if there’s a possible workaround in V3.

#3798 circumvents script push-down by converting the string literal to another DSL-recognizable literal then pushing down.

qianheng-aws · 2025-07-09T06:54:18Z

core/src/main/java/org/opensearch/sql/data/model/ExprTimestampValue.java

-              .toInstant();
-    } catch (DateTimeParseException e) {
+          DateTimeParser.parseDateOrTimestamp(timestamp).atZone(ZoneOffset.UTC).toInstant();
+    } catch (SemanticCheckException e) {


I think SemanticCheckException is inappropriate here. This issue isn't related to semantic at all. Should be something like IllegalArgumentException.

Changed to ExpressionEvaluationException when failed parsing datetime literals/strings.

qianheng-aws · 2025-07-09T07:24:20Z

core/src/main/java/org/opensearch/sql/data/model/ExprTimestampValue.java

-              .atZone(ZoneOffset.UTC)
-              .toInstant();
-    } catch (DateTimeParseException e) {
+          DateTimeParser.parseDateOrTimestamp(timestamp).atZone(ZoneOffset.UTC).toInstant();


Shall we support cast time str -> timestamp in the future? Maybe replace DATE_TIME_FORMATTER_VARIABLE_NANOS with DATE_TIME_FORMATTER_VARIABLE_NANOS_OPTIONAL directly if that's the case.

And I wonder if using DateTimeFormatter with multiple patterns should be more efficient than DateTimeParser.parseDateOrTimestamp which try to parseDate and fallback to parseTimeStamp when throwing exception.

Fixed by replacing parse*Or* calls with new DateTimeFormatters

qianheng-aws · 2025-07-09T07:41:08Z

Not related to this issue, I found there are some inefficient implementation for the date related functions, like branch selection at runtime for TIMESTAMP which should be determined in compiling phase based on the type of args, and redundant transformation between String and ExprStringValue for DATE. I don't check all of them.

Signed-off-by: Yuanchun Shen <yuanchu@amazon.com>

…ate/time strings Signed-off-by: Yuanchun Shen <yuanchu@amazon.com>

yuancu · 2025-07-10T07:35:31Z

One problem I discovered during testing:

Errors thrown during executing PPL with Calcite will often result in 500 error with Calcite.

For example query source=t0001 | where @timestamp > '12:00' has the following response with Calcite:

{
  "error": {
    "reason": "There was internal problem at backend",
    "details": "java.sql.SQLException: exception while executing query: timestamp:12:00 in unsupported format, please use 'yyyy-MM-dd HH:mm:ss[.SSSSSSSSS]'",
    "type": "RuntimeException"
  },
  "status": 500
}

But in v2, it's:

{
  "error": {
    "reason": "Invalid Query",
    "details": "timestamp:12:00 in unsupported format, please use 'yyyy-MM-dd HH:mm:ss[.SSSSSSSSS]'",
    "type": "ExpressionEvaluationException"
  },
  "status": 400
}

This is because exceptions thrown during statement.executeQuery() will be wrapped with SQLException (see AvaticaConnection.java#L579), which is then captured by us and wrapped with a RuntimeException:

  try (PreparedStatement statement = OpenSearchRelRunners.run(context, rel)) {
    ResultSet result = statement.executeQuery();
    buildResultSet(result, rel.getRowType(), context.querySizeLimit, listener);
  } catch (SQLException e) {
    throw new RuntimeException(e);
  }

A simple fix: retrieving the wrapped exception from the SQLException and re-throw it. Should I implement this?

example stacktrace

[2025-07-10T14:28:48,782][ERROR][o.o.s.p.r.RestPPLQueryAction] [7cf34de73b85] Error happened during query handling
java.lang.RuntimeException: java.sql.SQLException: exception while executing query: timestamp:12:00 in unsupported format, please use 'yyyy-MM-dd HH:mm:ss[.SSSSSSSSS]'
	at org.opensearch.sql.opensearch.executor.OpenSearchExecutionEngine.lambda$execute$6(OpenSearchExecutionEngine.java:203) ~[?:?]
	at java.base/java.security.AccessController.doPrivileged(AccessController.java:319) ~[?:?]
	at org.opensearch.sql.opensearch.executor.OpenSearchExecutionEngine.lambda$execute$7(OpenSearchExecutionEngine.java:196) ~[?:?]
	at org.opensearch.sql.opensearch.client.OpenSearchNodeClient.schedule(OpenSearchNodeClient.java:193) ~[?:?]
	at org.opensearch.sql.opensearch.executor.OpenSearchExecutionEngine.execute(OpenSearchExecutionEngine.java:194) ~[?:?]
	at org.opensearch.sql.executor.QueryService.lambda$executeWithCalcite$0(QueryService.java:107) ~[?:?]
	at java.base/java.security.AccessController.doPrivileged(AccessController.java:319) ~[?:?]
	at org.opensearch.sql.executor.QueryService.executeWithCalcite(QueryService.java:96) ~[?:?]
	at org.opensearch.sql.executor.QueryService.execute(QueryService.java:72) ~[?:?]
	at org.opensearch.sql.executor.execution.QueryPlan.execute(QueryPlan.java:69) ~[?:?]
	at org.opensearch.sql.opensearch.executor.OpenSearchQueryManager.lambda$submit$0(OpenSearchQueryManager.java:31) ~[?:?]
	at org.opensearch.sql.opensearch.executor.OpenSearchQueryManager.lambda$withCurrentContext$1(OpenSearchQueryManager.java:45) ~[?:?]
	at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:916) ~[opensearch-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
	at java.base/java.lang.Thread.run(Thread.java:1575) [?:?]
Caused by: java.sql.SQLException: exception while executing query: timestamp:12:00 in unsupported format, please use 'yyyy-MM-dd HH:mm:ss[.SSSSSSSSS]'
	at org.apache.calcite.avatica.Helper.createException(Helper.java:56) ~[?:?]
	at org.apache.calcite.avatica.Helper.createException(Helper.java:41) ~[?:?]
	at org.apache.calcite.avatica.AvaticaConnection.executeQueryInternal(AvaticaConnection.java:579) ~[?:?]
	at org.apache.calcite.avatica.AvaticaPreparedStatement.executeQuery(AvaticaPreparedStatement.java:137) ~[?:?]
	at org.opensearch.sql.opensearch.executor.OpenSearchExecutionEngine.lambda$execute$6(OpenSearchExecutionEngine.java:200) ~[?:?]
	... 15 more
Caused by: org.opensearch.sql.exception.ExpressionEvaluationException: timestamp:12:00 in unsupported format, please use 'yyyy-MM-dd HH:mm:ss[.SSSSSSSSS]'
	at org.opensearch.sql.data.model.ExprTimestampValue.<init>(ExprTimestampValue.java:43) ~[?:?]
	at org.opensearch.sql.calcite.utils.datetime.DateTimeConversionUtils.convertToTimestampValue(DateTimeConversionUtils.java:62) ~[?:?]
	at org.opensearch.sql.expression.function.udf.datetime.TimestampFunction.timestamp(TimestampFunction.java:75) ~[?:?]
	at Baz$1$1.moveNext(Unknown Source) ~[?:?]
	at org.apache.calcite.linq4j.Linq4j$EnumeratorIterator.<init>(Linq4j.java:666) ~[?:?]
	at org.apache.calcite.linq4j.Linq4j.enumeratorIterator(Linq4j.java:99) ~[?:?]
	at org.apache.calcite.linq4j.AbstractEnumerable.iterator(AbstractEnumerable.java:33) ~[?:?]
	at org.apache.calcite.avatica.MetaImpl.createCursor(MetaImpl.java:83) ~[?:?]
	at org.apache.calcite.avatica.AvaticaResultSet.execute(AvaticaResultSet.java:186) ~[?:?]
	at org.apache.calcite.jdbc.CalciteResultSet.execute(CalciteResultSet.java:64) ~[?:?]
	at org.apache.calcite.jdbc.CalciteResultSet.execute(CalciteResultSet.java:43) ~[?:?]
	at org.apache.calcite.avatica.AvaticaConnection.executeQueryInternal(AvaticaConnection.java:575) ~[?:?]
	at org.apache.calcite.avatica.AvaticaPreparedStatement.executeQuery(AvaticaPreparedStatement.java:137) ~[?:?]
	at org.opensearch.sql.opensearch.executor.OpenSearchExecutionEngine.lambda$execute$6(OpenSearchExecutionEngine.java:200) ~[?:?]
	... 15 more

yuancu · 2025-07-10T07:47:59Z

Not related to this issue, I found there are some inefficient implementation for the date related functions, like branch selection at runtime for TIMESTAMP which should be determined in compiling phase based on the type of args, and redundant transformation between String and ExprStringValue for DATE. I don't check all of them.

I spotted a few for functions under java/org/opensearch/sql/expression/function/udf/datetime. I can fix them in another new PR.

Besides, almost all datetime functions convert inputs to ExprValues to reuse v2's implementation. I can eliminate them to enhance efficiency if the anticipated improvement is significant.

yuancu · 2025-07-29T08:22:32Z

Should this PR be backported? @LantaoJin

opensearch-trigger-bot · 2025-07-29T09:49:12Z

The backport to 2.19-dev failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/sql/backport-2.19-dev 2.19-dev
# Navigate to the new working tree
pushd ../.worktrees/sql/backport-2.19-dev
# Create a new branch
git switch --create backport/backport-3831-to-2.19-dev
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 30aba6588291f4d3f88ae21f98788608da261667
# Push it to GitHub
git push --set-upstream origin backport/backport-3831-to-2.19-dev
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/sql/backport-2.19-dev

Then, create a pull request where the base branch is 2.19-dev and the compare/head branch is backport/backport-3831-to-2.19-dev.

LantaoJin · 2025-07-29T10:00:51Z

Should this PR be backported? @LantaoJin

Yes, please.

* Support following datetime casts in v2: date str -> timestamp timestamp str -> date timestamp str -> time Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Invoke datetime UDF to cast date/time/timestamp with Calcite Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Test cast to date/time/timestamp Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Use stricter date and time formatters Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Add a timestamp formatter for instantiating ExprTimestampValue Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Simplify date/time expr constructors & parsers Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Change to ExpressionEvaluationException when fail parsing malformat date/time strings Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> --------- Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> (cherry picked from commit 30aba65)

* Support following datetime casts in v2: date str -> timestamp timestamp str -> date timestamp str -> time * Invoke datetime UDF to cast date/time/timestamp with Calcite * Test cast to date/time/timestamp * Use stricter date and time formatters * Add a timestamp formatter for instantiating ExprTimestampValue * Simplify date/time expr constructors & parsers * Change to ExpressionEvaluationException when fail parsing malformat date/time strings --------- (cherry picked from commit 30aba65) Signed-off-by: Yuanchun Shen <yuanchu@amazon.com>

yuancu added 3 commits June 30, 2025 17:59

Support following datetime casts in v2:

7a120db

date str -> timestamp timestamp str -> date timestamp str -> time Signed-off-by: Yuanchun Shen <yuanchu@amazon.com>

Invoke datetime UDF to cast date/time/timestamp with Calcite

0e98650

Signed-off-by: Yuanchun Shen <yuanchu@amazon.com>

Test cast to date/time/timestamp

bc2c9e2

Signed-off-by: Yuanchun Shen <yuanchu@amazon.com>

yuancu requested review from GumpacG, LantaoJin, MaxKsyunz, Swiddis, YANG-DB, Yury-Fridlyand, acarbonetto, anirudha, dai-chen, derek-ho, forestmvey, joshuali925, kavithacm, mengweieric, noCharger, penghuo, ps48, qianheng-aws, seankao-az and ykmr1224 as code owners June 30, 2025 11:19

yuancu marked this pull request as draft June 30, 2025 11:27

Use stricter date and time formatters

3c42fe1

Signed-off-by: Yuanchun Shen <yuanchu@amazon.com>

yuancu changed the title ~~Support casting date string to timestamp~~ [BUG] Support casting date string to timestamp Jul 1, 2025

yuancu changed the title ~~[BUG] Support casting date string to timestamp~~ Support casting date string to timestamp Jul 1, 2025

LantaoJin added bug Something isn't working calcite calcite migration releated labels Jul 1, 2025

LantaoJin reviewed Jul 1, 2025

View reviewed changes

Merge remote-tracking branch 'origin/main' into cast-datetime

28d2461

Signed-off-by: Yuanchun Shen <yuanchu@amazon.com>

qianheng-aws reviewed Jul 9, 2025

View reviewed changes

LantaoJin mentioned this pull request Jul 10, 2025

Add ClickBench IT Suite #3860

Merged

7 tasks

yuancu added 2 commits July 10, 2025 14:02

Simplify date/time expr constructors & parsers

f8f01f4

Signed-off-by: Yuanchun Shen <yuanchu@amazon.com>

Change to ExpressionEvaluationException when fail parsing malformat d…

bd9cea2

…ate/time strings Signed-off-by: Yuanchun Shen <yuanchu@amazon.com>

yuancu requested review from LantaoJin and qianheng-aws July 10, 2025 08:36

qianheng-aws approved these changes Jul 15, 2025

View reviewed changes

LantaoJin approved these changes Jul 22, 2025

View reviewed changes

LantaoJin merged commit 30aba65 into opensearch-project:main Jul 22, 2025
24 checks passed

LantaoJin added the backport 2.19-dev label Jul 29, 2025

opensearch-trigger-bot bot added the backport-failed label Jul 29, 2025

yuancu mentioned this pull request Jul 30, 2025

[Backport 2.19-dev] Support casting date literal to timestamp (#3831) #3939

Merged

7 tasks

LantaoJin added the backport-manually Filed a PR to backport manually. label Jul 31, 2025

opensearch-project deleted a comment from opensearch-trigger-bot bot Aug 13, 2025

LantaoJin mentioned this pull request Aug 15, 2025

[GitHub Request] Add yuancu as maintainer of OpenSearch SQL opensearch-project/.github#374

Closed

Support casting date literal to timestamp #3831

Support casting date literal to timestamp #3831

Uh oh!

Conversation

yuancu commented Jun 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Related Issues

Check List

Uh oh!

LantaoJin Jul 1, 2025

Choose a reason for hiding this comment

Uh oh!

LantaoJin Jul 1, 2025

Choose a reason for hiding this comment

Uh oh!

yuancu Jul 1, 2025

Choose a reason for hiding this comment

Uh oh!

yuancu commented Jul 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

qianheng-aws Jul 9, 2025

Choose a reason for hiding this comment

Uh oh!

yuancu Jul 10, 2025

Choose a reason for hiding this comment

Uh oh!

qianheng-aws Jul 9, 2025

Choose a reason for hiding this comment

Uh oh!

yuancu Jul 10, 2025

Choose a reason for hiding this comment

Uh oh!

qianheng-aws commented Jul 9, 2025

Uh oh!

yuancu commented Jul 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

yuancu commented Jul 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

yuancu commented Jul 29, 2025

Uh oh!

opensearch-trigger-bot bot commented Jul 29, 2025

Uh oh!

LantaoJin commented Jul 29, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

yuancu commented Jun 30, 2025 •

edited

Loading

yuancu commented Jul 4, 2025 •

edited

Loading

yuancu commented Jul 10, 2025 •

edited

Loading

yuancu commented Jul 10, 2025 •

edited

Loading