Update `DATE_ADD`/`ADDDATE` and `DATE_SUB`/`SUBDATE` functions. (#122) #1182

Yury-Fridlyand · 2022-12-16T02:50:28Z

Signed-off-by: Yury-Fridlyand yuryf@bitquilltech.com

Description

I referred to MySQL docs and tried to reproduce MySQL v.8.0.30 behavior as a reference.

See team review and discussion in Bit-Quill#122

Rework on `DATE_ADD`/`ADDDATE` functions.

Changes

Fixes:

Function accepts TIME, considering today's date. If argument is DATE, it interpreted as midnight of this DATE.
Function returns always DATETIME.
Function can interact with any combination of argument types and date and time intervals both.

Future changes (TODOs):

Rework after [FEATURE] Add 3 interval types instead of 1 to support complex interval expressions #859
Accept and return STRING

Removed function `DATE_ADD(date, int)`

Left function (matches MySQL):

ADDDATE(date, interval)
DATE_ADD(date, interval)
ADDDATE(date, int)

Update signature list

Added TIME
Removed STRING - all datetime types have implicit cast from a string already

More details

Function Before After

DATE_ADD

(STRING/DATE/DATETIME/TIMESTAMP, INTERVAL) -> DATETIME

(DATE/DATETIME/TIMESTAMP/TIME, INTERVAL) -> DATETIME

ADDDATE

(DATE, LONG) -> DATE
(STRING/DATETIME/TIMESTAMP, LONG) -> DATETIME

(DATE, LONG) -> DATE
(TIME/DATETIME/TIMESTAMP, LONG) -> DATETIME

Rework on `DATE_SUB`/`SUBDATE` functions.

Changes

Removed function `DATE_SUB(date, int)`

The same fixes and notes as for `DATE_ADD`/`ADDDATE` functions.

Test queries:

NB: Due to #853 you can't test with DATETIME (without code changes), but I tested with it

OpenSearch

SELECT date0, DATE_ADD(CAST(date0 AS date), INTERVAL 1 HOUR) AS `date + 1h`,
       time0, DATE_ADD(CAST(time0 AS datetime), INTERVAL 1 HOUR) AS `datetime + 1h`,
	   time1, DATE_ADD(CAST(time1 AS time), INTERVAL 1 HOUR) AS `time + 1h`,
   datetime0, DATE_ADD(CAST(datetime0 AS timestamp), INTERVAL 1 HOUR) AS `timestamp + 1h` FROM calcs LIMIT 6;
   
SELECT date0, DATE_ADD(CAST(date0 AS date), INTERVAL 1 DAY) AS `date + 1d`,
       time0, DATE_ADD(CAST(time0 AS datetime), INTERVAL 1 DAY) AS `datetime + 1d`,
	   time1, DATE_ADD(CAST(time1 AS time), INTERVAL 1 DAY) AS `time + 1d`,
   datetime0, DATE_ADD(CAST(datetime0 AS timestamp), INTERVAL 1 DAY) AS `timestamp + 1d` FROM calcs LIMIT 6;
   
SELECT date0, DATE_ADD(CAST(date0 AS date), INTERVAL 1 HOUR) AS `date + 1h`, DATE_ADD(CAST(date0 AS date), INTERVAL 1 DAY) AS `date + 1d` FROM calcs LIMIT 6;
SELECT time0, DATE_ADD(CAST(time0 AS datetime), INTERVAL 1 HOUR) AS `datetime + 1h`, DATE_ADD(CAST(time0 AS datetime), INTERVAL 1 DAY) AS `datetime + 1d` FROM calcs LIMIT 6;
SELECT time1, DATE_ADD(CAST(time1 AS time), INTERVAL 24 HOUR) AS `time + 24h` FROM calcs LIMIT 6;
SELECT datetime0, DATE_ADD(CAST(datetime0 AS timestamp), INTERVAL 1 HOUR) AS `timestamp + 1h`, DATE_ADD(CAST(datetime0 AS timestamp), INTERVAL 1 DAY) AS `timestamp + 1d` FROM calcs LIMIT 6;

SELECT time1, CAST(time1 AS time), ADDDATE(CAST(time1 AS time), 1) FROM calcs LIMIT 6;
SELECT date0, CAST(date0 AS date), ADDDATE(CAST(date0 AS date), 1) FROM calcs LIMIT 6;
SELECT time0, CAST(time0 AS datetime), ADDDATE(CAST(time0 AS datetime), 1) FROM calcs LIMIT 6;
SELECT datetime0, CAST(datetime0 AS timestamp), ADDDATE(CAST(datetime0 AS timestamp), 1) FROM calcs LIMIT 6;

MySQL

SELECT date0, DATE_ADD(date0, INTERVAL 1 HOUR) AS `date + 1h`, time0, DATE_ADD(time0, INTERVAL 1 HOUR) AS `datetime + 1h`, time1, DATE_ADD(time1, INTERVAL 1 HOUR) AS `time + 1h`, datetime0, DATE_ADD(datetime0, INTERVAL 1 HOUR) AS `timestamp + 1h` FROM calcs LIMIT 6;
SELECT date0, DATE_ADD(date0, INTERVAL 1 DAY) AS `date + 1d`, time0, DATE_ADD(time0, INTERVAL 1 DAY) AS `datetime + 1d`, time1, DATE_ADD(time1, INTERVAL 1 DAY) AS `time + 1d`, datetime0, DATE_ADD(datetime0, INTERVAL 1 DAY) AS `timestamp + 1d` FROM calcs LIMIT 6;

SELECT time1, DATE_ADD(time1, INTERVAL 20 HOUR) AS `time + 20h` FROM calcs LIMIT 6;
SELECT date0, DATE_ADD(date0, INTERVAL 1 HOUR) AS `date + 1h`, DATE_ADD(date0, INTERVAL 1 DAY) AS `date + 1d` FROM calcs LIMIT 6;
SELECT time0, DATE_ADD(time0, INTERVAL 1 HOUR) AS `datetime + 1h`, DATE_ADD(time0, INTERVAL 1 DAY) AS `datetime + 1d` FROM calcs LIMIT 6;
SELECT datetime0, DATE_ADD(datetime0, INTERVAL 1 HOUR) AS `timestamp + 1h`, DATE_ADD(datetime0, INTERVAL 1 DAY) AS `timestamp + 1d` FROM calcs LIMIT 6;

SELECT time1, ADDDATE(time1, 1) FROM calcs LIMIT 6;
SELECT date0, ADDDATE(date0, 1) FROM calcs LIMIT 6;
SELECT time0, ADDDATE(time0, 1) FROM calcs LIMIT 6;
SELECT datetime0, ADDDATE(datetime0, 1) FROM calcs LIMIT 6;

PostgreSQL

SELECT time1, pg_typeof(time1), "time + 5h", pg_typeof("time + 5h"), "time + 1d", pg_typeof("time + 1d") FROM (SELECT time1, time1 + INTERVAL '5 HOURS' AS "time + 5h", time1 + INTERVAL '1 DAYS' AS "time + 1d" FROM calcs LIMIT 6) foo;

SELECT date0, pg_typeof(date0), "date + 5h", pg_typeof("date + 5h"), "date + 1d", pg_typeof("date + 1d") FROM (SELECT date0, date0 + INTERVAL '5 HOURS' AS "date + 5h", date0 + INTERVAL '1 DAYS' AS "date + 1d" FROM calcs LIMIT 6) foo;

Test data

I found that first 6 rows from date0, time0, time1, datetime0 are good for testing - these columns have different data types in MySQL. In OpenSearch SQL all [date][time] columns have timestamp type, so I use CAST for clear testing.

data

mysql> show fields from Calcs where field IN ('date0', 'time0', 'time1', 'datetime0');
+-----------+-----------+------+-----+---------+-------+
| Field     | Type      | Null | Key | Default | Extra |
+-----------+-----------+------+-----+---------+-------+
| date0     | date      | YES  |     | NULL    |       |
| time0     | datetime  | YES  |     | NULL    |       |
| time1     | time      | YES  |     | NULL    |       |
| datetime0 | timestamp | YES  |     | NULL    |       |
+-----------+-----------+------+-----+---------+-------+
4 rows in set (0.00 sec)
mysql> select date0, time0, time1, datetime0 from calcs;
+------------+---------------------+----------+---------------------+
| date0      | time0               | time1    | datetime0           |
+------------+---------------------+----------+---------------------+
| 2004-04-15 | 1899-12-30 21:07:32 | 19:36:22 | 2004-07-09 10:17:35 |
| 1972-07-04 | 1900-01-01 13:48:48 | 02:05:25 | 2004-07-26 12:30:34 |
| 1975-11-12 | 1900-01-01 18:21:08 | 09:33:31 | 2004-08-02 07:59:23 |
| 2004-06-04 | 1900-01-01 18:51:48 | 22:50:16 | 2004-07-05 13:14:20 |
| 2004-06-19 | 1900-01-01 15:01:19 | NULL     | 2004-07-28 23:30:22 |
| NULL       | 1900-01-01 08:59:39 | 19:57:33 | 2004-07-22 00:30:23 |
| NULL       | 1900-01-01 07:37:48 | NULL     | 2004-07-28 06:54:50 |
| NULL       | 1900-01-01 19:45:54 | 19:48:23 | 2004-07-12 17:30:16 |
| NULL       | 1900-01-01 09:00:59 | 22:20:14 | 2004-07-04 22:49:28 |
| NULL       | 1900-01-01 20:36:00 | NULL     | 2004-07-23 21:13:37 |
| NULL       | 1900-01-01 01:31:32 | 00:05:57 | 2004-07-14 08:16:44 |
| NULL       | 1899-12-30 22:15:40 | 04:40:49 | 2004-07-25 15:22:26 |
| NULL       | 1900-01-01 13:53:46 | 04:48:07 | 2004-07-17 14:01:56 |
| NULL       | 1900-01-01 04:57:51 | NULL     | 2004-07-19 22:21:31 |
| NULL       | 1899-12-30 22:42:43 | 18:58:41 | 2004-07-31 11:57:52 |
| NULL       | 1899-12-30 22:24:08 | NULL     | 2004-07-14 07:43:00 |
| NULL       | 1900-01-01 11:58:29 | 12:33:57 | 2004-07-28 12:34:28 |
+------------+---------------------+----------+---------------------+
17 rows in set (0.00 sec)

Related issue:

Fixes #291
https://forum.opensearch.org/t/subdate-date-sub-query-method-not-supported/9252/2

Check List

New functionality includes testing.
- All tests pass, including unit test, integration test and doctest
New functionality has been documented.
- New functionality has javadoc added
- New functionality has user manual doc added
Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

codecov-commenter · 2022-12-16T03:20:30Z

Codecov Report

Merging #1182 (fec8f3a) into main (1108379) will decrease coverage by 2.44%.
The diff coverage is 100.00%.

@@             Coverage Diff              @@
##               main    #1182      +/-   ##
============================================
- Coverage     98.35%   95.91%   -2.45%     
- Complexity     3604     3607       +3     
============================================
  Files           344      354      +10     
  Lines          8933     9592     +659     
  Branches        562      681     +119     
============================================
+ Hits           8786     9200     +414     
- Misses          142      334     +192     
- Partials          5       58      +53

Flag	Coverage Δ
query-workbench	`62.76% <ø> (?)`
sql-engine	`98.35% <100.00%> (+<0.01%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
...c/main/java/org/opensearch/sql/expression/DSL.java	`100.00% <ø> (ø)`
.../org/opensearch/sql/data/model/ExprValueUtils.java	`100.00% <100.00%> (ø)`
...arch/sql/expression/datetime/DateTimeFunction.java	`100.00% <100.00%> (ø)`
...search/sql/expression/datetime/IntervalClause.java	`100.00% <100.00%> (ø)`
...ublic/components/QueryResults/QueryResultsBody.tsx	`68.32% <0.00%> (ø)`
workbench/public/utils/PanelWrapper.tsx	`100.00% <0.00%> (ø)`
workbench/public/components/Header/Header.tsx	`100.00% <0.00%> (ø)`
workbench/public/components/PPLPage/PPLPage.tsx	`56.52% <0.00%> (ø)`
workbench/public/application.tsx	`0.00% <0.00%> (ø)`
workbench/public/components/app.tsx	`0.00% <0.00%> (ø)`
... and 4 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

core/src/main/java/org/opensearch/sql/data/model/ExprTimestampValue.java

penghuo · 2023-01-04T19:44:11Z

core/src/main/java/org/opensearch/sql/expression/datetime/DateTimeFunction.java

+   * (DATE, LONG) -> DATE
+   * (TIME/DATETIME/TIMESTAMP, LONG) -> DATETIME
+   */
+  private Stream<SerializableFunction<?, ?>> get_adddate_subdate_signatures(


seems cratedb normalized to arthemetic operation, https://github.com/crate/crate/blob/5.0/server/src/main/java/io/crate/expression/scalar/arithmetic/ArithmeticFunctions.java
could we do something similar?

penghuo · 2023-01-04T19:44:59Z

core/src/main/java/org/opensearch/sql/expression/datetime/DateTimeFunction.java

-        impl(nullMissingHandling(DateTimeFunction::exprAddDateInterval),
+  private Stream<SerializableFunction<?, ?>> get_date_add_date_sub_signatures(
+      SerializableTriFunction<FunctionProperties, ExprValue, ExprValue, ExprValue> function) {
+    return Stream.of(


Does Datetime required? I found you post same concern in #1176.

Unfortunately, yes, unless we have 4 types for date/time.

* Rework on `DATE_ADD`/`ADDDATE` and `DATE_SUB`/`SUBDATE` functions. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

Yury-Fridlyand · 2023-01-05T18:02:58Z

Rebased to resolve conflicts

…ulations

Yury-Fridlyand · 2023-01-07T02:23:47Z

Resolved merge conflicts.

Yury-Fridlyand · 2023-01-09T19:52:01Z

... and again.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

…ulations Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

#1182) * Update `DATE_ADD`/`ADDDATE` and `DATE_SUB`/`SUBDATE` functions. (#122) Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> (cherry picked from commit af188a3)

#1182) (#1325) * Update `DATE_ADD`/`ADDDATE` and `DATE_SUB`/`SUBDATE` functions. (#122) Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> (cherry picked from commit af188a3) Co-authored-by: Yury-Fridlyand <yury.fridlyand@improving.com>

Yury-Fridlyand requested review from penghuo and dai-chen December 16, 2022 02:50

Yury-Fridlyand requested a review from a team as a code owner December 16, 2022 02:50

Yury-Fridlyand mentioned this pull request Dec 17, 2022

SQL ingestion and ops - standup #791

Open

dai-chen reviewed Dec 19, 2022

View reviewed changes

core/src/main/java/org/opensearch/sql/data/model/ExprTimestampValue.java Outdated Show resolved Hide resolved

Yury-Fridlyand mentioned this pull request Dec 19, 2022

[FEATURE] Support date and time functions #722

Closed

87 tasks

dai-chen added enhancement New feature or request backport 2.x labels Dec 23, 2022

penghuo reviewed Jan 4, 2023

View reviewed changes

Update DATE_ADD/ADDDATE and DATE_SUB/SUBDATE functions. (#122)

99c3708

* Rework on `DATE_ADD`/`ADDDATE` and `DATE_SUB`/`SUBDATE` functions. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

Yury-Fridlyand force-pushed the integ-datetime-calculations branch from 679bf19 to 99c3708 Compare January 5, 2023 18:02

penghuo previously approved these changes Jan 5, 2023

View reviewed changes

Merge branch 'main' into integ-datetime-calculations

e6c1cfb

Yury-Fridlyand dismissed penghuo’s stale review via e6c1cfb January 6, 2023 18:55

Merge remote-tracking branch 'upstream/main' into integ-datetime-calc…

3b98dc1

…ulations

penghuo previously approved these changes Jan 9, 2023

View reviewed changes

Merge branch 'main' into integ-datetime-calculations

977355d

Yury-Fridlyand dismissed penghuo’s stale review via 977355d January 9, 2023 19:51

Typo fix in previous commit.

e8fb6bc

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

penghuo previously approved these changes Jan 9, 2023

View reviewed changes

Merge remote-tracking branch 'upstream/main' into integ-datetime-calc…

fec8f3a

…ulations Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

Yury-Fridlyand dismissed penghuo’s stale review via fec8f3a January 10, 2023 03:28

dai-chen requested a review from penghuo January 25, 2023 19:58

dai-chen approved these changes Jan 25, 2023

View reviewed changes

penghuo approved these changes Feb 2, 2023

View reviewed changes

Yury-Fridlyand merged commit af188a3 into opensearch-project:main Feb 4, 2023

Yury-Fridlyand deleted the integ-datetime-calculations branch February 4, 2023 02:26

opensearch-trigger-bot bot mentioned this pull request Feb 4, 2023

[Backport 2.x] Update DATE_ADD/ADDDATE and DATE_SUB/SUBDATE functions. (#122) #1325

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update `DATE_ADD`/`ADDDATE` and `DATE_SUB`/`SUBDATE` functions. (#122) #1182

Update `DATE_ADD`/`ADDDATE` and `DATE_SUB`/`SUBDATE` functions. (#122) #1182

Yury-Fridlyand commented Dec 16, 2022

codecov-commenter commented Dec 16, 2022 •

edited

Loading

penghuo Jan 4, 2023

penghuo Jan 4, 2023

Yury-Fridlyand Jan 5, 2023

Yury-Fridlyand commented Jan 5, 2023

Yury-Fridlyand commented Jan 7, 2023

Yury-Fridlyand commented Jan 9, 2023

Update DATE_ADD/ADDDATE and DATE_SUB/SUBDATE functions. (#122) #1182

Update DATE_ADD/ADDDATE and DATE_SUB/SUBDATE functions. (#122) #1182

Conversation

Yury-Fridlyand commented Dec 16, 2022

Description

Rework on DATE_ADD/ADDDATE functions.

Fixes:

Future changes (TODOs):

Removed function DATE_ADD(date, int)

Update signature list

Rework on DATE_SUB/SUBDATE functions.

Removed function DATE_SUB(date, int)

The same fixes and notes as for DATE_ADD/ADDDATE functions.

Test queries:

Test data

Related issue:

Check List

codecov-commenter commented Dec 16, 2022 • edited Loading

Codecov Report

penghuo Jan 4, 2023

Choose a reason for hiding this comment

penghuo Jan 4, 2023

Choose a reason for hiding this comment

Yury-Fridlyand Jan 5, 2023

Choose a reason for hiding this comment

Yury-Fridlyand commented Jan 5, 2023

Yury-Fridlyand commented Jan 7, 2023

Yury-Fridlyand commented Jan 9, 2023

Update `DATE_ADD`/`ADDDATE` and `DATE_SUB`/`SUBDATE` functions. (#122) #1182

Update `DATE_ADD`/`ADDDATE` and `DATE_SUB`/`SUBDATE` functions. (#122) #1182

Rework on `DATE_ADD`/`ADDDATE` functions.

Removed function `DATE_ADD(date, int)`

Rework on `DATE_SUB`/`SUBDATE` functions.

Removed function `DATE_SUB(date, int)`

The same fixes and notes as for `DATE_ADD`/`ADDDATE` functions.

codecov-commenter commented Dec 16, 2022 •

edited

Loading