Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support Alternate Date Formats #258

Closed
wants to merge 55 commits into from
Closed

Support Alternate Date Formats #258

Show file tree
Hide file tree
Changes from 38 commits
Commits
Show all changes
55 commits
Select commit Hold shift + click to select a range
8b2f65d
Add OpenSearchDateType as a datatype for matching with Date/Time Open…
acarbonetto Mar 29, 2023
c5ad8e0
Added IT Test
GabeFernandez310 Mar 30, 2023
a64b93e
Added Basic Implementation
GabeFernandez310 Apr 4, 2023
cbf3c38
Added A Comment
GabeFernandez310 Apr 4, 2023
a780bbd
Added Support For Default OpenSearch Formats
GabeFernandez310 Apr 5, 2023
ffede70
Code Cleanup
GabeFernandez310 Apr 5, 2023
3708a6f
Refactored 'parsetimestamp' Function
GabeFernandez310 Apr 6, 2023
bc15e3a
Cleanup
GabeFernandez310 Apr 9, 2023
a5fac5c
Removed Unnecessary Try-Catch Block
GabeFernandez310 Apr 9, 2023
9c2a7c8
Fixed Unit Tests
GabeFernandez310 Apr 13, 2023
0f01283
Fixed IT Test
GabeFernandez310 Apr 13, 2023
5c63d7f
Fixed Text Type and Disabled JSON Format Test
GabeFernandez310 Apr 17, 2023
675aa02
Fix Mapping Type Reporting As Date Instead Of Timestamp In Schema
GabeFernandez310 Apr 18, 2023
09d420b
Revert Breaking Test Changes
GabeFernandez310 Apr 18, 2023
a519501
Fixed Returned ExprCoreTypes
GabeFernandez310 Apr 18, 2023
ed2bcbb
Fixed OpenSearch Type Names In Mapping
GabeFernandez310 Apr 18, 2023
ef90ad6
Fixed LegacyTypeName Function And Refactored Code To Fix Calculations…
GabeFernandez310 Apr 18, 2023
418dc98
Changed Support For Passing Fields Into Text Type
GabeFernandez310 Apr 18, 2023
0670812
Reverted Some Changes And Commented Out A Test
GabeFernandez310 Apr 18, 2023
5543fd8
Commented Out A Breaking Change
GabeFernandez310 Apr 19, 2023
9325d24
Minor Refactoring Changes
GabeFernandez310 Apr 19, 2023
89cb4ce
Reworked parseTimestamp Function To Format Return Based On Enum
GabeFernandez310 Apr 19, 2023
2ab67ed
Fixed Returned ExprValue
GabeFernandez310 Apr 19, 2023
bbc6526
Reverted Changes To Test Files
GabeFernandez310 Apr 19, 2023
e07b4ca
Reverted Some Changes To Enum And Reworked Code
GabeFernandez310 Apr 19, 2023
dc5ef5b
Added Type Mapping To Allow Query To Be Parsed
GabeFernandez310 Apr 19, 2023
2f45ced
Fixed Some Unit Tests
GabeFernandez310 Apr 19, 2023
d1057e3
Fixed TypeActionMap To Still Support ExprCoreTypes
GabeFernandez310 Apr 20, 2023
225e8a9
Wrapped Some Function Returns in ExprValueFactory
GabeFernandez310 Apr 20, 2023
a7801f4
Fixed Integration Test Bug
GabeFernandez310 Apr 23, 2023
25e9f11
Slight Refactoring And Code Cleanup
GabeFernandez310 Apr 24, 2023
ebbcad2
Removed Unneeded Class Member
GabeFernandez310 Apr 24, 2023
dabf8bc
Removed Unneccessary MappingType Parameters
GabeFernandez310 Apr 24, 2023
be06b94
Refactored Code To Fix Integration Test
GabeFernandez310 Apr 24, 2023
5b879de
Merge branch 'integ-support-date-formats' into dev-support-date-formats
GabeFernandez310 Apr 24, 2023
282cc76
Merge branch 'dev-support-date-formats' of github.com:Bit-Quill/opens…
GabeFernandez310 Apr 24, 2023
0fe9ea0
Addressed JaCoCo Issues
GabeFernandez310 Apr 25, 2023
debda6a
Fixed Checkstyle
GabeFernandez310 Apr 25, 2023
568bbf3
Merge branch 'dev-fix-build' of github.com:Bit-Quill/opensearch-proje…
GabeFernandez310 Apr 25, 2023
0fba818
Addressed PR Comments
GabeFernandez310 Apr 25, 2023
6f7e633
Changed To Throw Exception When Encountering Parsing Failures
GabeFernandez310 Apr 26, 2023
a6341ac
Merge branch 'dev-fix-build' of github.com:Bit-Quill/opensearch-proje…
GabeFernandez310 Apr 26, 2023
7fa682e
Fixed Checkstyle
GabeFernandez310 Apr 26, 2023
752a718
Merge branch 'integ-support-date-formats' of github.com:Bit-Quill/ope…
GabeFernandez310 Apr 27, 2023
ace81ea
Removed Unneeded Imports
GabeFernandez310 Apr 27, 2023
2ccebd9
Re-enabled Some Tests
GabeFernandez310 Apr 27, 2023
3fc11a4
Keep up with refactoring in OpenSearch. (#1588)
Apr 27, 2023
93f212a
Merge branch 'main' of https://github.com/opensearch-project/sql into…
GumpacG Apr 28, 2023
a3de4e5
Merge branch 'opensearch-project:main' into dev-support-date-formats
GabeFernandez310 Apr 28, 2023
0df21dd
Addressed PR Comments
GabeFernandez310 Apr 28, 2023
28c1bca
Merge branch 'dev-support-date-formats' of github.com:Bit-Quill/opens…
GumpacG Apr 28, 2023
b140170
Changed test data
GumpacG May 1, 2023
e95fdde
Pulled out support for user-defined datetime formats
GumpacG May 3, 2023
4394315
Modified IT test data
GumpacG May 3, 2023
4e9bb15
Added IT tests
GumpacG May 4, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions integ-test/src/test/java/org/opensearch/sql/legacy/PrettyFormatResponseIT.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,7 @@
import org.json.JSONObject;
import org.junit.Ignore;
import org.junit.Test;
import org.junit.jupiter.api.Disabled;
import org.opensearch.client.Request;

/**
Expand Down Expand Up @@ -483,6 +484,7 @@ public void joinQuerySelectOnlyOnOneTable() throws Exception {
assertContainsData(getDataRows(response), fields);
}

@Disabled("Disabled temporarily due to JSON format incompatibility with V2 and Legacy")
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what do you mean by this? Is this test case now supported in V2 and it fails?

Copy link
Author

@GabeFernandez310 GabeFernandez310 Apr 27, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This may have been a mistake. I think this was failing due to the data type refactor at first, and when I asked about it JSON format was the reason that was suggested for why it started failing all of a sudden, so I just put that as the description. This has been fixed since then, so I will remove these annotations.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

2ccebd9

@Test
public void fieldOrder() throws IOException {

Expand All @@ -492,6 +494,8 @@ public void fieldOrder() throws IOException {
testFieldOrder(expectedFields, expectedValues);
}


@Disabled("Disabled temporarily due to JSON format incompatibility with V2 and Legacy")
@Test
public void fieldOrderOther() throws IOException {

Expand Down
4 changes: 4 additions & 0 deletions integ-test/src/test/java/org/opensearch/sql/legacy/SQLIntegTestCase.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -634,6 +634,10 @@ public enum Index {
"calcs",
getMappingFile("calcs_index_mappings.json"),
"src/test/resources/calcs.json"),
DATE_FORMATS(TestsConstants.TEST_INDEX_DATE_FORMATS,
"date_formats",
getMappingFile("date_formats_index_mapping.json"),
"src/test/resources/date_formats.json"),
WILDCARD(TestsConstants.TEST_INDEX_WILDCARD,
"wildcard",
getMappingFile("wildcard_index_mappings.json"),
Expand Down
1 change: 1 addition & 0 deletions integ-test/src/test/java/org/opensearch/sql/legacy/TestsConstants.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -56,6 +56,7 @@ public class TestsConstants {
public final static String TEST_INDEX_BEER = TEST_INDEX + "_beer";
public final static String TEST_INDEX_NULL_MISSING = TEST_INDEX + "_null_missing";
public final static String TEST_INDEX_CALCS = TEST_INDEX + "_calcs";
public final static String TEST_INDEX_DATE_FORMATS = TEST_INDEX + "_date_formats";
public final static String TEST_INDEX_WILDCARD = TEST_INDEX + "_wildcard";
public final static String TEST_INDEX_MULTI_NESTED_TYPE = TEST_INDEX + "_multi_nested";
public final static String TEST_INDEX_NESTED_WITH_NULLS = TEST_INDEX + "_nested_with_nulls";
Expand Down
10 changes: 9 additions & 1 deletion integ-test/src/test/java/org/opensearch/sql/sql/DateTimeFunctionIT.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,8 @@

import static org.opensearch.sql.legacy.TestsConstants.TEST_INDEX_BANK;
import static org.opensearch.sql.legacy.TestsConstants.TEST_INDEX_CALCS;
import static org.opensearch.sql.legacy.TestsConstants.TEST_INDEX_DATE_FORMATS;
import static org.opensearch.sql.legacy.TestsConstants.TEST_INDEX_PEOPLE2;
import static org.opensearch.sql.legacy.plugin.RestSqlAction.QUERY_API_ENDPOINT;
import static org.opensearch.sql.util.MatcherUtils.rows;
import static org.opensearch.sql.util.MatcherUtils.schema;
Expand Down Expand Up @@ -40,7 +42,7 @@ public void init() throws Exception {
loadIndex(Index.BANK);
loadIndex(Index.CALCS);
loadIndex(Index.PEOPLE2);
loadIndex(Index.CALCS);
loadIndex(Index.DATE_FORMATS);
}

// Integration test framework sets for OpenSearch instance a random timezone.
Expand Down Expand Up @@ -1278,6 +1280,12 @@ public void testTimeFormat() throws IOException {
verifyTimeFormat(timestamp, "timestamp", timestampFormat, timestampFormatted);
}

@Test
public void testReadingDateFormats() throws IOException {
String query = String.format("SELECT * FROM %s", TEST_INDEX_DATE_FORMATS);
executeQuery(query);
}

protected JSONObject executeQuery(String query) throws IOException {
Request request = new Request("POST", QUERY_API_ENDPOINT);
request.setJsonEntity(String.format(Locale.ROOT, "{\n" + " \"query\": \"%s\"\n" + "}", query));
Expand Down
166 changes: 166 additions & 0 deletions integ-test/src/test/resources/date_formats.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,166 @@
{"index": {}}
{"name": "epoch_millis", "epoch_millis": "450608862000.123456"}
{"index": {}}
{"name": "epoch_second", "epoch_second": "450608862.000123456"}
{"index": {}}
{"name": "date_optional_time", "date_optional_time": "1984-04-12T09:07:42.000Z"}
{"index": {}}
{"name": "strict_date_optional_time", "strict_date_optional_time": "1984-04-12T09:07:42.000Z"}
{"index": {}}
{"name": "strict_date_optional_time_nanos", "strict_date_optional_time_nanos": "1984-04-12T09:07:42.000123456Z"}
{"index": {}}
{"name": "basic_date", "basic_date": "19840412"}
{"index": {}}
{"name": "basic_date_time", "basic_date_time": "19840412T090742.000Z"}
{"index": {}}
{"name": "basic_date_time_no_millis", "basic_date_time_no_millis": "19840412T090742Z"}
{"index": {}}
{"name": "basic_ordinal_date", "basic_ordinal_date": "1984103"}
{"index": {}}
{"name": "basic_ordinal_date_time", "basic_ordinal_date_time": "1984103T090742.000Z"}
{"index": {}}
{"name": "basic_ordinal_date_time_no_millis", "basic_ordinal_date_time_no_millis": "1984103T090742Z"}
{"index": {}}
{"name": "basic_time", "basic_time": "090742.000Z"}
{"index": {}}
{"name": "basic_time_no_millis", "basic_time_no_millis": "090742Z"}
{"index": {}}
{"name": "basic_t_time", "basic_t_time": "T090742.000Z"}
{"index": {}}
{"name": "basic_t_time_no_millis", "basic_t_time_no_millis": "T090742Z"}
{"index": {}}
{"name": "basic_week_date", "basic_week_date": "1984W154"}
{"index": {}}
{"name": "strict_basic_week_date", "strict_basic_week_date": "1984W154"}
{"index": {}}
{"name": "basic_week_date_time", "basic_week_date_time": "1984W154T090742.000Z"}
{"index": {}}
{"name": "strict_basic_week_date_time", "strict_basic_week_date_time": "1984W154T090742.000Z"}
{"index": {}}
{"name": "basic_week_date_time_no_millis", "basic_week_date_time_no_millis": "1984W154T090742Z"}
{"index": {}}
{"name": "strict_basic_week_date_time_no_millis", "strict_basic_week_date_time_no_millis": "1984W154T090742Z"}
{"index": {}}
{"name": "date", "date": "1984-04-12"}
{"index": {}}
{"name": "strict_date", "strict_date": "1984-04-12"}
{"index": {}}
{"name": "date_hour", "date_hour": "1984-04-12T09"}
{"index": {}}
{"name": "strict_date_hour", "strict_date_hour": "1984-04-12T09"}
{"index": {}}
{"name": "date_hour_minute", "date_hour_minute": "1984-04-12T09:07"}
{"index": {}}
{"name": "strict_date_hour_minute", "strict_date_hour_minute": "1984-04-12T09:07"}
{"index": {}}
{"name": "date_hour_minute_second", "date_hour_minute_second": "1984-04-12T09:07:42"}
{"index": {}}
{"name": "strict_date_hour_minute_second", "strict_date_hour_minute_second": "1984-04-12T09:07:42"}
{"index": {}}
{"name": "date_hour_minute_second_fraction", "date_hour_minute_second_fraction": "1984-04-12T09:07:42.000"}
{"index": {}}
{"name": "strict_date_hour_minute_second_fraction", "strict_date_hour_minute_second_fraction": "1984-04-12T09:07:42.000"}
{"index": {}}
{"name": "date_hour_minute_second_millis", "date_hour_minute_second_millis": "1984-04-12T09:07:42.000"}
{"index": {}}
{"name": "strict_date_hour_minute_second_millis", "strict_date_hour_minute_second_millis": "1984-04-12T09:07:42.000"}
{"index": {}}
{"name": "date_time", "date_time": "1984-04-12T09:07:42.000Z"}
{"index": {}}
{"name": "strict_date_time", "strict_date_time": "1984-04-12T09:07:42.000123456Z"}
{"index": {}}
{"name": "date_time_no_millis", "date_time_no_millis": "1984-04-12T09:07:42Z"}
{"index": {}}
{"name": "strict_date_time_no_millis", "strict_date_time_no_millis": "1984-04-12T09:07:42Z"}
{"index": {}}
{"name": "hour", "hour": "09"}
{"index": {}}
{"name": "strict_hour", "strict_hour": "09"}
{"index": {}}
{"name": "hour_minute", "hour_minute": "09:07"}
{"index": {}}
{"name": "strict_hour_minute", "strict_hour_minute": "09:07"}
{"index": {}}
{"name": "hour_minute_second", "hour_minute_second": "09:07:42"}
{"index": {}}
{"name": "strict_hour_minute_second", "strict_hour_minute_second": "09:07:42"}
{"index": {}}
{"name": "hour_minute_second_fraction", "hour_minute_second_fraction": "09:07:42.000"}
{"index": {}}
{"name": "strict_hour_minute_second_fraction", "strict_hour_minute_second_fraction": "09:07:42.000"}
{"index": {}}
{"name": "hour_minute_second_millis", "hour_minute_second_millis": "09:07:42.000"}
{"index": {}}
{"name": "strict_hour_minute_second_millis", "strict_hour_minute_second_millis": "09:07:42.000"}
{"index": {}}
{"name": "ordinal_date", "ordinal_date": "1984-103"}
{"index": {}}
{"name": "strict_ordinal_date", "strict_ordinal_date": "1984-103"}
{"index": {}}
{"name": "ordinal_date_time", "ordinal_date_time": "1984-103T09:07:42.000123456Z"}
{"index": {}}
{"name": "strict_ordinal_date_time", "strict_ordinal_date_time": "1984-103T09:07:42.000123456Z"}
{"index": {}}
{"name": "ordinal_date_time_no_millis", "ordinal_date_time_no_millis": "1984-103T09:07:42Z"}
{"index": {}}
{"name": "strict_ordinal_date_time_no_millis", "strict_ordinal_date_time_no_millis": "1984-103T09:07:42Z"}
{"index": {}}
{"name": "time", "time": "09:07:42.000Z"}
{"index": {}}
{"name": "strict_time", "strict_time": "09:07:42.000Z"}
{"index": {}}
{"name": "time_no_millis", "time_no_millis": "09:07:42Z"}
{"index": {}}
{"name": "strict_time_no_millis", "strict_time_no_millis": "09:07:42Z"}
{"index": {}}
{"name": "t_time", "t_time": "T09:07:42.000Z"}
{"index": {}}
{"name": "strict_t_time", "strict_t_time": "T09:07:42.000Z"}
{"index": {}}
{"name": "t_time_no_millis", "t_time_no_millis": "T09:07:42Z"}
{"index": {}}
{"name": "strict_t_time_no_millis", "strict_t_time_no_millis": "T09:07:42Z"}
{"index": {}}
{"name": "week_date", "week_date": "1984-W15-4"}
{"index": {}}
{"name": "strict_week_date", "strict_week_date": "1984-W15-4"}
{"index": {}}
{"name": "week_date_time", "week_date_time": "1984-W15-4T09:07:42.000Z"}
{"index": {}}
{"name": "strict_week_date_time", "strict_week_date_time": "1984-W15-4T09:07:42.000Z"}
{"index": {}}
{"name": "week_date_time_no_millis", "week_date_time_no_millis": "1984-W15-4T09:07:42Z"}
{"index": {}}
{"name": "strict_week_date_time_no_millis", "strict_week_date_time_no_millis": "1984-W15-4T09:07:42Z"}
{"index": {}}
{"name": "weekyear", "weekyear": "1984"}
{"index": {}}
{"name": "strict_weekyear", "strict_weekyear": "1984"}
{"index": {}}
{"name": "weekyear_week", "weekyear_week": "1984-W15"}
{"index": {}}
{"name": "strict_weekyear_week", "strict_weekyear_week": "1984-W15"}
{"index": {}}
{"name": "weekyear_week_day", "weekyear_week_day": "1984-W15-4"}
{"index": {}}
{"name": "strict_weekyear_week_day", "strict_weekyear_week_day": "1984-W15-4"}
{"index": {}}
{"name": "year", "year": "1984"}
{"index": {}}
{"name": "strict_year", "strict_year": "1984"}
{"index": {}}
{"name": "year_month", "year_month": "1984-04"}
{"index": {}}
{"name": "strict_year_month", "strict_year_month": "1984-04"}
{"index": {}}
{"name": "year_month_day", "year_month_day": "1984-04-12"}
{"index": {}}
{"name": "strict_year_month_day", "strict_year_month_day": "1984-04-12"}
{"index": {}}
{"name": "yyyy-MM-dd", "yyyy-MM-dd": "1984-04-12"}
{"index": {}}
{"name": "HH:mm:ss", "HH:mm:ss": "09:07:42"}
{"index": {}}
{"name": "yyyy-MM-dd||uuuu-DDD", "yyyy-MM-dd||uuuu-DDD": "1984-04-12"}
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we should have data that formats to each of the provided date formats to make sure that they are all working.
ie add data that conforms to uuuu-DDD format

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

0df21dd

{"index": {}}
{"name": "hour_minute_second||t_time", "hour_minute_second||t_time": "09:07:42"}
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we add multi-format field that covers both dates and times?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

0df21dd

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please add data that is formatted by the t_time formatter

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

0df21dd

Loading