Fix date format in csv files #148

kavithacm · 2023-07-13T16:18:59Z

Description

Date format of nested fields are modified according to advanced settingsDate format of nested fields according to advanced settings

Issues Resolved

#106

Check List

New functionality includes testing.
- All tests pass, including unit test, integration test and doctest
New functionality has been documented.
- New functionality has javadoc added
- New functionality has user manual doc added
Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

server/routes/utils/dataReportHelpers.ts

Signed-off-by: Kavitha Conjeevaram Mohan <mohakavi@amazon.com>

kavithacm · 2023-09-20T17:40:33Z

The data used to test is the following, also have attached the csv report generated using it

data._source:  {
  key1: {
    key11: 'value11',
    key12: '2023-04-26T04:34:32Z',
    key13: '2023-04-26T04:34:32Z'
  },
  key2: '2023-04-21T04:34:32Z',
  key3: 'value3',
  a: { b: { c: '2023-07-21T18:40:01Z' } }
}

fixed-csv.csv

pjfitzgibbons

The output CSV is still posting all days as Jan 01.

_index,_id,_version,_score,_source.key1.key11,_source.key1.key12,_source.key1.key13,_source.key2,_source.key3,_source.a.b.c
test_index3,test,1,1,value11,01/01/2023,01/01/2023,01/01/2023,value3,01/01/2023

I am going to strongly recommend you include the recommendations in issue #196.

Since the fix of this code is this far off the mark, the refactor recommended should take less than a day, and will improve testability and readability and help in debugging the issue at hand.

Signed-off-by: Kavitha Conjeevaram Mohan <mohakavi@amazon.com>

kavithacm · 2023-09-20T18:31:33Z

The output CSV is still posting all days as Jan 01.
_index,_id,_version,_score,_source.key1.key11,_source.key1.key12,_source.key1.key13,_source.key2,_source.key3,_source.a.b.c
test_index3,test,1,1,value11,01/01/2023,01/01/2023,01/01/2023,value3,01/01/2023
I am going to strongly recommend you include the recommendations in issue #196.

Since the fix of this code is this far off the mark, the refactor recommended should take less than a day, and will improve testability and readability and help in debugging the issue at hand.

I added the wrong csv file, can you check this one?
correct-csv.csv

joshuali925

could you update unit tests to cover this fix?

Signed-off-by: Kavitha Conjeevaram Mohan <mohakavi@amazon.com>

joshuali925

some questions

joshuali925 · 2023-10-02T17:09:39Z

server/routes/utils/__tests__/savedSearchReportHelper.test.ts

+        {
+          customer_birth_date: '2023-04-26T04:34:32Z',
+          order_date: '2023-04-26T04:34:32Z',
+          'products.created_on': '2023-04-26T04:34:32Z',
+        }


what does adding this do?

the previous hit data was missing required date fields and a field object which consists of all date fields from backend. This data is required for the test to pass. Currently this fields object is being sent from backend, so while mocking the hits, we need to also include field object as well. Adding a current format for example

{ _index: 'test_index3', _id: 'test', _version: 1, _score: 1, _source: { key1: { key11: 'value11', key12: '2023-04-26T04:34:32Z', key13: '2023-04-26T04:34:32Z' }, key2: '2023-04-21T04:34:32Z', key3: 'value3', a: { b: [Object] } }, fields: { 'key1.key12': [ '2023-04-26T04:34:32.000' ], 'key1.key13': [ '2023-04-26T04:34:32.000' ], key2: [ '2023-04-21T04:34:32.000' ], 'a.b.c': [ '2023-07-21T18:40:01.000' ] } }

joshuali925 · 2023-10-02T17:10:02Z

server/routes/utils/__tests__/savedSearchReportHelper.test.ts

@@ -526,7 +1369,6 @@ test('create report with empty/one/multiple(list) date values', async () => {
    undefined,
    mockLogger
  );
-
  expect(dataUrl).toEqual(
    'category,customer_gender,order_date\n' +
      'c1,Ma,[]\n' +


why the expected data didn't change with the test input change?

because we aren't checking all the date fields available in all tests, just the ones we need, so the expected data isn't modified but for the function to work as expected, all date value fields should be included in the hits.

joshuali925

thanks

ps48 · 2023-10-02T17:25:34Z

server/routes/utils/__tests__/savedSearchReportHelper.test.ts

+          'products.created_on': '2023-04-26T04:34:32Z',
+        }
+      ),
+      // }),


minor: can remove this comment and one line 922

ps48 · 2023-10-02T17:26:22Z

server/routes/utils/dataReportHelpers.ts

+            fieldDateValue.length !== 0 &&
+            fieldDateValue instanceof Array
+          ) {
+            console.log('entered');


minor: can remove the console log from here.

* Fix date format in csv files Signed-off-by: Kavitha Conjeevaram Mohan <mohakavi@amazon.com> * covered all test cases Signed-off-by: Kavitha Conjeevaram Mohan <mohakavi@amazon.com> * add index Signed-off-by: Kavitha Conjeevaram Mohan <mohakavi@amazon.com> * fixed jest tests Signed-off-by: Kavitha Conjeevaram Mohan <mohakavi@amazon.com> * remove commented code Signed-off-by: Kavitha Conjeevaram Mohan <mohakavi@amazon.com> * remove commented line and console log Signed-off-by: Kavitha Conjeevaram Mohan <mohakavi@amazon.com> --------- Signed-off-by: Kavitha Conjeevaram Mohan <mohakavi@amazon.com> (cherry picked from commit 93a8704) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

* Fix date format in csv files --------- (cherry picked from commit 93a8704) Signed-off-by: Kavitha Conjeevaram Mohan <mohakavi@amazon.com> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

* Fix date format in csv files * covered all test cases * add index * fixed jest tests * remove commented code * remove commented line and console log --------- (cherry picked from commit 93a8704) Signed-off-by: Kavitha Conjeevaram Mohan <mohakavi@amazon.com> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

* Fix date format in csv files Signed-off-by: Kavitha Conjeevaram Mohan <mohakavi@amazon.com> * covered all test cases Signed-off-by: Kavitha Conjeevaram Mohan <mohakavi@amazon.com> * add index Signed-off-by: Kavitha Conjeevaram Mohan <mohakavi@amazon.com> * fixed jest tests Signed-off-by: Kavitha Conjeevaram Mohan <mohakavi@amazon.com> * remove commented code Signed-off-by: Kavitha Conjeevaram Mohan <mohakavi@amazon.com> * remove commented line and console log Signed-off-by: Kavitha Conjeevaram Mohan <mohakavi@amazon.com> --------- Signed-off-by: Kavitha Conjeevaram Mohan <mohakavi@amazon.com> (cherry picked from commit 93a8704) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

* Fix date format in csv files Signed-off-by: Kavitha Conjeevaram Mohan <mohakavi@amazon.com> * covered all test cases Signed-off-by: Kavitha Conjeevaram Mohan <mohakavi@amazon.com> * add index Signed-off-by: Kavitha Conjeevaram Mohan <mohakavi@amazon.com> * fixed jest tests Signed-off-by: Kavitha Conjeevaram Mohan <mohakavi@amazon.com> * remove commented code Signed-off-by: Kavitha Conjeevaram Mohan <mohakavi@amazon.com> * remove commented line and console log Signed-off-by: Kavitha Conjeevaram Mohan <mohakavi@amazon.com> --------- Signed-off-by: Kavitha Conjeevaram Mohan <mohakavi@amazon.com> (cherry picked from commit 93a8704)

* Fix date format in csv files * covered all test cases * add index * fixed jest tests * remove commented code * remove commented line and console log --------- (cherry picked from commit 93a8704) Signed-off-by: Kavitha Conjeevaram Mohan <mohakavi@amazon.com> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

* Fix date format in csv files (#148) * Fix date format in csv files Signed-off-by: Kavitha Conjeevaram Mohan <mohakavi@amazon.com> * covered all test cases Signed-off-by: Kavitha Conjeevaram Mohan <mohakavi@amazon.com> * add index Signed-off-by: Kavitha Conjeevaram Mohan <mohakavi@amazon.com> * fixed jest tests Signed-off-by: Kavitha Conjeevaram Mohan <mohakavi@amazon.com> * remove commented code Signed-off-by: Kavitha Conjeevaram Mohan <mohakavi@amazon.com> * remove commented line and console log Signed-off-by: Kavitha Conjeevaram Mohan <mohakavi@amazon.com> --------- Signed-off-by: Kavitha Conjeevaram Mohan <mohakavi@amazon.com> (cherry picked from commit 93a8704) * update snapshots to fix CIs Signed-off-by: Eric <menwe@amazon.com> --------- Signed-off-by: Eric <menwe@amazon.com> Co-authored-by: Kavitha Conjeevaram Mohan <mohakavi@amazon.com>

kavithacm requested review from pjfitzgibbons, ps48, derek-ho, joshuali925, dai-chen, YANG-DB, rupal-bq, mengweieric, vmmusings, Swiddis, penghuo, seankao-az and anirudha as code owners July 13, 2023 16:19

kavithacm added the backport 2.3 label Jul 13, 2023

joshuali925 reviewed Jul 13, 2023

View reviewed changes

server/routes/utils/dataReportHelpers.ts Outdated Show resolved Hide resolved

kavithacm added 2 commits September 20, 2023 10:22

Fix date format in csv files

73a2ac8

Signed-off-by: Kavitha Conjeevaram Mohan <mohakavi@amazon.com>

covered all test cases

89e0a07

Signed-off-by: Kavitha Conjeevaram Mohan <mohakavi@amazon.com>

kavithacm force-pushed the csv-date-format-bug-fix branch from 2dbbf93 to 89e0a07 Compare September 20, 2023 17:22

pjfitzgibbons mentioned this pull request Sep 20, 2023

[REFACTOR] Code-cleanliness recommendations #196

Open

pjfitzgibbons previously requested changes Sep 20, 2023

View reviewed changes

add index

e7d5290

Signed-off-by: Kavitha Conjeevaram Mohan <mohakavi@amazon.com>

joshuali925 reviewed Sep 20, 2023

View reviewed changes

kavithacm added 2 commits October 2, 2023 03:12

fixed jest tests

cf67f4a

Signed-off-by: Kavitha Conjeevaram Mohan <mohakavi@amazon.com>

remove commented code

d36d8db

Signed-off-by: Kavitha Conjeevaram Mohan <mohakavi@amazon.com>

joshuali925 reviewed Oct 2, 2023

View reviewed changes

joshuali925 approved these changes Oct 2, 2023

View reviewed changes

ps48 reviewed Oct 2, 2023

View reviewed changes

opensearch-trigger-bot bot mentioned this pull request Oct 5, 2023

[Backport 2.x] Fix date format in csv files #202

Merged

pjfitzgibbons added the backport 2.11 label Oct 5, 2023

opensearch-trigger-bot bot mentioned this pull request Oct 5, 2023

[Backport 2.11] Fix date format in csv files #203

Merged

kavithacm added backport 2.5 backport 2.6 backport 2.7 backport 2.8 backport 2.9 backport 2.10 labels Oct 23, 2023

This was referenced Oct 23, 2023

[Backport 2.7] Fix date format in csv files #232

Open

[Backport 2.5] Fix date format in csv files #233

Open

opensearch-trigger-bot bot mentioned this pull request Oct 23, 2023

[Backport 2.9] Fix date format in csv files #234

Open

This was referenced Oct 23, 2023

[Backport 2.10] Fix date format in csv files #235

Merged

[Backport 2.8] Fix date format in csv files #236

Open

opensearch-trigger-bot bot mentioned this pull request Oct 23, 2023

[Backport 2.6] Fix date format in csv files #237

Open

kavithacm mentioned this pull request Nov 6, 2023

Backport csv fix to 2.3 #248

Open

6 tasks

Swiddis removed the backport-failed label Mar 28, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix date format in csv files #148

Fix date format in csv files #148

kavithacm commented Jul 13, 2023 •

edited

Loading

kavithacm commented Sep 20, 2023 •

edited

Loading

pjfitzgibbons left a comment •

edited

Loading

kavithacm commented Sep 20, 2023

joshuali925 left a comment

joshuali925 left a comment

joshuali925 Oct 2, 2023

kavithacm Oct 2, 2023

joshuali925 Oct 2, 2023

kavithacm Oct 2, 2023 •

edited

Loading

joshuali925 left a comment

ps48 Oct 2, 2023

kavithacm Oct 2, 2023

ps48 Oct 2, 2023

kavithacm Oct 2, 2023

Fix date format in csv files #148

Fix date format in csv files #148

Conversation

kavithacm commented Jul 13, 2023 • edited Loading

Description

Issues Resolved

Check List

kavithacm commented Sep 20, 2023 • edited Loading

pjfitzgibbons left a comment • edited Loading

Choose a reason for hiding this comment

kavithacm commented Sep 20, 2023

joshuali925 left a comment

Choose a reason for hiding this comment

joshuali925 left a comment

Choose a reason for hiding this comment

joshuali925 Oct 2, 2023

Choose a reason for hiding this comment

kavithacm Oct 2, 2023

Choose a reason for hiding this comment

joshuali925 Oct 2, 2023

Choose a reason for hiding this comment

kavithacm Oct 2, 2023 • edited Loading

Choose a reason for hiding this comment

joshuali925 left a comment

Choose a reason for hiding this comment

ps48 Oct 2, 2023

Choose a reason for hiding this comment

kavithacm Oct 2, 2023

Choose a reason for hiding this comment

ps48 Oct 2, 2023

Choose a reason for hiding this comment

kavithacm Oct 2, 2023

Choose a reason for hiding this comment

kavithacm commented Jul 13, 2023 •

edited

Loading

kavithacm commented Sep 20, 2023 •

edited

Loading

pjfitzgibbons left a comment •

edited

Loading

kavithacm Oct 2, 2023 •

edited

Loading