[ETL-499] JSON to Parquet support for Garmin data types #66
Conversation
This requires us to use the sceptre keyword template_bucket_name in stack config so that we don't hit the --templateBody file size validation limit in `aws cloudformation`.
| spark_df = table.toDF() | ||
| if "InsertedDate" in field_names: |
There was a problem hiding this comment.
Was this a recent development that any data type with InsertedDate that has duplicates should be dropped based on InsertedDate? Besides just garmin data
There was a problem hiding this comment.
There was an email with Care Evolution. Sorry I forgot to cc you! Here is their words:
For files that contain "InsertedDate", the "InsertedDate" should be used to identify the most recent values and resolve duplicates. The "InsertedDate" is included in those files because it is possible for there to be duplicates within the same export file. For the files without InsertedDate, you should continue to use the date included in the filename to resolve duplicates.
There was a problem hiding this comment.
Does there need to be updates to the tests to have examples with InsertedDate for this new change?
Blocked by #65
Changes fall into three categories:
config/*files - this replaces references to the S3 bucket where we store our templates so that we don't run into the file size limitation on the--templateBodyparameter when theaws cloudformationtool validates the templates.src/glue/jobs/json_to_parquet.py- This adds the index fields for Garmin data types to ourINDEX_FIELD_MAPglobal, adds additional logic to reference theInsertedDatefield when dropping duplicates, and does a small amount of refactoring.src/glue/resources/table_columns.yaml- This adds the schema definitions for Garmin data types while reflecting the transformations we do for these data types in [ETL-505] Add Garmin transforms to S3 to JSON job #65 .