[ETL-616] Implement Great Expectations to run on parquet data

[rxu17]

Purpose:

This draft PR adds the Great Expectations (GX) Parquet Glue jobs to the Recover ETL workflow. When JSON to Parquet workflow finishes, this will run the GX job per data type.

Currently this just supports running GX with expectations for the fitbitdailydata and healthkitv2workouts datasets. All other data types will have their jobs error out.

Changes:

Highlights big changes

New Code:

• run_great_expectations_on_parquet.py : This is the script to be run by the GX on Parquet jobs. There are some workarounds in the script notably in the add_data_docs_sites and add_validation_results_to_store functions to allow us to add validation results to the validation store and also have the data docs (our GX report) render them since we use a EphemeralDataContext context object without having to create checkpoints, a GX config file (which would likely have us have to confirm to a specific GX repo structure), etc. If we prefer to switch to using a more persistent data content object like FileDataContext, that could be explored further in this ticket.
• data values validation suite: this will be where we manually add our expectations to in the future. I find this way the easiest way because we have to look through the expectations gallery in order to find the expectation we want to add and in case we want to be able to validate our outputted set of expectations against this list.

Changes to old Code:

• Updated glue workflow triggers : because there is a limitation to the number of jobs per trigger (50 max), I split the previous trigger into two, one for each set of jobs for CompareParquet and the new GX on Parquet jobs.
• Pinned dependency urllib3<2 as part of additional python modules: There is a compatibility issue with boto3 and urllib3 2.0. We need to pin urllib3<2

Tests:

• Integration testing in AWS (currently running)
• Unit tests
• Tests that script can produce reports from sample validation suite in the shareable artifacts bucket

Viewable reports at (with AWS VPN turned on):

• fitbitdailydata gx report
• healthkitv2workouts gx report

Sample screenshots of a report:

EDIT: Also added tests to our CI/CD as previously they weren't automatically running all of them each time

[sonarqubecloud]

Quality Gate passed

Issues
42 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud