Data requirements comparison script

.gitignore

@@ -12,3 +12,7 @@ regression/output/*-*
 regression/connectathon
 regression/ecqm-content-r4-2021
 regression/ecqm-content-qicore-2022
+data-requirements/fqm-e-dr/*
+data-requirements/jan-2024-connectathon/*
+data-requirements/elm-parser-dr/*
+data-requirements/elm-parser-for-ecqms

README.md

@@ -887,39 +887,41 @@ The order of these populations is determined by most inclusive to least inclusiv
 To disable this behavior, use the `disableHTMLOrdering` calculation option.
 
 ### Statement-level HTML
-Optionally, `fqm-execution` can generate the stylized HTML markup for each individual statement. To access the statement-level HTML, specify the `buildStatementLevelHTML` in the `CalculationOptions` prior to measure calculation. From the `detailedResults` returned for a given patient, the `statementLevelHTML` will be available as an element on each `statementResult` whose relevance is *not* N/A.
+
+Optionally, `fqm-execution` can generate the stylized HTML markup for each individual statement. To access the statement-level HTML, specify the `buildStatementLevelHTML` in the `CalculationOptions` prior to measure calculation. From the `detailedResults` returned for a given patient, the `statementLevelHTML` will be available as an element on each `statementResult` whose relevance is _not_ N/A.
 
 ```typescript
 [
   {
-    "patientId": "test-patient",
-    "detailedResults": [
+    patientId: 'test-patient',
+    detailedResults: [
       {
-        "groupId": "test-group",
-        "statementResults": [
+        groupId: 'test-group',
+        statementResults: [
           // no HTML returned since relevance is NA
           {
-            "libraryName": "MATGlobalCommonFunctionsFHIR4",
-            "statementName": "Patient",
-            "final": "NA",
-            "relevance": "NA",
-            "isFunction": false,
-            "pretty": "NA"
+            libraryName: 'MATGlobalCommonFunctionsFHIR4',
+            statementName: 'Patient',
+            final: 'NA',
+            relevance: 'NA',
+            isFunction: false,
+            pretty: 'NA'
           },
           {
-            "libraryName": "CancerScreening",
-            "statementName": "SDE Sex",
-            "final": "TRUE",
-            "relevance": "TRUE",
-            "isFunction": false,
-            "pretty": "CODE: http://hl7.org/fhir/v3/AdministrativeGender F, Female",
-            "statementLevelHTML": "<pre style=\"tab-size: 2; border-bottom-width: 4px; line-height: 1.4\"\n  data-library-name=\"CancerScreening\" data-statement-name=\"SDE Sex\">\n...\n</pre>"
-          },
+            libraryName: 'CancerScreening',
+            statementName: 'SDE Sex',
+            final: 'TRUE',
+            relevance: 'TRUE',
+            isFunction: false,
+            pretty: 'CODE: http://hl7.org/fhir/v3/AdministrativeGender F, Female',
+            statementLevelHTML:
+              '<pre style="tab-size: 2; border-bottom-width: 4px; line-height: 1.4"\n  data-library-name="CancerScreening" data-statement-name="SDE Sex">\n...\n</pre>'
+          }
         ]
       }
     ]
   }
-]
+];
 ```
 
 ## Group Clause Coverage Highlighting
@@ -1214,6 +1216,27 @@ const evaluateMeasure = async (args, { req }) => {
 };
 ```
 
+## Special Testing
+
+### Regression Testing
+
+The `./regression` directory is organized for internal calculation testing between branches of `fqm-execution`. The idea is that if changes are made to calculation on a local branch, running regression will compare the calculation output of measures in the following three repositories (`connectathon`, `ecqm-content-qi-2022`, `ecqm-content-r4-2021`) from the local branch to the calculation output of those measures from the master branch (or another user-specified branch). The `./run-regression.sh` script takes the following options:
+
+```
+-b, --base-branch <branch-name>                   Base branch to compare results with (default: master)
+-v, --verbose <boolean>                           Use verbose regression. Will print out diffs of failing JSON files with spacing (default: false)
+```
+
+To run the regression testing script, in the `./regression` directory run:
+
+```bash
+./run-regression
+```
+
+### Data Requirements Testing
+
+The `./data-requirements` directory is organized for internal data-requirements calculation testing between `fqm-execution` and the `fhir_review` branch of [elm-parser-for-ecqms](https://github.com/projecttacoma/elm-parser-for-ecqms/tree/fhir_review). See the [README](https://github.com/projecttacoma/fqm-execution/data-requirements/README.md) in this directory for more information.
+
 # Contributing
 
 For suggestions or contributions, please use [GitHub Issues](https://github.com/projecttacoma/fqm-execution/issues) or open a [Pull Request](https://github.com/projecttacoma/fqm-execution/pulls).

data-requirements/README.md

@@ -0,0 +1,54 @@
+# fqm-execution Data Requirements Output Testing/Comparison
+
+This directory includes scripts for comparing the data-requirements output of [fqm-execution](https://github.com/projecttacoma/fqm-execution) to the data-requirements output of the fhir_review branch of the [elm-parser-for-ecqms](https://github.com/projecttacoma/elm-parser-for-ecqms/tree/fhir_review).
+
+## Getting Data Requirements from the elm-parser-for-ecqms
+
+The scripts in this directory will get the data requirements output from the elm-parser-for-ecqms fhir_review branch for the January 2024 Connectathon bundles. On the fhir_review branch of elm-parser-for-ecqms, data-requirements are calculated for the measures in [elm-parser-for-ecqms/measures/qicore/measures](https://github.com/projecttacoma/elm-parser-for-ecqms/tree/fhir_review/measures/qicore/measures) by running the command `ruby parse_elm.rb --bundle qicore`. The results are outputted to JSON files per measure to `elm-parser-for-ecqms/data_requirements/library`. This is all done by the script and the results are moved to the `elm-parser-dr` directory.
+
+## Getting Data Requirements from fqm-execution
+
+The scripts in this directory will get the data requirements output from fqm-execution for the January 2024 Connectathon bundles. The data requirements are calculated with fqm-execution on every run for ease of testing changes to fqm-execution. Since the January 2024 Connectathon bundles are not in a GitHub repository, they will have to be manually dropped into the `jan-2024-connectathon` directory that is empty. The data requirements JSON output files will be moved to the `fqm-e-dr` directory after calculation.
+
+## Comparing Data Requirements
+
+Right now there are two ways to compare data-requirements. `compare.sh` takes a similar approach to the regression tests: data-requirements are calculated using fqm-execution and then using elm-parser-for-ecqms and their outputs are compared. This will likely not be useful with the current state of fqm-execution's data-requirements calculation as it is very different from the elm-parser-for-ecqms, but it may be in the future.
+
+`summary-compare.sh` also compares the data-requirements outputs from fqm-execution and elm-parser-for-ecqms, but in a way that parses through each of the data-requirements in the data-requirements array. This script may be more useful to look at how many data-requirements of each type are being outputted by either repository and if they match up. By default, this script compares the data requirements outputs of all of the measures, however using the -m|--measure flag, one can specify a single measure to compare.
+
+## Running the Scripts
+
+Before running any of the scripts, be sure to populate the `jan-2024-connectathon` directory with the corresponding measure bundles. Also confirm that additional dependencies required for these scripts are installed with `npm install`.
+
+To run `compare.sh`:
+
+```
+./compare.sh
+```
+
+To run `summary-compare.sh` for all measures:
+
+```
+./summary-compare.sh
+```
+
+To run `summary-compare.sh` for one measure (example: CMS996):
+
+```
+./summary-compare.sh -m CMS996
+```
+
+## Summary Compare Output
+
+Right now, the `summary-compare.sh` script compares three things: the types of data requirements in either output, the data requirements of each type in either output, and the mustSupports of each data requirement in either output.
+
+The summary output is currently structured in the following format:
+
+```
+-----Data Requirements Comparison for <CMS measure>-----
+[PASS/DIFF]: Whether or not either of the outputs has data requirements of a type that the other does not
+
+[PASS/FAIL (Data Requirement Type)]: Whether or not the data requirements of each output match my codeFilter.valueSet and provides details if they do not
+MUST SUPPORTS
+[MUST SUPPORTS PASS/FAIL(Data Requirement Type-ValueSet)]: Whether or not the mustSupports of the data requirements of each output match
+```

data-requirements/compare.sh

@@ -0,0 +1,80 @@
+#!/bin/bash
+
+RED='\033[0;31m'
+GREEN='\033[0;32m'
+NC='\033[0m'
+
+VERBOSE=false
+
+function usage() {
+    cat <<USAGE
+
+    Usage: $0 [-v|--verbose]
+
+    Options:
+        -v/--verbose:           Use verbose comparison. Will print out diffs of failing JSON files with spacing (default: false)
+USAGE
+    exit 1
+}
+
+while test $# != 0
+do
+    case "$1" in 
+    -v | --verbose) VERBOSE=true;;
+    esac
+    shift
+done
+
+echo "Gathering data-requirements output from the fhir_review branch of elm-parser-for-ecqms"
+
+# Clone the elm-parser-for-ecqms in the data-requirements directory if it hasn't been, swtich to the fhir_review branch,
+# run parse_elm.rb with --bundle qicore to get data-requirements for measure bundles from the January 2024 Connectathon
+if [ ! -d "elm-parser-for-ecqms" ]; then
+    git clone https://github.com/projecttacoma/elm-parser-for-ecqms.git
+    git fetch --all
+    cd elm-parser-for-ecqms
+    git checkout "fhir_review"
+    ruby parse_elm.rb --bundle qicore
+    cd ..
+    if [ -d "elm-parser-dr" ]; then
+        rm -rf elm-parser-dr/*.json
+    fi
+    mkdir elm-parser-dr
+    SOURCE_DIR="elm-parser-for-ecqms/data_requirements/library"
+    TARGET_DIR="elm-parser-dr"
+
+    # Move all files from source to target directory
+    mv "$SOURCE_DIR"/* "$TARGET_DIR"
+fi
+
+echo "Gathering data-requirements output from fqm-execution using the measure bundles from the January 2024 Connectathon"
+
+npx ts-node fqm-e-dr.ts
+
+for file in "fqm-e-dr"/*; do
+    BASE_PATH="$(basename "$file")"
+    BASE_NAME=${BASE_PATH%"-dr.json"*}
+
+    FQM_E_DR=$file
+    ELM_PARSER_DR="elm-parser-dr/${BASE_NAME}.xml.json"
+
+    if ! test -f "$FQM_E_DR"; then
+        echo -e "${RED}FAIL${NC}: $FQM_E_DR does not exist"
+    fi
+
+    if ! test -f "$ELM_PARSER_DR"; then
+        echo -e "${RED}FAIL${NC}: $ELM_PARSER_DR does not exist"
+    fi
+
+    if cmp --silent $FQM_E_DR $ELM_PARSER_DR; then  
+        echo -e "${GREEN}PASS${NC}: $BASE_NAME"
+    else 
+        echo -e "${RED}FAIL${NC}: $FQM_E_DR and $ELM_PARSER_DR are different"
+
+        if [ $VERBOSE = "true" ]; then
+            diff $FQM_E_DR $ELM_PARSER_DR
+        fi
+    fi
+done
+
+echo "Finished"

data-requirements/fqm-e-dr.ts

@@ -0,0 +1,65 @@
+import fs, { readdirSync } from 'fs';
+import path from 'path';
+import { Calculator } from '../src';
+
+const RESET = '\x1b[0m';
+const FG_YELLOW = '\x1b[33m';
+const FG_GREEN = '\x1b[32m';
+
+const JAN_2024_CONNECTATHON_BASE_PATH = path.join(__dirname, './jan-2024-connectathon');
+
+/**
+ * The purpose of this function is to go through all of the measure bundles in the jan-2024-connectathon
+ * directory, calculate their data-requirements, and output their data-requirements to a JSON file
+ * corresponding to the name of the measure in the fqm-e-dr directory
+ */
+async function main() {
+  // if the fqm-e-dr directory already exists, remove it and its contents
+  if (fs.existsSync('./fqm-e-dr')) {
+    readdirSync('./fqm-e-dr').forEach(file => {
+      if (file.endsWith('.json')) {
+        fs.rmSync(`./fqm-e-dr/${file}`);
+      }
+    });
+  } else {
+    // create new fqm-e-dr directory within the data-requirements directory
+    fs.mkdirSync('./fqm-e-dr');
+  }
+
+  // get all of the file names (short and fullPath) from the jan-2024-connectathon directory
+  const allBundles = fs
+    .readdirSync(JAN_2024_CONNECTATHON_BASE_PATH)
+    .filter(f => !f.startsWith('.'))
+    .map(f => ({
+      shortName: f.split('v')[0],
+      fullPath: path.join(JAN_2024_CONNECTATHON_BASE_PATH, f)
+    }));
+
+  for (const bundle of allBundles) {
+    const measureBundle = JSON.parse(fs.readFileSync(bundle.fullPath, 'utf8')) as fhir4.Bundle;
+
+    // try to calculate the data requirements for the measure bundle
+    try {
+      const { results } = await Calculator.calculateDataRequirements(measureBundle, {});
+
+      // write the data-requirements results to the measure's shortName-dr.json file in the fqm-e-dr directory
+      fs.writeFileSync(`./fqm-e-dr/${bundle.shortName}-dr.json`, JSON.stringify(results, undefined, 2), 'utf8');
+
+      console.log(`${FG_GREEN}%s${RESET}: Results written to ./fqm-e-dr/${bundle.shortName}-dr.json`, 'SUCCESS');
+    } catch (e) {
+      if (e instanceof Error) {
+        fs.writeFileSync(
+          `./fqm-e-dr/${bundle.shortName}-dr.json`,
+          JSON.stringify({ error: e.message }, undefined, 2),
+          'utf8'
+        );
+        console.log(
+          `${FG_YELLOW}%s${RESET}: Results written to ./fqm-e-dr/${bundle.shortName}-dr.json`,
+          'EXECUTION ERROR'
+        );
+      }
+    }
+  }
+}
+
+main().then(() => console.log('fqm-execution data-requirement calculation finished'));

data-requirements/summary-compare.sh

@@ -0,0 +1,59 @@
+#!/bin/bash
+
+RED='\033[0;31m'
+GREEN='\033[0;32m'
+NC='\033[0m'
+
+MUST_SUPPORT=false
+
+function usage() {
+    cat <<USAGE
+
+    Usage: $0 [-m|--measure <measure-name>] 
+
+    Options:
+        -m/--measure:   Name of the CMS measure to compare data requirements results (default: all)
+USAGE
+    exit 1
+}
+
+while test $# != 0
+do 
+    case "$1" in
+    -m | --measure) 
+        shift
+        MEASURE=$1
+        ;;
+    *) usage ;;
+    esac
+    shift
+done
+
+
+echo "Gathering data-requirements output from the fhir_review branch of elm-parser-for-ecqms"
+
+# Clone the elm-parser-for-ecqms in the data-requirements directory if it hasn't been, swtich to the fhir_review branch,
+# run parse_elm.rb with --bundle qicore to get data-requirements for measure bundles from the January 2024 Connectathon
+if [ ! -d "elm-parser-for-ecqms" ]; then
+    git clone https://github.com/projecttacoma/elm-parser-for-ecqms.git
+    git fetch --all
+    cd elm-parser-for-ecqms
+    git checkout "fhir_review"
+    ruby parse_elm.rb --bundle qicore
+    cd ..
+    if [ -d "elm-parser-dr" ]; then
+        rm -rf elm-parser-dr/*.json
+    fi
+    mkdir elm-parser-dr
+    SOURCE_DIR="elm-parser-for-ecqms/data_requirements/library"
+    TARGET_DIR="elm-parser-dr"
+
+    # Move all files from source to target directory
+    mv "$SOURCE_DIR"/* "$TARGET_DIR"
+fi
+
+echo "Gathering data-requirements output from fqm-execution using the measure bundles from the January 2024 Connectathon"
+
+npx ts-node fqm-e-dr.ts
+
+npx ts-node summary-compare.ts $MEASURE