Question about value sets and OIDs files

[kchapple]

Greetings! @okeefm @AndrewBird81 @adongare

I'm developing an integration with cqm-execution with OpenEMR. We have some measures that we would like to run through the service, but they are not in your repository.

I'm wondering if there's a tool, or any documentation available on how you generate the value_sets.json that go with each measure? Are the files available somewhere for download, or are they generated from the XML value sets provided by NIH like these? https://vsac.nlm.nih.gov/download/ecqm?rel=20200507

I'm also wondering the same about the OIDS file. where do you obtain the oids file that is used for your model generation? I see the modelinfo files here: https://github.com/cqframework but don't see the OIDs files other than in your repository. Do you know where the most reliable place is where we can obtain updates to the OIDs JSON and qdm-modelinfo XML files for our PHP model generator?

Thanks,
Ken

[adongare]

Hi @kchapple,

I'm assuming this is for QDM measures.

We have some measures that we would like to run through the service, but they are not in your repository.

We don't maintain measures in our repository. We get measures from Measure Authoring Tool which then will be parsed using cqm-parsers library and converted into models using cqm-models. cqm-models is the library for model generation.

valuesets.json
To produce valuset.json, there is no such document as far as I know. We have a ruby library called cqm-parsers that makes an API call to vsac SVC api to get the valuesets and convert them into valueset model/json. Here is the SVC API guide that we use to get the valusets by OID https://www.nlm.nih.gov/vsac/support/usingvsac/vsacsvsapiv2.html

Modelinfo xml
We get the modelinfo file from cqframework clinical_quality_language repository. You can find modelinfo versions here: https://github.com/cqframework/clinical_quality_language/tree/master/Src/java/qdm/src/main/resources/gov/healthit/qdm
we use this file to generate the models in our cqm-models

OID Json file
I'm not sure how is this file generated or where to get this file from because this was there before I joined the project but I can tell you, you can find all the qdm datatypes and their templates including OIDs in volume 3 of CQL-based HQMF IG STU 4. You would probably find it here: https://www.hl7.org/implement/standards/product_brief.cfm?product_id=405

@okeefm, @AndrewBird81 feel free to add if I missed anything.

Thanks!

[kchapple]

Thank you so much for responding @adongare ! The information you provided is very helpful.

Yes, this is for CQM measures. I have ported your cqm-models to PHP and generating PHP QDM models from the modelinfo XML for use with the cqm-execution library. The model-generation tool requires the oids file, and the cqm-execution tool requires the additional value_sets.json file (which I now know how to generate.)

I thought that these files contained in your repo were the JSON measures directly from CMS, but maybe they went through some sort of transformation to be compatible for use with cqm-execution?
https://github.com/projecttacoma/cqm-execution/tree/master/spec/fixtures/json/cqm_measures

[adongare]

JSON files are not directly from CMS. We generate them before sending to cqm-execution. Some massaging happens in cqm-parsers like parsing measure's HQMF xml file, fetching valuesets from VSAC APIs and populating the models. models then can be converted to json.

[kchapple]

Awesome thank you. I will check out your parsers library. Cheers!

[kchapple]

@adongare Sorry, one more question! Is there a reason why you don't include all the CMS measures in the repo? Are the other measures untested, or known to not work with the calculator? Would it be possible to add the other measures to the repo (the json fixtures and value-sets)?
Thanks!

[adongare]

@kchapple No worries! The measures that we included are used just for unit testing to make sure the calculation is working and we have enough test coverage. however, the actual testing is done in Bonnie(https://bonnie.healthit.gov/) Users uploads their CMS measures or any test measures to Bonnie and then create the test patient data to evaluate and view the results in UI. We could include all the CMS measures but that doesn't add much value as measures keep changing and they are supposed to be evaluated by measure developers in UI Bonnie.

[kchapple]

Ahh, makes sense! So Bonnie must do the parsing on the fly? Unfortunately, we're in PHP land, and the parsers are in Ruby. I wonder what strategies others have taken to utilize the CQM calculator in a system outside of Bonnie. We'll probably have to have the parsers running off to the side, and check the measure-supporting files into our repo periodically, or can Bonnie be used as a service to take the raw measures and provide the massaged files as output? By the way, you've been super helpful @adongare , thank you so much!

[adongare]

Sorry for my slow responses. lots of things going on at the moments.
We do have the Bonnie APIs(https://bonnie.healthit.gov/api) but quite different from what you are describing. Currently these APIs expect the measure and patient data to be present in Bonnie system and then you make an API call that will give you calculation results for all the test patients. I know this would not fulfill your requirement.
We can run cqm-parsers as service that would take measure package and return back the cqm-execution executable measure however I would not be able to decide on that. I'll have to talk to the leads and take their thoughts and get back to you.

[kchapple]

That would be amazing. A service like that would be very valuable for projects like OpenEMR trying to integrate the calculator. Let me know what the leads think. Thanks very much!

[adongare]

@kchapple, I ran this idea by our leads and learned that the creation of a service is not currently included in our immediate product release roadmap, however we will include it for future consideration. We would suggest your team continues to explore building upon the available code to develop a service specific to your use case. If our team can be of assistance to you as you research the current code, features and functions, please let us know. We appreciate your interest in the these tools.

[kchapple]

Thanks! @adongare
For the time-being I will spin up a build of the parsers so we can generate the JSON value-sets and format the CMS measures, and we'll probably just store those in our repo, or in a module like you guys do with Bonnie measures. For now, we will just have to specify an external process for OpenEMR admins to run the parsers to format those files when new versions of the measures are released. Hopefully it works! I'll let you know how it goes.

Just to confirm, the Ruby parser tool should be able to parse these measures' eCQM structure, and produce JSON that can be run through the cqm-execution?

I wonder how your team decides which CMS measures to keep in your repos? There seems to be many, but not all of them coincide with the requirements for Eligible Professional / Eligible Clinician eCQMs.

Thanks again!

[adongare]

@kchapple Yeah, it should with lil bit of changes. If you had gone through the cqm-models repository, you would have noticed two types of models-

1. ruby mongoids- https://github.com/projecttacoma/cqm-models/tree/master/app/models which we use for bonnie backend functionality.
2. javascript mongoose models- https://github.com/projecttacoma/cqm-models/tree/master/app/assets/javascripts which we use to execute the measure directly in browser.

So, this is how we use it-
cqm-parsers parses the measure HQMF file and converts it into Ruby mongoid models that can be converted to JSON(probably you would have to implement the endpoint/action to convert ruby measure model to JSON and return it back in your service). On execution side, once we have JSON, we need to convert it into javascript models and pass it to cqm-execution calculator https://github.com/projecttacoma/cqm-execution/blob/master/lib/models/calculator.js.

Regarding keeping CMS measures:
We really don't decide on which CMS measures to keep. The ones that you see in repository are just for our internal unit/integration test purpose. Just consider them as some test measures, nothing to do with CMS measures. Also, we don't need to keep all CMS measures because measure developers use Bonnie tool for CMS measure testing. So they use Measure Authoring Tool to author/download masures and upload to bonnie and see the calculation results themselves.

This might sound bit confusing. Feel free if you have further questions. Thanks!

[kchapple]

I think I got it. Thank you for being so helpful!

[kchapple]

I am thinking the easiest solution for OpenEMR would be a simple command-line Ruby script, with a dependency on this cqm-parsers module, that takes a directory as input and just runs over the packages in the directory and outputs the JSON measures in the same directory. So it would take CMS2v9.zip and put the JSON output in to a directory 'CMS2v9'

I'm looking at the usage section here:
https://github.com/projecttacoma/cqm-parsers#usage-mat-package-loading

Is there a value-set loader implementation that loads files from the file system (not from VSAC API)?

If not, is there a loader that doesn't use a username and password? The VSAC no longer accepts password auth, but we can generate API keys.

Thanks!

Ken

[adongare]

Hi @kchapple, there is no valueset loader implementation that loads valuesets from filesystem. But we have recently updated cqm-parsers bonnie_version branch(https://github.com/projecttacoma/cqm-parsers/tree/bonnie_version) to work with VSAC API key along with sme bug fixes. We have not merged those changes to master branch yet. You can use this branch. I'll update you once we merge it to master.

[kchapple]

Hi @adongare
I wrote a little Ruby script to create the value_sets and measure JSON. (disclaimer: I've never worked in Ruby before)
https://github.com/mi-squared/oe-cqm-parsers

git clone https://github.com/mi-squared/oe-cqm-parsers

cd oe-cqm-parsers

put your VSAC API key in script.rb around line 12:

api_key: 'put your api key here'

bundle install

bundle exec ruby script.rb

It's working so far with a little fix to add require 'singleton' to lib/utils/counter.rb (maybe your rails autoloader did this automatically?)
mi-squared/cqm-parsers@27e9143

Now I'm processing the measures, and the script works to extract the JSON for the first 14 CMS measures, but the script chokes on measure CMS138v8.zip (attached to this issue) on this line:
https://github.com/mi-squared/cqm-parsers/blob/cc6c0ef9c999191a7f9ce893ebc380beacf512b0/lib/measure-loader/cql_loader.rb#L111

Do you have any pointers on how we might fix it so we can process all the CMS measures?

Here's the exception:

Uncaught exception: Value Set (2.16.840.1.113883.3.526.3.1020) is empty.
	/Users/kchapple/.rvm/gems/ruby-2.7.0/bundler/gems/cqm-parsers-27e9143b5149/lib/util/vsac_api.rb:237:in `process_and_validate_vsac_response'
	/Users/kchapple/.rvm/gems/ruby-2.7.0/bundler/gems/cqm-parsers-27e9143b5149/lib/util/vsac_api.rb:213:in `block in get_multiple_valuesets'
	/Users/kchapple/.rvm/gems/ruby-2.7.0/bundler/gems/cqm-parsers-27e9143b5149/lib/util/vsac_api.rb:211:in `map'
	/Users/kchapple/.rvm/gems/ruby-2.7.0/bundler/gems/cqm-parsers-27e9143b5149/lib/util/vsac_api.rb:211:in `get_multiple_valuesets'
	/Users/kchapple/.rvm/gems/ruby-2.7.0/bundler/gems/cqm-parsers-27e9143b5149/lib/measure-loader/vsac_value_set_loader.rb:34:in `retrieve_and_modelize_value_sets_from_vsac'
	/Users/kchapple/.rvm/gems/ruby-2.7.0/bundler/gems/cqm-parsers-27e9143b5149/lib/measure-loader/cql_loader.rb:111:in `create_measure'
	/Users/kchapple/.rvm/gems/ruby-2.7.0/bundler/gems/cqm-parsers-27e9143b5149/lib/measure-loader/cql_loader.rb:21:in `extract_measures'
	/Users/kchapple/RubymineProjects/oe-cqm-parsers/script.rb:34:in `block in <top (required)>'
	/Users/kchapple/RubymineProjects/oe-cqm-parsers/script.rb:23:in `each'
	/Users/kchapple/RubymineProjects/oe-cqm-parsers/script.rb:23:in `<top (required)>'

Process finished with exit code 1

CMS138v8.zip

[adongare]

@kchapple That usually happens when valueset is empty and by that I mean valueset doesn't have codes in it. cqm-parsers doesn't allow empty valuesets. However for this valueset that's not the case. I see there is one code for this valueset.
Can you verify that you are passing all the correct options to the API? VSAC API call would look like like this:

https://vsac.nlm.nih.gov/vsac/svs/RetrieveMultipleValueSets?id=2.16.840.1.113883.3.526.3.1020&profile=eCQM%20Update%202021-05-06&includeDraft=yes&ticket=MY_VALID_TICKET

Just a guess- you probably have wrong profile in url

[adongare]

Profile may vary for Valuesets. May not be same for all the measures. You can find VSAC valueset profiles here: https://vsac.nlm.nih.gov/vsac/profiles

[kchapple]

Thanks! I think you're probably correct. Is there a way to map a measure to the profile, or can I pass multiple profiles to the API?

[adongare]

measure to profile mapping:
That's hard to tell. We provide profile options in UI to the users and let them choose while uploading measures to Bonnie. They know which profile is suitable for their measures since they are the measure authors/testers.

Providing Multiple profiles:
I'm not sure about this if VSAC API takes multiple profiles. You would have to reach out to VSAC PAI team to confirm that.

[kchapple]

Thanks!
I was setting the 'default_profile' in APP_CONFIG env. I changed my vsav_options to this and it made it through all of the CMS measures:

vsac_options = {
  options: {
    profile: 'eCQM Update 2021-05-06'
  },
  api_key: 'my-api-key'
}

... and after this I was able to parse all the measures!

Another question (sorry I know I'm probably getting annoying!)

I get the parsed measures using measures = loader.extract_measures

I'm trying to create the input files for cqm-execution. When I create the JSON files from the measure objects I get from the loader, which attributes of the measure objects should I be using to create the file? I don't want to make an assumption, but currently I'm using:
measure.value_sets.to_json to create the value_sets.json file and
measure.to_json to create the actual measure JSON like CMS117v8.json

Is that correct to recreate the structure found in cqm-execution/spec/fixtures/cqm_measures or is there reference code in Bonnie that you can point me to that does that particular file-generation?

[adongare]

So when you say measure.value_sets.to_json or measure.to_json, is that a ruby measure model or javascript measure model? We never used/tested ruby measure json. We used javascript one.
I think first you have to convert your measure json response from cqm-parsers to this one https://github.com/projecttacoma/cqm-models/blob/master/app/assets/javascripts/cqm/Measure.js and then stringify it.

[kchapple]

measure.value_sets.to_json or measure.to_json, is that a ruby measure model or javascript measure model?

This is in Ruby, so I'll have to do further conversion to JavaScript, no prob!

Once it's in a JavaScript measure object, then is the entire Measure Object stringified for the measure JSON file (like CMS117v8.json) and just the value_set attribute for the value_sets.json?

[adongare]

That's right.

[kchapple]

@adongare Do you know where in the code there might be an example where the Ruby CMS measure models get converted/loaded into the JavaScript Mongoose models? Maybe in the cqm-models or in Bonnie somewhere?

[kchapple]

I have the Ruby models, and now I realize I don't really know what to do with them, LOL. I suppose I can ingest them into a standard JavaScript object and use that to populate the Mongoose models, but I'd rather see how you guys do it so the process is standard.

[kchapple]

Hi @okeefm @AndrewBird81 @adongare @sjpadgett
OpenEMR is very close to getting the measure-parsing working, but we are missing one key piece.

We run a Ruby script and are using the Measures::CqlLoader to generate the JSON. The problem we are having is that the JSON that is generated isn't compatible with the cqm-cexecution Calculator. Our generated JSON files are missing all of the "_type" fields that contain the QDM type strings. Those types seem to be required by the calculator.

Here is an example. On the left is your CMS137v7.json and on the right is our CMS137v8.json.
The "_type": "QDM::PatientCharacteristicPayer" is present in your example, but not in ours.

Is there a configuration option that we can pass to the CqlLoader to generate these types, or is there a step we're missing that adds the "_types" to the JSON?

Here is the basic script we're running, using default options for measure_details:

vsac_options = {
  options: {
    profile: 'eCQM Update 2021-05-06'
  },
  api_key: ENV['VSAC_API_KEY']
}

# Set the measure details. For defaults, you can just pass in {}.
measure_details = {}

# Initialize a value set loader, in this case we are using the VSACValueSetLoader.
value_set_loader = Measures::VSACValueSetLoader.new(vsac_options)

# Load a MAT package from test fixtures.
measure_file = File.new('cms_measures/' + measure_file_name)

# Initialize the CqlLoader with the needed parameters.
loader = Measures::CqlLoader.new(measure_file, measure_details, value_set_loader)

[sjpadgett]

The "_type": "QDM::PatientCharacteristicPayer" is present in your example, but not in ours.

While the measures source_data_criteria data models are generated with the _type attribute from the parser, the attribute is not included during converting to json using object.to_json.

To include _type add Mongoid.include_type_for_serialization = true somewhere in scope in your script.

@kchapple we may be able to close this issue if you want with our thanks to @adongare