Accuracy
This page describes the work performed in order to demonstrate the accuracy of the OMOP2OBO mappings. This work was specifically designed to verify the accuracy of manually constructed mappings (i.e. mappings that were not created from automatic alignment of existing database cross-references or exact string mappings). A subset of the most difficult manual and manual constructor mappings were randomly selected and verified by members of the clinical team shown below. Additional information about this task is provided below.
| Timeline | Documentation | Ontologies |
|---|---|---|
10/2018-11/2019 Perform Mapping04/2019-06/2019 Clinician verification |
Instructions for Experts Verification Spreadsheet |
Human Phenotype Ontology (HPO) Disease Ontology (DOID) Mondo Disease Ontology (MONDO) |
Domain Expert: Jordan Wyrwa, DO
This form was used by our domain expert to verify a random subset (n=2,000) of the mappings between OMOP condition concepts and ontology terms from HPO and DOID.
As described in the referenced document, the domain expert was asked to do the following when completing this task:
- Verify the condition concept mappings to
DOIDandHPO - For incorrect mappings, search each condition concept in the ontology browser to find a better mapping
- Discuss all incorrect mappings with project lead until an agreement was reached
Updates: Since the verification was performed, all DOID concepts were replaced by MONDO. Exact match alignments between these ontologies were utilized to ensure accurate translation between ontologies. In addition to providing the exact mappings to MONDO, the spreadsheet (link above) also contains updated mappings (marked with V1.0 in the header) to illustrate how the mappings have been updated as the algorithm evolves.
A screenshot of the results spreadsheet (link provided in table at top of section) is provided below:
- The number of needed iterations to reach agreement on the mappings varied with:
- 1,747 (87.35%) requiring no iterations
- 168 (8.40%) requiring 1 iteration
- 70 (3.50%) requiring 2 iterations
- 15 (0.75%) requiring 3 or more iterations
- The majority of the mappings were perceived to be correct (64.10%; n=1,282) and 22.95% (n=459) were found to be incorrect. Just under 13% of the mappings required additional discussion of which, 56 (2.80%) were found to be incorrect
| Timeline | Documentation | Ontologies |
|---|---|---|
01/2019-10/2019 Perform Mapping02/2019-04/2019 Pharmacist verification |
Instructions for Experts Verification Spreadsheet |
Chemical Entities of Biological Interest (ChEBI) Vaccine Ontology (VO) NCBI Organism Taxonomy (NCBITaxon) Protein Ontology (PRO) |
Domain Expert: Jessica Sinclair, PharmD
This form was used by our domain expert to verify a random subset (n=116) of the mappings between OMOP drug exposure ingredient concepts and ontology terms from CHEBI, PRO, NCBItaxon, and VO.
As described in the referenced document, the domain expert was asked to do the following when completing this task:
- Verify the
OMOPdrug exposure ingredient mappings toCHEBI,PRO,NCBItaxonandVO - For incorrect mappings, search each ingredient concept in the ontology browser to find a better mapping
- Discuss all incorrect mappings with project lead until an agreement was reached
Updates: Since the initial verification was performed, significant improvements have been made to the mapping algorithm and the way that drug mappings are conceptualized. The results of this work have been applied to update the verification spreadsheet, but the initial results remain intact.
A screenshot of the results spreadsheet (link provided in table at top of section) is provided below:
- The number of needed iterations to reach agreement on the mappings varied with:
- 67 (57.76%) requiring no iterations
- 48 (41.38%) requiring 1 iteration
- 1 (0.86%) requiring 3 iterations
- The majority of the mappings were perceived to be correct (59.60%; n=66) and 15.52% (n=18) were found to be incorrect. Just under 30% of the mappings required additional discussion of which, 16 (50%) were found to be incorrect
| Timeline | Documentation | Ontologies |
|---|---|---|
01/2019-10/2019 Perform Mapping10/2018-01/2019 Survey01/2019-03/2019 LOINC2HPO Extension |
Instructions for Experts Survey Verification Spreadsheet |
Chemical Entities of Biological Interest (ChEBI) Human Phenotype Ontology (HPO) NCBI Organism Taxonomy (NCBITaxon) Protein Ontology (PRO) Uber-Anatomy Ontology (UBERON) Cell Ontology (CL) |
Domain Experts
 |
 |
 |
 |
 |
|---|---|---|---|---|
| Tell Bennett, MD ⚕️ | James Feinstein, MD ⚕️ | Blake Martin, MD ⚕️ | Adrianne L. Stefanski, PhD 🔬 | Nicole Vasilevsky, PhD 🖥️ |
| Survey | Survey | Survey | Survey | LOINC2HPO Extension |
A subset of pediatric-specific laboratory test result mappings (n=270) were independently validated by five domain experts (i.e., three pediatric clinicians, a PhD-level molecular biologist, and a master’s-level epidemiologist). The study was approved by the Colorado Multiple Institutional Review Board (15-0445). To perform this validation, a Qualtrics survey (see QR code below) was designed so that each question featured a laboratory test description and set of reasonable HPO concepts.
After completing the survey, any laboratory test mapping that did not meet agreement by at least one clinician and both the biologist and epidemiologist were re-evaluated with one clinician until consensus was reached. These terms were additionally vetted on the loinc2hpoAnnotation GitHub tracker by the entire team of HPO biocurators.
This form was used by our domain expert to verify a random subset of the mappings between 600 OMOP lab tests (1,800) lab test result concepts and ontology terms from HPO.
As described in the referenced document, the domain expert was asked to do the following when completing this task:
- Verify the
OMOPlab test result mappings toHPO - Discuss all incorrect mappings with project lead until an agreement was reached
Updates: Since the initial verification was performed significant improvements have been made to the mapping algorithm and the way that lab test results are conceptualized. The results of this work have been applied to update the verification spreadsheet, but the initial results remain in tact. Additionally, since the verification was performed, all of the mappings have been extended to include the following additional ontologies: UBERON, CHEBI, PRO, NCBItaxon, and CL
A screenshot of the survey results (link provided in table at top of section) is provided below:
- Clinician reverification was performed for 58 terms where at least 2 clinicians did not agree
- 2 out of 3 clinicians agreed on 230 result annotations (85.19%)
- The biologist and epidemiologist agreed on 214 result annotations (79.26%)
- The biologist or epidemiologist agreed with at least 1 clinician on 251 result annotations (92.96%)
- There was a strong agreement found between 1,314 (73.00%) mappings when considering all labs
- When removing the LOINC2HPO purposefully unannotated results, agreement rose to 97.33%
