omop2obo is a collection of health system-wide, disease-agnostic mappings between standardized clinical terminologies in the Observational Medical Outcomes Partnership (OMOP) common data model and several Open Biomedical Ontologies (OBOs) foundry ontologies.

Motivation

Common data models have solved many challenges of utilizing electronic health records, but have not yet meaningfully integrated clinical and molecular data. Aligning clinical data to open biological ontologies (OBOs), which provide semantically computable representations of biological knowledge, requires extensive manual curation and expertise.

Objective

To address these limitations, we have developed OMOP2OBO, the first health system-wide integration and alignment between the Observational Health Data Sciences and Informatics' Observational Medical Outcomes Partnership (OMOP) standardized clinical terminologies and eight OBO biomedical ontologies spanning diseases, phenotypes, anatomical entities, cell types, organisms, chemicals, metabolites, hormones, vaccines, and proteins. To verify that the mappings are both clinically and biologically meaningful, we have performed extensive experiments to verify the accuracy, generalizability, and logical consistency of each released mapping set.

Through this repository we provide the following:

1. Mappings: A free set of omop2obo mappings that can be used out of the box (requires no coding) covering OMOP Conditions, Drug Exposures, and Measurements. These mappings are available in several formats including: .txt, .xlsx, and .dump. We also provide a semantic representation of the mappings, integrated with the OBO biomedical ontologies, available as an edge list (.txt) and as an .owl file. See current release for more details.
2. A Mapping Framework: An algorithm and mapping pipeline that enables one to construct their set of omop2obo mappings. The figure below provides a high-level overview of the algorithm workflow. The code provided in this repository facilitates all of the automatic steps shown in this figure except for the manual mapping (for now, although we are currently working on a deep learning model to address this).

• Join an existing or start a new Discussion
• The Project Wiki for more details on the omop2obo mappings, algorithm, and information on the experiments we ran to ensure each mapping set released is accurate, generalizable, and consistent!

• All code and mappings for each release are free to download, see Wiki
• We are working on a publishing this work, but if you'd like to use the validated mappings immediately, please send us an email.

Current Release:

Release currently in preparation and planned for *December 2020*! - v1.0.0 ➞ data and code can be directly downloaded here.

NOT YET AVAILABLE - COMING SOON!!

This program requires Python version 3.6. To install the library from PyPI, run:

pip install omop2obo

You can also clone the repository directly from GitHub by running:

git clone https://github.com/callahantiff/OMOP2OBO.git

The omop2obo library requires a specific project directory structure. Please make sure that your project directory includes the following sub-directories:

OMOP2OBO/
    |
    |---- resources/
    |         |
    |     clinical_data/
    |         |
    |     mappings/
    |         |
    |     ontologies/

Results will be output to the mappings directory.

APPLICATIONS

• This software also relies on OWLTools. If cloning the repository, the owltools library file will automatically be included and placed in the correct repository.
• The National of Library Medicine's Unified Medical Language System (UMLS) MRCONSO and MRSTY. Using these data requires a license agreement. Note that in order to get the MRSTY file you will need to download the UMLS Metathesaurus and run MetamorphoSys. Once both data sources are obtained, please place the files in the resources/mappings directory.

DATA

• Clinical Data: This repository assumes that the clinical data that needs mapping has been placed in the resources/clinical_data repository. Each data source provided in this repository is assumed to have been extracted from the OMOP CDM. An example of what is expected for this input can be found here.
• Ontology Data: Ontology data is automatically downloaded from the user provided input file ontology_source_list.txt (here).
• Vocabulary Source Code Mapping: To increase the likelihood of capturing existing database cross-references, omop2obo provides a file that maps different clinical vocabulary source code prefixes between the UMLS, ontologies, and clinical EHR data (i.e. "SNOMED", "SNOMEDCT", "SNOMEDCT_US") source_code_vocab_map.csv (here). Please note this file builds off of these UMLS provided abbreviation mappings. Currently, this file is updated for ontologies released july 2020, clinical data normlaized to OMOP_v5.0, and UMLS 2020AA.
• Semantic Mapping Representation: In order to create a semantic representation of the omop2obo mappings, an ontological specification for creating classes that span multiple ontologies (reosurces/mapping_semantics/omop2obo). This document only needs to be altered if you plan to utilize the semantic mapping transformation algorithm and want to use a different knowledge representation. Please the following README for additional details on these resources.

There are a few ways to run omop2obo. An example workflow is provided below.

import glob
import pandas as pd
import pickle

from datetime import date, datetime

from omop2obo import ConceptAnnotator, OntologyDownloader, OntologyInfoExtractor, SimilarStringFinder


# set some global variables
outfile = 'resources/mappings/OMOP2OBO_MAPPED_'
date_today = '_' + datetime.strftime(datetime.strptime(str(date.today()), '%Y-%m-%d'), '%d%b%Y').upper()

# download ontologies
ont = OntologyDownloader('resources/ontology_source_list.txt')
ont.downloads_data_from_url()

# process ontologies
ont_explorer = OntologyInfoExtractor('resources/ontologies', ont.data_files)
ont_explorer.ontology_processor()

# create master dictionary of processed ontologies
ont_explorer.ontology_loader()

# read in ontology data
with open('resources/ontologies/master_ontology_dictionary.pickle', 'rb') as handle:
    ont_data = pickle.load(handle)
handle.close()

# process clinical data
mapper = ConceptAnnotator(clinical_file='resources/clinical_data/omop2obo_conditions_june2020.csv',
                          ontology_dictionary={k: v for k, v in ont_data.items() if k in ['hp', 'mondo']},
                          merge=True,
                          primary_key='CONCEPT_ID',
                          concept_codes=tuple(['CONCEPT_SOURCE_CODE']),
                          concept_strings=tuple(['CONCEPT_LABEL', 'CONCEPT_SYNONYM']),
                          ancestor_codes=tuple(['ANCESTOR_SOURCE_CODE']),
                          ancestor_strings=tuple(['ANCESTOR_LABEL']),
                          umls_mrconso_file=glob.glob('resources/mappings/*MRCONSO*')[0] if len(glob.glob('resources/mappings/*MRCONSO*')) > 0 else None,
                          umls_mrsty_file=glob.glob('resources/mappings/*MRCONSO*')[0] if len(glob.glob('resources/mappings/*MRCONSO*')) > 0 else None)

   exact_mappings = mapper.clinical_concept_mapper()
   exact_mappings.to_csv(outfile + 'CONDITIONS' + date_today + '.csv', sep=',', index=False, header=True)
   # get column names -- used later to organize output
   start_cols = [i for i in exact_mappings.columns if not any(j for j in ['STR', 'DBXREF', 'EVIDENCE'] if j in i)]
   exact_cols = [i for i in exact_mappings.columns if i not in start_cols]

   # perform similarity mapping
   if tfidf_mapping is not None:
       sim = SimilarStringFinder(clinical_file=outfile + 'CONDITIONS' + date_today + '.csv',
                                 ontology_dictionary={k: v for k, v in ont_data.items() if k in ['hp', 'mondo']},
                                 primary_key='CONCEPT_ID',
                                 concept_strings=tuple(['CONCEPT_LABEL', 'CONCEPT_SYNONYM']))

       sim_mappings = sim.performs_similarity_search()
       sim_mappings = sim_mappings[['CONCEPT_ID'] + [x for x in sim_mappings.columns if 'SIM' in x]].drop_duplicates()
       # get column names -- used later to organize output
       sim_cols = [i for i in sim_mappings.columns if not any(j for j in start_cols if j in i)]

       # merge dbXref, exact string, and TF-IDF similarity results
       merged_scores = pd.merge(exact_mappings, sim_mappings, how='left', on='CONCEPT_ID')
       # re-order columns and write out data
       merged_scores = merged_scores[start_cols + exact_cols + sim_cols]
       merged_scores.to_csv(outfile + clinical_domain.upper() + date_today + '.csv', sep=',', index=False, header=True)

COMMAND LINE ➞ main.py

python main.py --help
Usage: main.py [OPTIONS]

The OMOP2OBO package provides functionality to assist with mapping OMOP standard clinical terminology
concepts to OBO terms. Successfully running this program requires several input parameters, which are
specified below:


PARAMETERS:
    ont_file: 'resources/oontology_source_list.txt'
    tfidf_mapping: "yes" if want to perform cosine similarity mapping using a TF-IDF matrix.
    clinical_domain: clinical domain of input data (i.e. "conditions", "drugs", or "measurements").
    merge: A bool specifying whether to merge UMLS SAB codes with OMOP source codes once or twice.
    onts: A comma-separated list of ontology prefixes that matches 'resources/oontology_source_list.txt'.
    clinical_data: The filepath to the clinical data needing mapping.
    primary_key: The name of the file to use as the primary key.
    concept_codes: A comma-separated list of concept-level codes to use for DbXRef mapping.
    concept_strings: A comma-separated list of concept-level strings to map to use for exact string mapping.
    ancestor_codes: A comma-separated list of ancestor-level codes to use for DbXRef mapping.
    ancestor_strings: A comma-separated list of ancestor-level strings to map to use for exact string mapping.
    outfile: The filepath for where to write output data to.

Several dependencies must be addressed before running this file. Please see the README for instructions.

Options:
  --ont_file PATH          [required]
  --tfidf_mapping TEXT     [required]
  --clinical_domain TEXT   [required]
  --merge                  [required]
  --ont TEXT               [required]
  --clinical_data PATH     [required]
  --primary_key TEXT       [required]
  --concept_codes TEXT     [required]
  --concept_strings TEXT
  --ancestor_codes TEXT
  --ancestor_strings TEXT
  --outfile TEXT           [required]
  --help                   Show this message and exit.

If you follow the instructions for how to format clinical data (here) and/or if taking the data that results from running our queries here), omop2obo can be run with the following call on the command line (with minor updates to the csv filename):

python main.py --clinical_domain condition --onts hp --onts mondo --clinical_data resources/clinical_data/omop2obo_conditions_june2020.csv

JUPYTER NOTEBOOK ➞ omop2obo_notebook.ipynb

Please read CONTRIBUTING.md for details on our code of conduct, and the process for submitting pull requests to us.

This project is licensed under MIT - see the LICENSE.md file for details.

@software{callahan_tiffany_j_2020_3902767,
          author     =  {Callahan, Tiffany J},
          title      = {OMOP2OBO},
          month      = jun,
          year       = 2020,
          publisher  = {Zenodo},
          version    = {v1.0.0},
          doi        = {10.5281/zenodo.3902767},
          url        = {https://doi.org/10.5281/zenodo.3902767}.
   }

We’d love to hear from you! To get in touch with us, please join or start a new Discussion, create an issue or send us an email 💌