Indication as Prior Knowledge for Multimodal Disease Classification in Chest Radiographs with Transformers

Code for the paper:

Jacenków, G., O’Neil, A.Q. and Tsaftaris, S.A., 2022, March. Indication as Prior Knowledge for Multimodal Disease Classification in Chest Radiographs with Transformers. In 2022 IEEE 19th International Symposium on Biomedical Imaging (ISBI). IEEE.

Requirements

The project was developed in Python 3.7.9, PyTorch 1.8.2 with CUDA 11.1 acceleration. We use the MMF framework, which provides a boilerplate code for data loaders and common visual-linguistic models. Please, follow the MMF website on how to install the framework.

We use the MIMIC-CXR dataset for our experiments. Due to the licensing agreement, we cannot share the images and radiology reports. Please, refer to the PhysioNet website for details on how to acquire the dataset. We recommend downloading the MIMIC-CXR-JPG dataset to avoid additional pre-processing of DICOM images.

Setup

We expect the images in JPG format to be placed in dataset/mimic_cxr/subjects folder. Please, follow the folder to see an example. All textual information, i.e., corresponding radiology reports, ground truth labels, subject and study IDs, are stored as jsonl files in the dataset/mimic_cxr/annotations folder. We expect three files to run the experiments, i.e., test.jsonl, training.jsonl, and val.jsonl. Please, see the example.

Tools

We add several tools to ease the preprocessing and training processes, i.e.,

* image_preprocessing.py - resize all images to 224 x 224 size.
* mimic_cxr_preprocess.py - extract indication/history fields from the full radiology reports.
* precalculate_embeddings.py - precalculate BioWordVec embeddings to avoid bottlenecking during training.

First run

Please, follow the commands to run the experiments (training and evaluation).

pip install -r requirements.txt  # run once, after installing the MMF framework.
pip install -e .  # run once, install the project as a Python package.
./run.sh  # run training and evaluation protocol.

Evaluation

The evaluation protocol will store predictions as json files. Please, use the Jupyter notebooks to evaluate the predictions with the aforementioned files.