Skip to content

Code and dataset for our Bioinformatics 2022 paper: "A Benchmark for Automatic Medical Consultation System: Frameworks, Tasks and Datasets"

Notifications You must be signed in to change notification settings

lemuria-wchen/imcs21

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

30 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

IMCS-21

This repo contains a new corpus benchmark called IMCS-21 for automated medical consultation system, as well as the code for reproducing the experiments in our Bioinformatics 2022 paper A Benchmark for Automatic Medical Consultation System: Frameworks, Tasks and Datasets.

News

TODO

  • Update the results of dev set for DDP task
  • Detailed documents of instruction on DDP task

Overview of Experiments

We provide the code for most of the baseline models, all of which are based on python 3, and we provide the environment and running procedure for each baseline.

The baseline includes:

Note:

  1. The results reported on the github are slightly different from those reported in the paper. This is because we retrained the model. For fair comparison, readers are recommended to either compare the results in the paper or compare the results reported in this document.
  2. If you need to compare the results on the dev set, these results are only available in this document.

Results of NER Task

To evaluate NER task, we use two types of metrics, entity-level and token-level. Due to space limitations, we only keep the results of token-level metrics in our paper.

For entity-level, we report token-level F1 score for each entity category, as well as the overall F1 score, following the setting of CoNLL-2003.

For token-level, we report the Precision, Recall and F1 score (micro).

The follow baseline codes are available:

Models Split Entity-Level Token-Level
SX DN DC EX OP Overall P R F1
Lattice LSTM Dev 90.61 88.12 90.89 90.44 91.14 90.33 89.62 91.00 90.31
Test 90.00 87.84 91.32 90.55 93.42 90.10 89.37 90.84 90.10
BERT Dev 91.15 89.74 90.97 90.74 92.57 90.95 88.99 92.43 90.68
Test 90.59 89.97 90.54 90.48 94.39 90.64 88.46 92.35 90.37
ERNIE Dev 91.28 89.68 90.92 91.15 92.65 91.08 89.36 92.46 90.88
Test 90.67 89.89 90.73 90.97 94.33 90.78 88.87 92.27 90.53
FLAT Dev 90.90 89.95 90.64 90.58 93.14 90.80 88.89 92.23 90.53
Test 90.45 89.67 90.35 91.12 93.47 90.58 88.76 92.07 90.38
LEBERT Dev 92.61 90.67 90.71 92.39 92.30 92.11 86.95 93.05 89.90
Test 92.14 90.31 91.16 92.35 93.94 91.92 86.53 92.91 89.60

Results of DAC Task

To evaluate DAC task, we report the Precision, Recall, F1 score (macro), as well as Accuracy.

The follow baseline codes are available:

Models Split P R F1 Acc
TextCNN Dev 73.09 70.26 71.26 77.77
Test 74.02 70.92 72.22 78.99
TextRNN Dev 74.02 68.43 70.71 78.14
Test 73.07 69.88 70.96 78.53
TextRCNN Dev 71.43 72.68 71.50 77.67
Test 73.82 72.53 72.89 79.40
DPCNN Dev 70.10 70.91 69.85 77.14
Test 74.30 69.45/td> 71.28 78.75
BERT Dev 75.19 76.31 75.66 81.00
Test 75.53 77.24 76.28 81.65
ERNIE Dev 76.04 76.82 76.37 81.60
Test 75.35 77.16 76.14 81.62

Error Analysis of DAC Task

The visualization of the classification confusion matrix predicted by ERNIE model on the test set is demonstrated in the below figure. It can be seen that there are few classification errors in most utterance categories, except for OTHER category.

Confusion

Results of SLI Task

For the evaluation of SLI-EXP and SLI-IMP task, there is a little difference since SLI-IMP task needs to additionally predict the symptom label. We divide the evaluation process into two steps, the first step is to evaluate the performance of symptom recognition, and the second step is to evaluate the performance of symptom (label) inference.

For symptom recognition, it cares only whether the symptom entities are identified or not. We use metrics of multi-label classification that are widely explored in the paper A Unified View of Multi-Label Performance Measures. It includes example-based metrics: Subset Accuracy (SA), Hamming Loss (HL), Hamming Score (HS), and label-based metrics: Precision (P), Recall (R) and F1 score (F1) (micro).

For symptom label inference, it evaluates only on those symptoms that are correctly identified, about whether their label is correct or not. We report the F1 score for each symptom label (Positive, Negative and Not sure), as well as the overall F1 score (macro) and the accuracy.

The follow baseline codes are available:

NOTE: BERT-MLC are valid for SLI-EXP and SLI-IMP tasks, while BERT-MTL is valid only for SLI-IMP task.

Models Split Example-based Label-based
SA HL HS P R F1
SLI-EXP (Symptom   Recognition)
BERT-MLC Dev 75.63 10.12 86.53 86.50 93.80 90.00
Test 73.24 10.10 84.58 86.33 93.14 89.60
SLI-IMP (Symptom   Recognition)
BERT-MLC Dev 33.61 40.87 81.34 85.03 95.40 89.91
Test 34.16 39.52 82.22 84.98 94.81 89.63
BERT-MTL Dev 36.61 38.12 84.33 95.83 86.67 91.02
Test 35.88 38.77 83.76 96.11 86.18 90.88
SLI-IMP (Symptom   Inference)
POS NEG NS Overall Acc
BERT-MLC Dev 81.85 47.99 58.42 62.76 72.84
Test 81.25 46.53 59.14 62.31 71.99
BERT-MTL Dev 79.83 53.38 60.94 64.72 71.38
Test 79.64 53.87 60.20 64.57 71.08

Results of MRG Task

In MRG task, we use the concatenation of all NON-OTHER categories of utterances to generate medical reports. During inference, the categories of utterances in the test set is pre-predicted by the trained ERNIE model of DAC task.

To evaluate MRG task, we report both BLEU and ROUGE score, i.e., BLEU-2/4 and ROUGE-1/2/L. We also report Concept F1 score (C-F1) to measure the model’s effectiveness in capturing the medical concepts that are of importance, and Regex-based Diagnostic Accuracy (RD-Acc), to measure the model’s ability to judge the disease.

The follow baseline codes are available:

NOTE: To calculate C-F1 score, the trained BERT model in NER task is utilized. See details in eval_ner_f1.py. To calculate RD-Ac, a simple regex-based method is utilized, see details in eval_acc.py.

Models B-2 B-4 R-1 R-2 R-L C-F1 D-Acc
Seq2Seq 54.43 43.95 54.13 43.98 50.42 36.73 48.34
PG 58.31 49.31 59.46 49.79 56.34 46.36 56.60
Transformer 58.57 47.67 57.25 46.29 53.29 40.64 54.50
T5 62.57 52.48 61.20 50.98 58.18 46.55 47.60
ProphetNet 58.11 49.06 61.18 50.33 57.94 49.61 55.36

Results of DDP Task

To evaluate DDP task, we report Symptom Recall (Rec), Diagnostic Accuracy (Acc) and the average number of interactions (# Turns).

The follow baseline codes are available:

NOTE: We use the open source implementation for all baselines, since none of these papers provide any official repos or codes.

Models Rec Acc # Turns
DQN 0.047 0.408 9.750
REFUEL 0.262 0.505 5.500
KR-DQN 0.279 0.485 5.950
GAMP 0.067 0.500 1.780
HRL 0.295 0.556 6.990

How to Cite

If you extend or use this work, please cite the paper where it was introduced.

@article{10.1093/bioinformatics/btac817,
    author = {Chen, Wei and Li, Zhiwei and Fang, Hongyi and Yao, Qianyuan and Zhong, Cheng and Hao, Jianye and Zhang, Qi and Huang, Xuanjing and Peng, Jiajie and Wei, Zhongyu},
    title = "{A Benchmark for Automatic Medical Consultation System: Frameworks, Tasks and Datasets}",
    journal = {Bioinformatics},
    year = {2022},
    month = {12},
    abstract = "{In recent years, interest has arisen in using machine learning to improve the efficiency of automatic medical consultation and enhance patient experience. In this article, we propose two frameworks to support automatic medical consultation, namely doctor-patient dialogue understanding and task-oriented interaction. We create a new large medical dialogue dataset with multi-level fine-grained annotations and establish five independent tasks, including named entity recognition, dialogue act classification, symptom label inference, medical report generation and diagnosis-oriented dialogue policy.We report a set of benchmark results for each task, which shows the usability of the dataset and sets a baseline for future studies.Both code and data is available from https://github.com/lemuria-wchen/imcs21.Supplementary data are available at Bioinformatics online.}",
    issn = {1367-4803},
    doi = {10.1093/bioinformatics/btac817},
    url = {https://doi.org/10.1093/bioinformatics/btac817},
    note = {btac817},
    eprint = {https://academic.oup.com/bioinformatics/advance-article-pdf/doi/10.1093/bioinformatics/btac817/48290490/btac817.pdf},
}

About

Code and dataset for our Bioinformatics 2022 paper: "A Benchmark for Automatic Medical Consultation System: Frameworks, Tasks and Datasets"

Topics

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published