This repo provides a test to measure language models' ability to answer medical questions in Chinese. (Both Traditional and Simplified)
EMPC contains the recent 10 years of test questions of Taiwan's Professional and Technical Examinations for Medical Personnel.
We collect the tests for various medical professionals such as Medical Technologist, Medical Radiation Technologist, Registered Professional Nurse, Physical Therapist et. al. There are in total of 81761 single-choice questions covering a wide range of subjects including General Clinical Psychology, Anatomy and Physiology, Fundamentals of Respiratory Care, and Occupational Therapy Techniques et.al.
We tested the 2023 first-time examination set (2840) on multiple LLMs with zero-shot the results are shown as follows:
| Model | Traditional | Zero-Shot |
|---|---|---|
| GPT3.5-Turbo | No | 52.25% |
| GPT4 | No | 65.46% |
| Llama-70b-Chat | No | 33.17% |
EMPC forms a remarkable challenge for AI models and can serve as an effective tool to evaluate models' medical knowledge encoded in Chinese. We hope EMPC could support the exploration and building of Large Multi-lingual or Chinese Language Models, especially in the medical domain.
Our dataset can be downloaded from Google Drive and 百度网盘.
The distribution of the questions over subjects is shown here
.
And the distribution over professions is as follows:
More details can be found in the upcoming report for this test.
If you find EMPEC useful, please consider citing us.
@misc{EMPEC,
title={EMPEC, Examinations-for-Medical-PErsonnel-in-Chinese},
author={Zheheng Luo},
year = {2024},
publisher = {GitHub},
journal = {GitHub repository},
howpublished = {\url{https://github.com/zhehengluoK/Examinations-for-Medical-PErsonnel-in-Chinese}},
}