SemCat-CECCT

Semantic categorization of Chinese eligibility criteria in clinical trials using machine learning methods.

data collection

We downloaded the clinical trials registration files from website of Chinese Clinical Trials Registry (ChiCTR), and extracted the eligibility criteria text, including Chinese inclusion criteria(CIC), English inclusion criteria(EIC), Chinese exclusion criteria(CEC) and English exclusion criteria(EEC). An eligibility criteria example (registration number: ChiCTR1800016069) shows below:

Chinese inclusion criteria	English inclusion criteria
1）首次于本中心行肝癌切除术，术后组织病理学证实HCC； 2）术前血清HBsAg（+）； 3）血清HBV-DNA低于检测下限（内标法）； 4）肿瘤未侵犯门静脉、肝静脉或胆管的主要分支； 5）肝功能Child-Pugh A或B级； 6）年龄18～65岁，性别不限； 7）单发肿瘤者，最大径≤5cm；多发肿瘤者，瘤体数≤3个且各瘤体最大径≤3cm；	1. patient receives hepatectomy of HCC, which was confirmed with pathology; 2. serum HBsAg（+）is confirmed preoperatively; 3. serum HBV-DNA is lower than minimum of detection; 4. tumor did not invade potal vein,hepatic vein or major branch of biliary tract; 5. liver function with Child-Pugh A or B; 6. aged from 18-65 years male or female; 7. single tumor≤5cm.for multiple tumors,number of tumor≤3 and every tumor≤3cm.
Chinese exclusion criteria	English exclusion criteria
1）肝细胞癌合并肝外转移； 2) 近1年内有抗病毒治疗； 3）合并其他部位恶性肿瘤，或其他器官功能衰竭； 4）合并其它肝炎病毒重叠感染； 5）术前接受TACE或其他抗肿瘤治疗； 6）肝功能Child-Pugh C级 7）存在顽固性腹水或肝性脑病； 8）因精神病、心理性疾病或其他原因不能配合治疗或不能完成疗程者。	1. HCC with metastasis out of the liver; 2. antiviral therapy to HBV within one year; 3. with other malignancy or organ failure; 4. combined with other hepatic virus infection; 5. recieved TACE or other antitumor therapy; 6. liver function with Child-Pugh C; 7. refractory Ascites or hepatic encephalopathy; 8. patient who can not cooperate due to mental diseases or other reasons.

The eligibility criteria text is organized with multiple sentences. After sentence segmentation and filter, we randomly selected 19185 sentences used in the unsupervised hierarchical clustering section for criteria categories induction, and 38341 sentences used in supervised criteria classification section for the classification capacity validation of induced semantic categories.

clustering

We used the agglomerative hierarchical clustering algorithm, works in a “bottom-up” manner, to cluster constructed semantic feature matrix and generated clusters based on criteria sentences similarity..

1. get 19185 Chinese-English eligibility criteria sentences pairs.
2. get UMLS semantic types of English eligibility criteria sentences using MetaMap.
3. convert criteria sentences to feature matrix based on UMLS semantic types.
4. perform hierarchical clustering and summarize semantic categories.

classification

To assess the classification capacity of our induced semantic categories used for the Chinese eligibility criteria sentences classification, we randomly selected 38341 Chinese eligibility criteria sentences and implemented 9 classification algorithms.

• Machine learning algorithms
  • NB
  • kNN
  • LR
  • SVM
• Deep learning algorithms
  • CNN
  • RNN
  • FastText
• Pre-trained language models
  • BERT
  • ERNIE

classification results:

Models		Macro-average			Micro-average
		precision	recall	F1-score	precision	recall	F1-score
Machine learning algorithms	NB	0.5398	0.7403	0.5965	0.6312	0.6312	0.6312
	kNN	0.7531	0.6693	0.6948	0.7632	0.7632	0.7632
	LR	0.8017	0.7574	0.7732	0.8173	0.8173	0.8173
	SVM	0.8196	0.7712	0.7899	0.8293	0.8293	0.8293
Deep learning algorithms	CNN	0.8004	0.6951	0.7258	0.8142	0.8142	0.8142
	RNN	0.7837	0.6925	0.7170	0.8138	0.8138	0.8138
	FastText	0.7645	0.7188	0.7341	0.8182	0.8182	0.8182
Pre-trained language models	BERT	0.7994	0.8023	0.7958	0.8447	0.8447	0.8447
	ERNIE	0.7964	0.8074	0.7980	0.8484	0.8484	0.8484

we also organized a shared task on fifth China Conference on Health Information Processing (CHIP 2019). As organizers, we released our labeled data and defined 44 categories. A total 75 teams participated in the task and 27 of them submitted results. The best performing system achieved a macro F1 score of 0.81 by applied multiple pre-trained language models and ensemble modeling.

How to cite

Zong, H., Yang, J., Zhang, Z. et al. Semantic categorization of Chinese eligibility criteria in clinical trials using machine learning methods. BMC Med Inform Decis Mak 21, 128 (2021). https://doi.org/10.1186/s12911-021-01487-w

Contacts

Zong Hui , zonghui@tongji.edu.cn

Tongji University, Shanghai, 200092, China