CDEvalSumm: Cross-Dataset Evaluation for Summarization

Descriptions and metrics code for EMNLP2020 findings paper:

CDEvalSumm: An Empirical Study of Cross-Dataset Evaluation for Neural Summarization Systems

(Yiran Chen*, Pengfei Liu*, Ming Zhong, Zi-Yi Dou, Danqing Wang, Xipeng Qiu, Xuanjing Huang)

Motivation

Many work evaluate summarization systems on in-domain setting (the model is trained and tested on the same dataset). In this work we try to understand model performance on different perspectives on a cross-dataset setting. The picture blow represents the main motivation (summarization systems get different rankings when evaluated under different measures where abstractive models are red while extractive ones are blue):

Two Research Questions

Q1: How do different neural architectures of summarizers influence the cross-dataset generalization performances?
Q2: Do different generation ways (extractive and abstractive) of summarizers influence the cross-dataset generalization ability?

Evaluation Systems

• Extractive summarizers:

• Abstractive summarizers:

Systems		Paper	Bib
Abs-Sum	LSTM_{non}	Content Selection in Deep Learning Models of Summarization	Bib
	Trans_non	Text Summarization with Pretrained Encoders	Bib
	Trans_{auto}	Searching for Effective Neural Extractive Summarization: What works and What’s Next	Bib
	BERT_{non}	Text Summarization with Pretrained Encoders	Bib
	BERT_{match}	Extractive Summarization as Text Matching	Bib
Ext-Sum	L2L^{cov}_{ptr}	Get to the point: Summarization with Pointer-Generator Networks	Bib
	L2L_{ptr}	Get to the point: Summarization withpointer-generator networks	Bib
	L2L	CDEvalSumm: An Empirical Study of Cross-Dataset Evaluationfor Neural Summarization Systems	Bib
	T2T	Text Summarization with Pretrained Encoders	Bib
	BE2T	Text Summarization with Pretrained Encoders	Bib
	BART	Bart: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension	Bib

Datasets

• Datasets
  • CNN/Dailymail
  • Xsum
  • Pubmed
  • Bigpatent B
  • Reddit TIFU

Evaluation Metrics

• Semantic Equivalenc (ROUGE)
• Factuality (Factcc)
• Dataset bias (Detailed explanation is displayed in our paper and the code can refer to Data-bias-metrics/)
  • Coverage
  • Copy length
  • Novelty
  • Repetition
  • Sentence fusion score

Cross-dataset Measures

• Stiffness
  
  
  : the metric score when model is trained on dataset i and tested on dataset j.
• Stableness
  
  
  : the metric score when model is trained on dataset i and tested on dataset j.

Experiment Results

The stiffness and stableness of various summarizers are displayed below. For fine-grained results and comprehensive analysis please refer to the paper.