[Feature] Support MMLU-CF Benchmark

README.md

@@ -57,6 +57,7 @@ Just like a compass guides us on our journey, OpenCompass will guide you through
 
 ## 🚀 What's New <a><img width="35" height="20" src="https://user-images.githubusercontent.com/12782558/212848161-5e783dd6-11e8-4fe0-bbba-39ffb77730be.png"></a>
 
+- **\[2024.12.27\]** We now support the Microsoft's Contamination-Free Multi-task language Understanding Benchmark [MMLU-CF](https://huggingface.co/datasets/microsoft/MMLU-CF). Feel free to give it a try! 🔥🔥🔥
 - **\[2024.12.17\]** We have provided the evaluation script for the December [CompassAcademic](configs/eval_academic_leaderboard_202412.py), which allows users to easily reproduce the official evaluation results by configuring it.
 - **\[2024.11.14\]** OpenCompass now offers support for a sophisticated benchmark designed to evaluate complex reasoning skills — [MuSR](https://arxiv.org/pdf/2310.16049). Check out the [demo](configs/eval_musr.py) and give it a spin! 🔥🔥🔥
 - **\[2024.11.14\]** OpenCompass now supports the brand new long-context language model evaluation benchmark — [BABILong](https://arxiv.org/pdf/2406.10149). Have a look at the [demo](configs/eval_babilong.py) and give it a try! 🔥🔥🔥

README_zh-CN.md

@@ -57,6 +57,7 @@
 
 ## 🚀 最新进展 <a><img width="35" height="20" src="https://user-images.githubusercontent.com/12782558/212848161-5e783dd6-11e8-4fe0-bbba-39ffb77730be.png"></a>
 
+- **\[2024.12.27\]** 现已支持Microsoft去污染多任务语言理解数据集[MMLU-CF](https://huggingface.co/datasets/microsoft/MMLU-CF)，欢迎尝试! 🔥🔥🔥
 - **\[2024.12.17\]** 我们提供了12月CompassAcademic学术榜单评估脚本 [CompassAcademic](configs/eval_academic_leaderboard_202412.py)，你可以通过简单地配置复现官方评测结果。
 - **\[2024.10.14\]** 现已支持OpenAI多语言问答数据集[MMMLU](https://huggingface.co/datasets/openai/MMMLU)，欢迎尝试! 🔥🔥🔥
 - **\[2024.09.19\]** 现已支持[Qwen2.5](https://huggingface.co/Qwen)(0.5B to 72B) ，可以使用多种推理后端(huggingface/vllm/lmdeploy), 欢迎尝试! 🔥🔥🔥

configs/datasets/mmlu_cf/mmlu_cf_categories.py

@@ -0,0 +1,16 @@
+categories = [
+    'Math',
+    'Physics',
+    'Chemistry',
+    'Law',
+    'Engineering',
+    'Other',
+    'Economics',
+    'Health',
+    'Psychology',
+    'Business',
+    'Biology',
+    'Philosophy',
+    'Computer_Science',
+    'History',
+]

configs/datasets/mmlu_cf/mmlu_cf_few_shot.py

@@ -0,0 +1,64 @@
+from mmengine.config import read_base
+from opencompass.openicl.icl_prompt_template import PromptTemplate
+from opencompass.openicl.icl_retriever import FixKRetriever
+from opencompass.openicl.icl_inferencer import GenInferencer
+from opencompass.openicl.icl_evaluator import AccwithDetailsEvaluator
+from opencompass.datasets import MMLUCFDataset
+from opencompass.utils.text_postprocessors import first_option_postprocess
+
+with read_base():
+    from .mmlu_cf_categories import categories
+
+mmlu_cf_reader_cfg = dict(
+    input_columns=['input', 'A', 'B', 'C', 'D'],
+    output_column='target',
+    train_split='dev')
+
+mmlu_cf_datasets = []
+for _name in categories:
+    _hint = f'There is a single choice question (with answers). Answer the question by replying A, B, C or D.'
+    mmlu_cf_infer_cfg = dict(
+        ice_template=dict(
+            type=PromptTemplate,
+            template=dict(round=[
+                dict(
+                    role='HUMAN',
+                    prompt=
+                    f'{_hint}\nQuestion: {{input}}\nA. {{A}}\nB. {{B}}\nC. {{C}}\nD. {{D}}\nAnswer: '
+                ),
+                dict(role='BOT', prompt='{target}\n')
+            ]),
+        ),
+        prompt_template=dict(
+            type=PromptTemplate,
+            template=dict(
+                begin='</E>',
+                round=[
+                    dict(
+                        role='HUMAN',
+                        prompt=f'{_hint}\nQuestion: {{input}}\nA. {{A}}\nB. {{B}}\nC. {{C}}\nD. {{D}}\nAnswer: '
+                    ),
+                ],
+            ),
+            ice_token='</E>',
+        ),
+        retriever=dict(type=FixKRetriever, fix_id_list=[0, 1, 2, 3, 4]),
+        inferencer=dict(type=GenInferencer),
+    )
+
+    mmlu_cf_eval_cfg = dict(
+        evaluator=dict(type=AccwithDetailsEvaluator),
+        pred_postprocessor=dict(type=first_option_postprocess, options='ABCD'))
+
+    mmlu_cf_datasets.append(
+        dict(
+            abbr=f'mmlu_cf_{_name}',
+            type=MMLUCFDataset,
+            path='opencompass/mmlu_cf',
+            name=_name,
+            reader_cfg=mmlu_cf_reader_cfg,
+            infer_cfg=mmlu_cf_infer_cfg,
+            eval_cfg=mmlu_cf_eval_cfg,
+        ))
+
+del _name, _hint

configs/datasets/mmlu_cf/mmlu_cf_gen.py

@@ -0,0 +1,64 @@
+from mmengine.config import read_base
+from opencompass.openicl.icl_prompt_template import PromptTemplate
+from opencompass.openicl.icl_retriever import FixKRetriever
+from opencompass.openicl.icl_inferencer import GenInferencer
+from opencompass.openicl.icl_evaluator import AccwithDetailsEvaluator
+from opencompass.datasets import MMLUCFDataset
+from opencompass.utils.text_postprocessors import first_option_postprocess
+
+with read_base():
+    from .mmlu_cf_categories import categories
+
+mmlu_cf_reader_cfg = dict(
+    input_columns=['input', 'A', 'B', 'C', 'D'],
+    output_column='target',
+    train_split='dev')
+
+mmlu_cf_datasets = []
+for _name in categories:
+    _hint = f'There is a single choice question. Answer the question by replying A, B, C or D.'
+    mmlu_cf_infer_cfg = dict(
+        ice_template=dict(
+            type=PromptTemplate,
+            template=dict(round=[
+                dict(
+                    role='HUMAN',
+                    prompt=
+                    f'{_hint}\nQuestion: {{input}}\nA. {{A}}\nB. {{B}}\nC. {{C}}\nD. {{D}}\nAnswer: '
+                ),
+                dict(role='BOT', prompt='{target}\n')
+            ]),
+        ),
+        prompt_template=dict(
+            type=PromptTemplate,
+            template=dict(
+                begin='</E>',
+                round=[
+                    dict(
+                        role='HUMAN',
+                        prompt=f'{_hint}\nQuestion: {{input}}\nA. {{A}}\nB. {{B}}\nC. {{C}}\nD. {{D}}\nAnswer: '
+                    ),
+                ],
+            ),
+            ice_token='</E>',
+        ),
+        retriever=dict(type=FixKRetriever, fix_id_list=[0, 1, 2, 3, 4]),
+        inferencer=dict(type=GenInferencer),
+    )
+
+    mmlu_cf_eval_cfg = dict(
+        evaluator=dict(type=AccwithDetailsEvaluator),
+        pred_postprocessor=dict(type=first_option_postprocess, options='ABCD'))
+
+    mmlu_cf_datasets.append(
+        dict(
+            abbr=f'mmlu_cf_{_name}',
+            type=MMLUCFDataset,
+            path='opencompass/mmlu_cf/',
+            name=_name,
+            reader_cfg=mmlu_cf_reader_cfg,
+            infer_cfg=mmlu_cf_infer_cfg,
+            eval_cfg=mmlu_cf_eval_cfg,
+        ))
+
+del _name, _hint

configs/datasets/mmlu_cf/mmlu_cf_zero_shot.py

@@ -0,0 +1,64 @@
+from mmengine.config import read_base
+from opencompass.openicl.icl_prompt_template import PromptTemplate
+from opencompass.openicl.icl_retriever import ZeroRetriever
+from opencompass.openicl.icl_inferencer import GenInferencer
+from opencompass.openicl.icl_evaluator import AccwithDetailsEvaluator
+from opencompass.datasets import MMLUCFDataset
+from opencompass.utils.text_postprocessors import first_option_postprocess
+
+with read_base():
+    from .mmlu_cf_categories import categories
+
+mmlu_cf_reader_cfg = dict(
+    input_columns=['input', 'A', 'B', 'C', 'D'],
+    output_column='target',
+    train_split='dev')
+
+mmlu_cf_datasets = []
+for _name in categories:
+    _hint = f'There is a single choice question (with answers). Answer the question by replying A, B, C or D.'
+    mmlu_cf_infer_cfg = dict(
+        ice_template=dict(
+            type=PromptTemplate,
+            template=dict(round=[
+                dict(
+                    role='HUMAN',
+                    prompt=
+                    f'{_hint}\nQuestion: {{input}}\nA. {{A}}\nB. {{B}}\nC. {{C}}\nD. {{D}}\nAnswer: '
+                ),
+                dict(role='BOT', prompt='{target}\n')
+            ]),
+        ),
+        prompt_template=dict(
+            type=PromptTemplate,
+            template=dict(
+                begin='</E>',
+                round=[
+                    dict(
+                        role='HUMAN',
+                        prompt=f'{_hint}\nQuestion: {{input}}\nA. {{A}}\nB. {{B}}\nC. {{C}}\nD. {{D}}\nAnswer: '
+                    ),
+                ],
+            ),
+            ice_token='</E>',
+        ),
+        retriever=dict(type=ZeroRetriever),
+        inferencer=dict(type=GenInferencer),
+    )
+
+    mmlu_cf_eval_cfg = dict(
+        evaluator=dict(type=AccwithDetailsEvaluator),
+        pred_postprocessor=dict(type=first_option_postprocess, options='ABCD'))
+
+    mmlu_cf_datasets.append(
+        dict(
+            abbr=f'mmlu_cf_{_name}',
+            type=MMLUCFDataset,
+            path='opencompass/mmlu_cf',
+            name=_name,
+            reader_cfg=mmlu_cf_reader_cfg,
+            infer_cfg=mmlu_cf_infer_cfg,
+            eval_cfg=mmlu_cf_eval_cfg,
+        ))
+
+del _name, _hint

configs/summarizers/groups/mmlu_cf.py

@@ -0,0 +1,5 @@
+categories = ['Computer Science', 'Math', 'Physics', 'Chemistry', 'Law', 'Engineering', 'Other', 'Economics', 'Health', 'Psychology', 'Business', 'Biology', 'Philosophy',  'History']
+
+mmlu_cf_summary_groups = [
+    {'name': 'mmlu_cf', 'subsets': ['mmlu_cf_' + c.replace(' ', '_') for c in categories]},
+]

configs/summarizers/mmlu_cf.py

@@ -0,0 +1,25 @@
+from mmengine.config import read_base
+
+with read_base():
+    from .groups.mmlu_cf import mmlu_cf_summary_groups
+
+summarizer = dict(
+    dataset_abbrs=[
+        'mmlu_cf',
+        'mmlu_cf_Biology',
+        'mmlu_cf_Business',
+        'mmlu_cf_Chemistry',
+        'mmlu_cf_Computer_Science',
+        'mmlu_cf_Economics',
+        'mmlu_cf_Engineering',
+        'mmlu_cf_Health',
+        'mmlu_cf_History',
+        'mmlu_cf_Law',
+        'mmlu_cf_Math',
+        'mmlu_cf_Philosophy',
+        'mmlu_cf_Physics',
+        'mmlu_cf_Psychology',
+        'mmlu_cf_Other',
+    ],
+    summary_groups=sum([v for k, v in locals().items() if k.endswith('_summary_groups')], []),
+)

opencompass/configs/datasets/mmlu_cf/mmlu_cf_categories.py

@@ -0,0 +1,16 @@
+categories = [
+    'Math',
+    'Physics',
+    'Chemistry',
+    'Law',
+    'Engineering',
+    'Other',
+    'Economics',
+    'Health',
+    'Psychology',
+    'Business',
+    'Biology',
+    'Philosophy',
+    'Computer_Science',
+    'History',
+]

opencompass/configs/datasets/mmlu_cf/mmlu_cf_few_shot.py

@@ -0,0 +1,64 @@
+from mmengine.config import read_base
+from opencompass.openicl.icl_prompt_template import PromptTemplate
+from opencompass.openicl.icl_retriever import FixKRetriever
+from opencompass.openicl.icl_inferencer import GenInferencer
+from opencompass.openicl.icl_evaluator import AccwithDetailsEvaluator
+from opencompass.datasets import MMLUCFDataset
+from opencompass.utils.text_postprocessors import first_option_postprocess
+
+with read_base():
+    from .mmlu_cf_categories import categories
+
+mmlu_cf_reader_cfg = dict(
+    input_columns=['input', 'A', 'B', 'C', 'D'],
+    output_column='target',
+    train_split='dev')
+
+mmlu_cf_datasets = []
+for _name in categories:
+    _hint = f'There is a single choice question (with answers). Answer the question by replying A, B, C or D.'
+    mmlu_cf_infer_cfg = dict(
+        ice_template=dict(
+            type=PromptTemplate,
+            template=dict(round=[
+                dict(
+                    role='HUMAN',
+                    prompt=
+                    f'{_hint}\nQuestion: {{input}}\nA. {{A}}\nB. {{B}}\nC. {{C}}\nD. {{D}}\nAnswer: '
+                ),
+                dict(role='BOT', prompt='{target}\n')
+            ]),
+        ),
+        prompt_template=dict(
+            type=PromptTemplate,
+            template=dict(
+                begin='</E>',
+                round=[
+                    dict(
+                        role='HUMAN',
+                        prompt=f'{_hint}\nQuestion: {{input}}\nA. {{A}}\nB. {{B}}\nC. {{C}}\nD. {{D}}\nAnswer: '
+                    ),
+                ],
+            ),
+            ice_token='</E>',
+        ),
+        retriever=dict(type=FixKRetriever, fix_id_list=[0, 1, 2, 3, 4]),
+        inferencer=dict(type=GenInferencer),
+    )
+
+    mmlu_cf_eval_cfg = dict(
+        evaluator=dict(type=AccwithDetailsEvaluator),
+        pred_postprocessor=dict(type=first_option_postprocess, options='ABCD'))
+
+    mmlu_cf_datasets.append(
+        dict(
+            abbr=f'mmlu_cf_{_name}',
+            type=MMLUCFDataset,
+            path='opencompass/mmlu_cf',
+            name=_name,
+            reader_cfg=mmlu_cf_reader_cfg,
+            infer_cfg=mmlu_cf_infer_cfg,
+            eval_cfg=mmlu_cf_eval_cfg,
+        ))
+
+del _name, _hint