Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add dataset collection #253

Open
wants to merge 16 commits into
base: main
Choose a base branch
from
Open

Add dataset collection #253

wants to merge 16 commits into from

Conversation

Yunnglin
Copy link
Collaborator

@Yunnglin Yunnglin commented Dec 18, 2024

  1. 注册数据集:
    • race
    • trivia_qa 不支持service
    • truthful_qa
    • mmlu
    • humaneval
    • general_qa
    • cmmlu
    • arc
    • hellaswag 不支持 service
    • bbh
    • ceval
    • gsm8k
    • competition_math
  2. 支持指定URL模型服务评测
  3. 支持数据混合评测,自定义collection schema
schema = CollectionSchema(name='math&reasoning', datasets=[
            CollectionSchema(name='math', datasets=[
                    DatasetInfo(name='gsm8k', weight=1, task_type='math', tags=['en', 'math']),
                    DatasetInfo(name='competition_math', weight=1, task_type='math', tags=['en', 'math']),
                    DatasetInfo(name='cmmlu', weight=2, task_type='math', tags=['zh', 'math'], args={'subset_list': ['college_mathematics', 'high_school_mathematics']}),
                    DatasetInfo(name='ceval', weight=3, task_type='math', tags=['zh', 'math'], args={'subset_list': ['advanced_mathematics', 'high_school_mathematics', 'discrete_mathematics', 'middle_school_mathematics']}),
            ]),
            CollectionSchema(name='reasoning', datasets=[
                    DatasetInfo(name='arc', weight=1, task_type='reasoning', tags=['en', 'reasoning']),
                    DatasetInfo(name='ceval', weight=1, task_type='reasoning', tags=['zh', 'reasoning'], args={'subset_list': ['logic']}),
                    DatasetInfo(name='race', weight=1, task_type='reasoning', tags=['en', 'reasoning']),
            ]),
        ])



task_cfg = TaskConfig(
    model='qwen2.5',
    api_url='http://127.0.0.1:8801/v1/chat/completions',
    api_key='EMPTY',
    eval_type=EvalType.SERVICE,
    datasets=['data_collection'],
    dataset_args={'data_collection': {
        'local_path': 'outputs/mixed_data_test.jsonl'
    }},
)
run_task(task_cfg=task_cfg)

输出:

task_type dataset_name subset_name average_score count
math ceval advanced_mathematics 0.25 12
math ceval discrete_mathematics 0.333333 3
math ceval high_school_mathematics 0 3
math ceval middle_school_mathematics 0 3
math cmmlu college_mathematics 0.2 5
math cmmlu high_school_mathematics 0.555556 9
math competition_math default 0 7
math gsm8k main 0.428571 7
reasoning arc ARC-Challenge 0.166667 6
reasoning arc ARC-Easy 0.5 10
reasoning ceval logic 0.25 16
reasoning race high 0.285714 14
reasoning race middle 0.8 5

@Yunnglin Yunnglin changed the title [WIP] Add dataset collection Add dataset collection Dec 24, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant