Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MTBenchEvaluator and MMLUEvaluator should be/have static methods #26

Open
Tracked by #31 ...
JamesKunstle opened this issue Jun 26, 2024 · 2 comments
Open
Tracked by #31 ...
Labels
mmlu Pertains to MMLU mtbench Pertains to MTBench

Comments

@JamesKunstle
Copy link
Contributor

Evaluator objects shouldn't be reused- once we've evaluated a checkpoint or model, we want to save the score and move on to the next. This motivates a reasonable design change, implementing something like:

class MMLUEvaluator(Evaluator):

    def __init__(self):
        # optional empty initialization 
        ...

    def run(self):
       ...

    @staticmethod
    def run(self, model, tasks, few_shot, batch):
        ...
@nathan-weinberg nathan-weinberg added mmlu Pertains to MMLU mtbench Pertains to MTBench labels Jun 30, 2024
@nathan-weinberg
Copy link
Member

@JamesKunstle if we're outputting the evaluation results and saving it to a file or variable in memory, why not reuse the evaluator class?

This was referenced Jul 1, 2024
@alinaryan
Copy link
Member

@JamesKunstle do you have a PR for this already?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
mmlu Pertains to MMLU mtbench Pertains to MTBench
Projects
None yet
Development

No branches or pull requests

3 participants