Unitxt evaluator

[Roni-Friedman]

Adding unitxt evaluator.

To be complemented by adding unitxt as a benchmark in instructlab repo

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. @Roni-Friedman please rebase it. https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[nathan-weinberg]

@Mergifyio refresh

[mergify]

refresh

✅ Pull request refreshed

[danmcp]

Thanks for the PR!

[danmcp]

This seems like too much detail to ask for from the cli in 1 long string. Are all the values things we want users to specify or could some of them be implementation details? For the ones that are, I think we need to break them down into individual params.

[Roni-Friedman]

Users should have the flexibility to specify all details. I can suggest the following, tell me which you prefer:
1.a - have the recipe written down in a file and then in the cli just provide a path
1.b - have the recipe written down in a file, but in a json format (e.g. {card: ..., template: ...}, which is more friendly
2 - add some prefixed parameters such as card and template, but also a freetext parameter, as there are many customization a unitxt user may want to make.
3 - keep it as is :)

[danmcp]

Is this using a local directory? If so, it needs to be built off a param like output_dir with mt_bench.

[Roni-Friedman]

this is a temporary directory, deleted at the end of the evaluation process. Would you prefer the user specified an output dir? It does not contain anything of use for the user, just the files required for lm eval to run unitxt.

[danmcp]

I think it would make sense to use the output_dir so it doesn't confuse the user in a local directory. Also, since you do want to remove at the end, the create/remove logic should probably be in a try/finally block.

[Roni-Friedman]

so a user would specify an output dir but will find it doesn't exist at the end of the run?
If the user specifies it, I guess I will not delete it, right?

[danmcp]

so a user would specify an output dir but will find it doesn't exist at the end of the run?

The current output dir is a working dir for mt_bench.

If the user specifies it, I guess I will not delete it, right?

I was expecting you would create your directory inside the output_dir and then delete it when you are done. Or you could delete the directory before you start if there is some value in leaving it around.

[jwm4]

Would it make sense to put all this into a memory filesystem? In general, it is best to avoid unnecessary disk writes, especially for something that's likely to run in a cloud service where it may or may not have write permissions on some sort of disk.

[Roni-Friedman]

@jwm4 memory filesystem does not seem to work well, as directory is later accessed also by lm-eval inside the mmlu class and I don't want to start passing this filesystem around (unless owners support such an overall change)

[Roni-Friedman]

I think it would make sense to use the output_dir so it doesn't confuse the user in a local directory.

@danmcp So I'm not entirely sure what you mean here. I see mt_bench has output_dir: str = "eval_output",, but this is created only if one calls for mt_bench and does not enter a different output dir. Doing the following, although not sure it makes a lot of sense:

def assign_tasks_dir(self, task_name):
return os.path.join( "eval_output" ,f"{TEMP_DIR_PREFIX}_{task_name}")

[danmcp]

Apologies if my request wasn't clear, my suggestion was like mt_bench:

• Accept the root dir to use for output as a var
• Default it to the same root dir as mt_bench

[alimaredia]

@Roni-Friedman Could you explain in the description what benefit the Unitxt evaluator would have? Why would a user run the unitxt evaluator over just using the MMLUBranchEvaluator?

[Roni-Friedman]

@Roni-Friedman Could you explain in the description what benefit the Unitxt evaluator would have? Why would a user run the unitxt evaluator over just using the MMLUBranchEvaluator?

Let's discuss this in our meeting as well. I initially wrote a generic evaluator that can use all of unitxt features, but now my understanding is that there are two clear use cases and PR should be adjusted accordingly:
1 - run evaluation on user data
2 - run bluebench