[WIP] Trainer supporting evaluation on multiple datasets

[timbmg]

What does this PR do?

With this PR Trainer and Seq2SeqTrainer support evaluating on multiple datasets. For this, the eval_dataset and compute_metrics parameters have been updated. In order to evaluate on multiple datasets, eval_dataset should be a dict mapping a dataset name to a Dataset. In _maybe_log_save_evaluate we then loop over the dict, calling evaluate with each Dataset. The metric prefix is also updated to contain the dataset name. Furthermore, each eval dataset can optionally have its own compute_metrics function. For this, compute_metrics should be a dict where the keys match with eval_dataset.

Fixes #15857

Before submitting

• Did you read the contributor guideline,
  Pull Request section?
• Was this discussed/approved via a Github issue or the forum? Please add a link
  to it if that's the case.
• Did you make sure to update the documentation with your changes? Here are the
  documentation guidelines, and
  here are tips on formatting docstrings.
• Did you write any new necessary tests?

Who can review?

@sgugger

[HuggingFaceDocBuilderDev]

The documentation is not available anymore as the PR was closed or merged.

[timbmg]

Hey @sgugger, I mostly followed your suggestion in #15857, except instead of having a list of eval_datasets and another training arg, I solved it via passing a dict of eval_datasets. I thought a dict would work better because we also need multiple compute_metric functions. This way it is all lined up and less error-prone. However, let me know if you think otherwise.

Also, could you suggest what tests to write for this PR? I am not really sure, since the major change is in _maybe_log_save_evaluate and I didn't find a test for that.

[sgugger]

Thanks for your PR! Having the multiple datasets as a dict solves the problem of distinguishing a single dataset that is a list or a list of datasets. So I like this part.

However I didn't see anything in the issue regarding using several compute_metrics function. If there is a need for different metrics, it probably means different Trainer should be built as it represents different tasks/problems. That change should be reverted, as the part where compute_metrics can be passed along to the evaluate/predict function.

[timbmg]

Thanks for checking it so quickly!

In my case, I am training a seq2seq QA model and evaluating it on multiple datasets. However, they have different formats (eg extractive qa like SQuAD, or multiple-choice qa like commonsese QA). Using a seq2seq model for multiple formats has been for example proposed in the UnifiedQA paper. Having multiple trainers has the limitation that I could only train on a single dataset at a time, but not train on multiple ones at the same time. However, note that if you pass multiple eval_datasets as a dict, but only a single compute_metric callable, the same compute_metrics function will be called on all the eval_datasets. That's what this if statement is doing. So the original scenario described in the Issue is also solved.

[sgugger]

It's too niche of a use-case to allow for support, especially when we have other tools that easily let you more customizable training/evaluation loops like Accelerate.

[timbmg]

Alright, I have reverted the change. Let me know in case of anything else:)

[sgugger]

Looking good, thanks a lot for amending the PR!

[jurrr]

I'm trying to take advantage of the feature to include multiple eval_datasets in the trainer. Maybe I'm misreading the documentation, I've tried several ways to present the eval_dataset, but keep getting a KeyError when I include a DatasetDict / dict with datasets for the eval_dataset parameter. Am I doing something wrong? Do I need to specify the compute_metrics differently? Couldn't find anything on that.

Here's an example notebook resulting in the ValueError: https://colab.research.google.com/drive/1yLo9iqY4Cz9_h8BtAvcYRCtK5O_xa5jP?usp=sharing

[alexcoca]

Thanks for your PR! Having the multiple datasets as a dict solves the problem of distinguishing a single dataset that is a list or a list of datasets. So I like this part.

However I didn't see anything in the issue regarding using several compute_metrics function. If there is a need for different metrics, it probably means different Trainer should be built as it represents different tasks/problems. That change should be reverted, as the part where compute_metrics can be passed along to the evaluate/predict function.

@sgugger passing multiple compute_metrics functions for evaluation purposes can actually be more general than stated by @timbmg. For example, suppose we are doing multi-task training and we wish to evaluate on the same or held-out tasks as we train. This is common in recent research publications (eg FLAN-T5). Would you accept to support the multiple compute metrics functions? Or would your advice be to not use the trainer altogether and look towards using accelerate? I was worried that accelerate for training is a big step back towards writing a lot of boilerplate and code that the Trainer saves us.

[sgugger]

I'd recommend using Accelerate instead of the Trainer for this use case.

[ZQ-Dev8]

@sgugger Any examples out there of using Accelerate for this? I would also like to evaluate on multiple datasets while training. Thanks!

[sieu-n]

fyi, @dcruiz01 I guess this can be implemented by overriding the Trainer class following the initial commit? Or some mixin regarding multiple trainers or even monkey patching it. Another workaround that comes to my mind is identifying the eval dataset with an additional field and splitting them in the compute_metrics function. I'll try to share if any actually works. Both do feel really hacky though..

Though I agree this is less common for standard fine-tuning, I want to add use cases where we want the model to perform well on multiple tasks in a zero-shot manner.

• fine-tuning LLMs, in which we want to measure performance on many many different task / metrics / dataset.
• More meta-benchmarks such as MTEB

[ZQ-Dev8]

@sieu-n Any luck with the experiments you mentioned?

[lhallee]

@sgugger @timbmg it looks like the eval_datasets change was pushed for support as a dict but the compute_metrics changes were not pushed. Will this change be made or no?

[0seba]

Hey, I think the additional feature to use separated data collators would be useful