-
Notifications
You must be signed in to change notification settings - Fork 2.1k
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Wow move fast! Great to see
- Add tests for all of them, please
- Add a MultiTeacher as the new default for glue and superglue. I'll show you an example shortly
parlai/tasks/glue/agents.py
Outdated
class AxTeacher(AbstractHuggingFaceTeacher): | ||
hf_path = 'glue' | ||
hf_name = 'ax' | ||
hf_text_fields = ['premise', 'hypothesis'] | ||
hf_label_field = 'label' | ||
hf_splits_mapping = {'train': 'test', 'valid': 'test', 'test': 'test'} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is an evaluation dataset (see https://huggingface.co/datasets/glue) so it only has a test split.
An alternative is to have train and valid map to None.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Good call out. Not sure what's going to happen if we make an empty teacher but we can try
ParlAI/parlai/tasks/decanlp/agents.py Lines 67 to 83 in 2f0e387
Example of a MultiTeacher |
hf_name = 'ax' | ||
hf_text_fields = ['premise', 'hypothesis'] | ||
hf_label_field = 'label' | ||
hf_splits_mapping = {'train': None, 'valid': None, 'test': 'test'} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
for evaluation datasets, changed mapping of train and valid to None e.g. hf_splits_mapping = {'train': None, 'valid': None, 'test': 'test'}
and removed the task from the tests and the multitaskteacher, otherwise will produce errors. e.g. display_data -t glue:ax
will work only if you set the datatype to test now.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Awesome!
Adding all glue datasets (https://huggingface.co/datasets/glue) and super glue datasets (https://huggingface.co/datasets/super_glue).
Added test.py for both glue and superglue and all pass.