Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add a two-class variant of the MNLI task #976

Merged
merged 4 commits into from
Dec 20, 2019
Merged

Add a two-class variant of the MNLI task #976

merged 4 commits into from
Dec 20, 2019

Conversation

sleepinyourhat
Copy link
Contributor

This adds the option to train a model on the three-class MNLI dataset and evaluate it on comparable two-class datasets like RTE or HANS. It does this by overriding the usual accuracy metric with an NLI-specific accuracy metric that collapses 'neutral' and 'contradiction'.

@sleepinyourhat sleepinyourhat added low-priority Only if you're bored. Ask Sam/Ian/Alex before starting. new-task Adding support for a new task. labels Dec 16, 2019
W4ngatang
W4ngatang approved these changes Dec 20, 2019
View reviewed changes
Copy link
Collaborator

@W4ngatang W4ngatang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think a more robust way to implement this would be to have a variable (and global, so we can share between NLI tasks) label indexing instead of a specific scorer, but seems like it'll work as is.

@sleepinyourhat sleepinyourhat merged commit 12bc45e into master Dec 20, 2019
phu-pmh pushed a commit that referenced this pull request Apr 17, 2020
@sleepinyourhat
Add a two-class variant of the MNLI task (#976)
db0fdfd 
* Part I

* Add metric

* Clarifications

* Update tasks.py
@jeswan jeswan mentioned this pull request Sep 17, 2020
Closed
@jeswan jeswan added the jiant-v1-legacy Relevant to versions <= v1.3.2 label Sep 17, 2020
@jeswan jeswan deleted the nli-metrics branch September 22, 2020 03:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@W4ngatang W4ngatang W4ngatang approved these changes

@iftenney iftenney Awaiting requested review from iftenney

@pruksmhc pruksmhc Awaiting requested review from pruksmhc

Assignees
No one assigned
Labels
jiant-v1-legacy Relevant to versions <= v1.3.2 low-priority Only if you're bored. Ask Sam/Ian/Alex before starting. new-task Adding support for a new task.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@sleepinyourhat @W4ngatang @jeswan