Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add ignore_index option to exact_match metric #1540

Merged
merged 9 commits into from
Feb 23, 2023
Merged

add ignore_index option to exact_match metric #1540

merged 9 commits into from
Feb 23, 2023

Conversation

ItamarChinn
Copy link
Contributor

Fixes 1508

This fix sets preds to equal the ignore_index in the same indices that target == ignore_index such that if all other elements in preds except the ones at indices where target == ignore_index are identical this constitutes an exact match.

Before submitting
  • Was this discussed/agreed via a Github issue? (no need for typos and docs improvements)
  • Did you read the contributor guideline, Pull Request section?
  • Did you make sure to update the docs? -> unnecessary, the issue is that the docs claim a functionality that doesn't exist
  • Did you write any new necessary tests? -> added minimal test in the comments
PR review

Anyone in the community is free to review the PR once the tests have passed.
If we didn't discuss your PR in Github issues there's a high chance it will not be merged.

The following test checks that it functions correctly:

import torch
import torchmetrics

preds = torch.tensor([[0,0,0,1,1,1,1], [1,1,1,1,1,1,1]])
target = torch.tensor([[0,0,0,1,-100,-100,-100], [0,0,1,1,-100,-100,-100]])
exact_match = torchmetrics.classification.MulticlassExactMatch(task='multiclass', num_classes=2, ignore_index=-100)
exact_match.update(preds, target)
assert exact_match.compute() == torch.tensor(0.5), "MulticlassExactMatch behaving unexpectedly" 

@Borda Borda changed the title add ignore_index option to exact_match metric add ignore_index option to exact_match metric Feb 22, 2023
@Borda Borda added this to the v0.12 milestone Feb 22, 2023
@SkafteNicki SkafteNicki added the bug / fix Something isn't working label Feb 23, 2023
@SkafteNicki SkafteNicki enabled auto-merge (squash) February 23, 2023 11:26
@codecov
Copy link

codecov bot commented Feb 23, 2023

Codecov Report

Merging #1540 (1b06756) into master (b3915a3) will decrease coverage by 36%.
The diff coverage is 100%.

Additional details and impacted files
@@           Coverage Diff            @@
##           master   #1540     +/-   ##
========================================
- Coverage      87%     51%    -36%     
========================================
  Files         216     216             
  Lines       11331   11332      +1     
========================================
- Hits         9842    5752   -4090     
- Misses       1489    5580   +4091     

@SkafteNicki SkafteNicki merged commit a9def1c into Lightning-AI:master Feb 23, 2023
@mergify mergify bot added the ready label Feb 23, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug / fix Something isn't working ready
Projects
None yet
Development

Successfully merging this pull request may close these issues.

MulticlassExactMatch agnostic to 'ignore_index'
3 participants