-
Notifications
You must be signed in to change notification settings - Fork 402
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Recall and Specificity values are the same #1131
Comments
Hi! thanks for your contribution!, great first issue! |
can you actually give a few sample data (just raw tensor numbers that you think should produce different outputs? I am asking because I again checked both the formula and the implementation and they seem to be correct. Ideally you can just generate random number with a fixed seed. But as I said, for me this is not reproducible |
Issue will be fixed by classification refactor: see this issue #1001 and this PR #1195 for all changes Small recap: This issue describe that metric Recall and Specificity are all the same in the binary setting, which is wrong. The problem with the current implementation is that the metrics are calculated as average over the 0 and 1 class, which makes all the scores collapse into the same metric essentially. Using the new binary_* versions of all the metrics: from torchmetrics.functional import binary_recall, binary_specificity
import torch
preds = torch.rand(10)
target = torch.randint(0, 2, (10,))
binary_recall(preds, target) # tensor(0.5000)
binary_specificity(preds, target) # tensor(0.6250) which also corresponds to what sklearn is giving. Sorry for the confusion that this have given rise to. |
🐛 Bug
I'm using Recall and Specificity module metrics when training a binary classification model. The graphs in MLFlow are the exact same. Please see code below for how I'm using the module metrics. I'm pretty sure I'm doing it correctly because I'm following the torchmetrics doc as close as I can.
Code sample
Expected behavior
Expecting Recall and Specificity to be different values.
Environment
conda
,pip
, build from source): pip, 0.9.2Additional context
If you guys want any other info please let me know. See below for MLFlow graphs.
The text was updated successfully, but these errors were encountered: