AUROC with reorder?

[FluidSense]

I'm currently using the AUROC metric for my binary classifier. However, as I am working with multi-task learning, I sometimes have to keep iterating over epochs although the binary classifier part of the system already is overfitted. When I try to do this, the training sometimes crashes as a ValueError is thrown with the message The 'x' tensor is neither increasing or decreasing. Try setting the reorder argument to 'True'.
I have to admit I'm not certain of how this would help.
I see there is no way of setting reorder=True on the AUROC module. Is it intentional? Would it make sense to be able to set reorder=True on the module?

[SkafteNicki]

Hi @FluidSense,
The reason that a reorder argument is not exposed in the AUROC metric is because auc is not called directly on the inputs (therefore they do not need to be sorted) but on calculated fpr and tpr scores that should already be sorted. Could you provide a reproducible example?

[FluidSense]

https://github.com/FluidSense/minimal-auroc-example
I extended the BoringModel (copy-pasted as I couldn't find a place to import it from) and added AUROC and Accuracy as a MetricCollection.

The dataset is generated by fake/generateFakeData.py, python -m fake.generateFakeData but is also included so that it can be tested directly through pip install -r requirements.txt && python main.py or poetry install && poetry run python main.py if you use Poetry.
It contains random values between -.5 and .5, and the target value is 1 if the sum is larger than 0.5 else 0.

By running this for an extended amount of epochs (I set it to 1000, but it threw the error at epoch 479) I was able to reproduce it.

[SkafteNicki]

Hi @FluidSense,
I now had time to look at it. It seems like the checks we have for direction are not robust towards numerical instabilities:
https://github.com/PyTorchLightning/metrics/blob/5510ccd99eaec5ab8175bbd5e2ad9e66e82d10e4/torchmetrics/functional/classification/auc.py#L42-L43
I my run it raises that error, because the dx tensor somewhere has both positive and negative values. However, when inspective the dx tensor the smallest number is

tensor(-1.1921e-07) 

which comes from subtracting these two tensors:

x[271948]
tensor(0.9985)
x[271949]
tensor(0.9985)

so small numerical difference on one of the last digits makes it fail. The solution is to make the checks a bit more robust:

tol = 1e-6
dx = x[1:] - x[:-1]
if (dx < -tol).any():
    if (dx <= -tol).all():

[SkafteNicki]

Hi @FluidSense,
The issue have been resolved by PR #230. Please install from master if you want the fix right away :]
Thanks for using torchmetrics!