Implement partial auroc metric

[0xRampey]

What does this PR do?

Working implementation for #3675
Some notes:

• Wrote very basic tests. Need some advice for more extensive testing.
• Commented the @auc_decorator out because the area had to be corrected after calculation. See comments.
• Perhaps the code can be optimized more to reduce trips between GPU and CPU. Planning to do that once extensive tests are in place.

Before submitting

• Was this discussed/approved via a Github issue? (no need for typos and docs improvements)
• Did you read the contributor guideline, Pull Request section?
• Did you make sure your PR does only one thing, instead of bundling different changes together? Otherwise, we ask you to create a separate PR for every change.
• Did you make sure to update the documentation with your changes?
• Did you write any new necessary tests?
• Did you verify new and existing tests pass locally with your changes?
• If you made a notable change (that affects users), did you update the CHANGELOG?

PR review

Anyone in the community is free to review the PR once the tests have passed.
If we didn't discuss your PR in Github issues there's a high chance it will not be merged.

Did you have fun?

Ofc ;)

Resources

• scikit's PR on partial AUC partial AUC scikit-learn/scikit-learn#3273

[pep8speaks]

Hello @prampey! Thanks for updating this PR.

There are currently no PEP 8 issues detected in this Pull Request. Cheers! 🍻

Comment last updated at 2020-12-29 17:47:04 UTC

[mergify]

This pull request is now in conflict... :(

[0xRampey]

I have added tests across the range (0, 1] for max_fpr. However, the tests are failing because there is a difference in float precision of 1e-3 popping up between the torch and sklearn results. Here's an example:

E       AssertionError: assert False                                                                                                                            
E        +  where False = <built-in method allclose of type object at 0x7fa60c1a3a80>(tensor(0.5120, device='cuda:0'), tensor(0.5122, device='cuda:0'))         
E        +    where <built-in method allclose of type object at 0x7fa60c1a3a80> = torch.allclose                                                                

This error originated because I changed the following code from
pred = torch.randint(n_cls_pred, (300,), device=device) to pred = torch.rand((300,), device=device) .
The AUC metric is usually applied to prediction probabilities and not prediction labels themselves, hence the need to generate random prediction values between 0 and 1.

I'd appreciate any help on taking this forward :)

[mergify]

This pull request is now in conflict... :(

[ddrevicky]

Hey @prampey the problem was with the reorder=True param to auc which uses unstable torch.argsort and this can cause the tpr/fpr pairs from which the AUC is calculated to be switched and that leads to computation of the wrong value. I tried setting it to False and your tests seem to pass locally :). Good job with the PR.

I think you just need to update the docstring and CHANGELOG now.

[SkafteNicki]

Hi @prampey, how is it going here?

[0xRampey]

Hi @SkafteNicki , haven't had an opportunity to work on this again. Will hopefully put in some time this week :)

[0xRampey]

Hey @SkafteNicki , sorry I've been hung up with my job and sending college applications. 🙈
Everything seems to be working fine, added a few more values to test out the whole range of (0,1] for max_fpr

[SkafteNicki]

LGTM

The base branch was changed.

[rohitgr7]

just some comments.