Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Refactor] Classification 9/n #1189

Merged
merged 13 commits into from
Aug 26, 2022

Conversation

SkafteNicki
Copy link
Member

@SkafteNicki SkafteNicki commented Aug 20, 2022

What does this PR do?

Continues work on classification refactor #1001
Prior work:

At this point the core implementations are in place and clean up is beginning. This PR takes care of 3 issues:

  1. functional auc is deprecated as an metric and moved to utilities as an compute tool instead. Module AUC is completly deprecated and will be removed in future.
  2. kl_divergence is deprecated from classification and moved to regression where it better fits (based on the fact that target should be a float tensor for this metric)
  3. the default value of average is changed from micro to macro. This is to prevent uses getting the same value from precision, recall and f1 in the multiclass/multilabel cases. Essentially the following holds:
    from torchmetrics.functional import multiclass_precision, multiclass_recall
    import torch
    p, t = torch.randint(3, (20,)), torch.randint(3, (20,))
    multiclass_precision(p, t, num_classes=3, average="micro") == multiclass_recall(p, t, num_classes=3, average="micro")
    # tensor(True)
    multiclass_precision(p, t, num_classes=3, average="macro") == multiclass_recall(p, t, num_classes=3, average="macro")
    # tensor(False)
    it is just not wrong to use micro in these cases, the underlying statistics just collapse into the same numbers in these cases. This change should hopefully prevent such confusion for users.

Before submitting

  • Was this discussed/approved via a Github issue? (no need for typos and docs improvements)
  • Did you read the contributor guideline, Pull Request section?
  • Did you make sure to update the docs?
  • Did you write any new necessary tests?

PR review

Anyone in the community is free to review the PR once the tests have passed.
If we didn't discuss your PR in Github issues there's a high chance it will not be merged.

Did you have fun?

Make sure you had fun coding 🙃

@SkafteNicki SkafteNicki added enhancement New feature or request refactoring refactoring and code health labels Aug 20, 2022
@SkafteNicki SkafteNicki added this to the v0.10 milestone Aug 20, 2022
@SkafteNicki SkafteNicki changed the title Refactor/classification 9 [Refactor] Classification 9/n Aug 21, 2022
@codecov
Copy link

codecov bot commented Aug 21, 2022

Codecov Report

Merging #1189 (6875845) into devel/classification (031ffdd) will decrease coverage by 0%.
The diff coverage is 79%.

@@                 Coverage Diff                  @@
##           devel/classification   #1189   +/-   ##
====================================================
- Coverage                    89%     89%   -0%     
====================================================
  Files                       189     191    +2     
  Lines                     10767   10786   +19     
====================================================
- Hits                       9587    9577   -10     
- Misses                     1180    1209   +29     

Copy link
Member

@justusschock justusschock left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We probably don't strictly need to follow update-compute in our utils, especially if there is not modular interface so I propose a slightly more descriptive name :)

Other than that: LGTM as always :)

src/torchmetrics/utilities/compute.py Outdated Show resolved Hide resolved
src/torchmetrics/utilities/compute.py Outdated Show resolved Hide resolved
SkafteNicki and others added 3 commits August 26, 2022 08:35
Co-authored-by: Justus Schock <12886177+justusschock@users.noreply.github.com>
@mergify mergify bot added the ready label Aug 26, 2022
@justusschock justusschock merged commit 033e458 into devel/classification Aug 26, 2022
@justusschock justusschock deleted the refactor/classification_9 branch August 26, 2022 12:39
Borda pushed a commit that referenced this pull request Aug 26, 2022
* micro to macro

* new kl

* deprecate old

* move auc

* move tests around

* change import paths

* update test path

* change ref

* Apply suggestions from code review

Co-authored-by: Justus Schock <12886177+justusschock@users.noreply.github.com>

* update doctests

* doctests fix

Co-authored-by: Justus Schock <12886177+justusschock@users.noreply.github.com>
@SkafteNicki SkafteNicki mentioned this pull request Aug 28, 2022
4 tasks
Borda pushed a commit that referenced this pull request Sep 12, 2022
* micro to macro

* new kl

* deprecate old

* move auc

* move tests around

* change import paths

* update test path

* change ref

* Apply suggestions from code review

Co-authored-by: Justus Schock <12886177+justusschock@users.noreply.github.com>

* update doctests

* doctests fix

Co-authored-by: Justus Schock <12886177+justusschock@users.noreply.github.com>
Borda pushed a commit that referenced this pull request Sep 12, 2022
* micro to macro

* new kl

* deprecate old

* move auc

* move tests around

* change import paths

* update test path

* change ref

* Apply suggestions from code review

Co-authored-by: Justus Schock <12886177+justusschock@users.noreply.github.com>

* update doctests

* doctests fix

Co-authored-by: Justus Schock <12886177+justusschock@users.noreply.github.com>
Borda pushed a commit that referenced this pull request Sep 13, 2022
* micro to macro

* new kl

* deprecate old

* move auc

* move tests around

* change import paths

* update test path

* change ref

* Apply suggestions from code review

Co-authored-by: Justus Schock <12886177+justusschock@users.noreply.github.com>

* update doctests

* doctests fix

Co-authored-by: Justus Schock <12886177+justusschock@users.noreply.github.com>
Borda pushed a commit that referenced this pull request Sep 13, 2022
* micro to macro

* new kl

* deprecate old

* move auc

* move tests around

* change import paths

* update test path

* change ref

* Apply suggestions from code review

Co-authored-by: Justus Schock <12886177+justusschock@users.noreply.github.com>

* update doctests

* doctests fix

Co-authored-by: Justus Schock <12886177+justusschock@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request ready refactoring refactoring and code health
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants