revamp entire metrics

[ananyahjha93]

Co-authored-by: Teddy Koker teddy.koker@gmail.com

What does this PR do?

Deletes metrics

Fixes #3877

Before submitting

• Was this discussed/approved via a Github issue? (no need for typos and docs improvements)
• Did you read the contributor guideline, Pull Request section?
• Did you make sure your PR does only one thing, instead of bundling different changes together? Otherwise, we ask you to create a separate PR for every change.
• Did you make sure to update the documentation with your changes?
• Did you write any new necessary tests?
• Did you verify new and existing tests pass locally with your changes?
• If you made a notable change (that affects users), did you update the CHANGELOG?

PR review

Anyone in the community is free to review the PR once the tests have passed.
If we didn't discuss your PR in Github issues there's a high chance it will not be merged.

Did you have fun?

Always 💯
If we didn't discuss your PR in Github issues there's a high chance it will not be merged.

Did you have fun?

Make sure you had fun coding 🙃

[pep8speaks]

Hello @ananyahjha93! Thanks for updating this PR.

• In the file pytorch_lightning/metrics/metric.py:

Line 184:17: W503 line break before binary operator
Line 185:17: W503 line break before binary operator

Comment last updated at 2020-10-06 19:47:07 UTC

[codecov]

Codecov Report

Merging #3868 into master will decrease coverage by 4%.
The diff coverage is 76%.

@@           Coverage Diff           @@
##           master   #3868    +/-   ##
=======================================
- Coverage      88%     83%    -4%     
=======================================
  Files         117     108     -9     
  Lines        9147    8666   -481     
=======================================
- Hits         8007    7215   -792     
- Misses       1140    1451   +311     

[justusschock]

I really like the new interface. I only had some minor comments (e.g. let's keep functional metrics for now, some naming stuff and some type annotations). I assume docstrings will be added later?

[Borda]

we rather move the actual metrics to pl.legacy.metrics then completely before we fill all metrics we had before...
do the same with metric tests aka tests.legacy.metrics

[ananyahjha93]

@justusschock we are probably going to deprecate functional metric as with the whole update, reset and compute format, functional metrics don't make sense. If you want a single function to compute your metric value:

acc = Accuracy()

for batch in dataloader:
    step_acc_val = acc(preds, target)

Having a functional API with epoch aggregation and ddp sync per step doesn't make sense

[mergify]

This pull request is now in conflict... :(

[justusschock]

@ananyahjha93 the functional API does none of these things. It's just batch in, Value out and I really would keep these (same reason why pytorch has functionals of their loss functions).

[SkafteNicki]

PR is looking good. Really like many of the changes.

Question: is the idea to do to the hole update in this PR or just lay out the foundation for the metric package here, and the follow up with PRs for individual metrics?

As @justusschock has already mentioned in his review, I see 3 major breaking changes that we at least need to discuss:

1. Do we completely remove the functional interface?
2. Do we remove support for numpy arrays (by removing NumpyMetric subclass)?
3. If 2 is true, I assume we also remove sklearn interface?

[AtlantixJJ]

Hi guys, I just install the pytorch-lightning package with the latest version in master, but found out that the metrics.functional was gone and the metrics were almost empty.

Is that true for the moment due to the revamping?

Would you recommend a usable version as of now? Thanks