Metric bootstrapper

[SkafteNicki]

Before submitting

• Was this discussed/approved via a Github issue? (no need for typos and docs improvements)
• Did you read the contributor guideline, Pull Request section?
• Did you make sure to update the docs?
• Did you write any new necessary tests?

What does this PR do?

Fixes #96
Implements a wrapper metric that allows the user to get bootstrapped metric values easily:

metric = BootStrapper(Accuracy(), num_bootstraps=10)
metric.update(preds, target)
mean, std = metric.compute()

where mean, std will be the mean and standard deviation calculated over 10 different metrics that have each been updated with resampled (with replacement) versions of preds, target.
Efficient way to determine confidence interval for metric values.

PR review

Anyone in the community is free to review the PR once the tests have passed.
If we didn't discuss your PR in Github issues there's a high chance it will not be merged.

Did you have fun?

Make sure you had fun coding 🙃

[codecov]

Codecov Report

Merging #101 (03024d8) into master (386b7e4) will decrease coverage by 8.94%.
The diff coverage is 82.08%.

@@            Coverage Diff             @@
##           master     #101      +/-   ##
==========================================
- Coverage   96.89%   87.95%   -8.95%     
==========================================
  Files         148      152       +4     
  Lines        4638     4757     +119     
==========================================
- Hits         4494     4184     -310     
- Misses        144      573     +429     

Flag	Coverage Δ
Linux	79.20% <82.08%> (+0.03%)	⬆️
Windows	79.20% <82.08%> (+0.03%)	⬆️
cpu	79.20% <82.08%> (-17.69%)	⬇️
gpu	96.68% <ø> (-0.22%)	⬇️
macOS	79.20% <82.08%> (-17.69%)	⬇️
pytest	87.95% <82.08%> (-8.95%)	⬇️
python3.6	?
python3.8	?
python3.9	?
torch1.4.0	?
torch1.8.1	?

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
torchmetrics/collections.py	73.46% <ø> (-24.49%)	⬇️
torchmetrics/wrappers/bootstrapping.py	79.31% <79.31%> (ø)
torchmetrics/__init__.py	100.00% <100.00%> (ø)
torchmetrics/utilities/imports.py	80.00% <100.00%> (+1.42%)	⬆️
torchmetrics/wrappers/__init__.py	100.00% <100.00%> (ø)
torchmetrics/utilities/distributed.py	22.85% <0.00%> (-74.29%)	⬇️
torchmetrics/classification/auc.py	47.61% <0.00%> (-52.39%)	⬇️
torchmetrics/functional/classification/auroc.py	46.15% <0.00%> (-40.01%)	⬇️
torchmetrics/metric.py	55.25% <0.00%> (-39.77%)	⬇️
torchmetrics/functional/regression/psnr.py	60.60% <0.00%> (-36.37%)	⬇️
... and 38 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 386b7e4...03024d8. Read the comment docs.

[pep8speaks]

Hello @SkafteNicki! Thanks for updating this PR.

There are currently no PEP 8 issues detected in this Pull Request. Cheers! 🍻

Comment last updated at 2021-03-27 08:11:51 UTC