[Docs] Note on running metric in dp

[SkafteNicki]

What does this PR do?

Fixes #4353
Metrics needs to be treated differently in DP mode (until we have statefull DP #1895). This PR adds a note to the docs what the user can do for now.

Before submitting

• Was this discussed/approved via a Github issue? (no need for typos and docs improvements)
• Did you read the contributor guideline, Pull Request section?
• Did you make sure your PR does only one thing, instead of bundling different changes together? Otherwise, we ask you to create a separate PR for every change.
• Did you make sure to update the documentation with your changes?
• Did you write any new necessary tests?
• Did you verify new and existing tests pass locally with your changes?
• If you made a notable change (that affects users), did you update the CHANGELOG?

PR review

• Is this pull request ready for review? (if not, please submit in draft mode)

Anyone in the community is free to review the PR once the tests have passed.
If we didn't discuss your PR in Github issues there's a high chance it will not be merged.

Did you have fun?

Make sure you had fun coding 🙃

[codecov]

Codecov Report

Merging #4494 (9f2ed8f) into master (ee35907) will decrease coverage by 0%.
The diff coverage is n/a.

@@          Coverage Diff           @@
##           master   #4494   +/-   ##
======================================
- Coverage      93%     93%   -0%     
======================================
  Files         116     116           
  Lines        8833    8833           
======================================
- Hits         8235    8232    -3     
- Misses        598     601    +3     

[tchaton]

Great catch ! Any idea on how to solve ?

[SkafteNicki]

@tchaton Only solution is to make a custom version of DataParallel (as PR #1895 tried to do) where we can insert hooks between these lines of code:
https://github.com/pytorch/pytorch/blob/73e121de1c69bba6863f38bea4b0044a920fdb20/torch/nn/parallel/data_parallel.py#L167-L168
such that metrics can accumulate their internal state after the data is send through the model, but before the results are gathered.