Optionally Avoid recomputing features

[justusschock]

What does this PR do?

Optionally avoids recomputing the features.

Before submitting

• Was this discussed/approved via a Github issue? (no need for typos and docs improvements)
• Did you read the contributor guideline, Pull Request section?
• Did you make sure to update the docs?
• Did you write any new necessary tests?

PR review

Anyone in the community is free to review the PR once the tests have passed.
If we didn't discuss your PR in Github issues there's a high chance it will not be merged.

Did you have fun?

Make sure you had fun coding 🙃

[SkafteNicki]

exclude_states should also be added to:

• MetricCollection
• MinMaxMetric
• MultioutputWrapper
• MetricTracker

for consistency

[Borda]

@justusschock still draft or ready to read? 🐰

[Borda]

is it related to #709? 🐰

[justusschock]

@Borda waiting for some user feedback on slack :)

I think not directly related to #709

[Borda]

waiting for some user feedback on slack :)

what kind of feedback, could you please remind me of the thread... 🐰

[ebartrum]

I believe this currently breaks loading from checkpoint

[justusschock]

To avoid bug when passing None

[SkafteNicki]

@justusschock instead of making this PR have influence on the base class, what about re-implementing just reset in the relevant metrics:

def reset(self):
  if not self.reset_real_features:
    # remove temporarily to avoid resetting
    value = self._defaults.pop('real_features')
    super().reset()
    self._defaults['real_features'] = value
  else:
    super().reset()

this feature seems to only be relevant to Inception, FID and KID

[justusschock]

@SkafteNicki works for me. However, afaik there are some similar metrics in NLP which might be added some day. This is why I aimed for a more general approach here. I'm fine with both, just let me know your preference :)

[SkafteNicki]

@justusschock I would go with the more local solution for the affected metrics for now. If we then in the future have other metrics that need a similar feature then I am fine with revisiting this :]

[codecov]

Codecov Report

Merging #722 (e817ad1) into master (0d56376) will decrease coverage by 12%.
The diff coverage is 91%.

@@          Coverage Diff           @@
##           master   #722    +/-   ##
======================================
- Coverage      95%    83%   -12%     
======================================
  Files         173    173            
  Lines        7381   7401    +20     
======================================
- Hits         7012   6149   -863     
- Misses        369   1252   +883     

[SkafteNicki]

@justusschock took the liberty to update the PR, please take a look :]

[justusschock]

@SkafteNicki Thanks for updating. Seems you were faster :) LGTM :)