Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Fix] Fix gradient cumulative optimizer when resuming #2093

Merged
merged 2 commits into from
Aug 15, 2022

Conversation

shinya7y
Copy link
Contributor

@shinya7y shinya7y commented Jul 1, 2022

Motivation

Fix #2083

Modification

  • Fix the issue
  • Eliminate the calculation of residual_iters to avoid complicated implementation of mmcv/runner/hooks/optimizer.py and unit tests (I think the logic has another bug because every_n_iters has not been modified for it.)
  • Put the same code into _get_loss_factor

BC-breaking (Optional)

Since the bug causes wrong loss, models trained by simultaneous use of GradientCumulativeOptimizerHook and resume will show different results before and after this PR.

Checklist

Before PR:

  • I have read and followed the workflow indicated in the CONTRIBUTING.md to create this PR.
  • Pre-commit or linting tools indicated in CONTRIBUTING.md are used to fix the potential lint issues.
  • Bug fixes are covered by unit tests, the case that causes the bug should be added in the unit tests.
  • New functionalities are covered by complete unit tests. If not, please add more unit test to ensure the correctness.
  • The documentation has been modified accordingly, including docstring or example tutorials.

After PR:

  • If the modification has potential influence on downstream or other related projects, this PR should be tested with some of those projects, like MMDet or MMCls.
  • CLA has been signed and all committers have signed the CLA in this PR.

@HAOCHENYE
Copy link
Collaborator

Thanks for your contribution! LGTM~

@HAOCHENYE HAOCHENYE requested a review from mzr1996 July 2, 2022 16:06
@shinya7y shinya7y changed the title [WIP] Fix gradient cumulative optimizer when resuming [Fix] Fix gradient cumulative optimizer when resuming Jul 9, 2022
@ZwwWayne ZwwWayne merged commit 304f184 into open-mmlab:master Aug 15, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

nan or wrong loss by simultaneous use of GradientCumulativeOptimizerHook and resume
4 participants