-
Notifications
You must be signed in to change notification settings - Fork 3.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Missing training_step outputs in training_epoch_end #2327
Missing training_step outputs in training_epoch_end #2327
Conversation
…ss all the batch outputs to training_epoch_end(if user defined this method)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
mind add a test for this case? probably some simple example from #2320
Codecov Report
@@ Coverage Diff @@
## master #2327 +/- ##
======================================
Coverage 88% 88%
======================================
Files 70 70
Lines 5501 5503 +2
======================================
+ Hits 4834 4836 +2
Misses 667 667 |
@mmiakashs mind trying master now? the solution in this PR wasn't quite 100% right and needed more testing. This PR is likely not needed anymore but we need to add you as co-author to #2328 @Borda |
This pull request is now in conflict... :( |
yes... but this PR was incorrect |
@williamFalcon Thanks a lot for the PR. One confusion: I just noticed that all the training_step end log metrics are combined with the dict key named |
@williamFalcon I debug again and found out that the issue #2320 still occurred only for training_step outputs. training_step outputs for the first optimizer iteration are missing, however, the second optimizer iteration outputs are merged properly. |
@mmiakashs do you see a fix for it, mind send a PR? |
Possible bug fix of #2320