You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Not sure why this happened but found the solution by deleting the prior checkpoints
It turns out that one of the checkpoints (30 epochs) for some reason had two files, and I guess the script was getting confused by the filename so defaulted to the 30 epoch checkpoint.
I have a model that I am currently training with a goal of 200 epochs.
The latest checkpoint is epoch 140 but for some reason it keeps loading in checkpoint for epoch 30 when I try to resume training.
train.log
The text was updated successfully, but these errors were encountered: