142: Load best model instead of last one #151
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Proposed saving/reading of training checkpoints. Steps for correct use:
Launch the training. New flags were implemented to complement the saving of checkpoints:
a.
save_top_k
: The user can now choose the amount of best_checkpoints to store. He can also decide to save all checkpoints, using--save_top_k -1
.b.
monitor
: Metric stored in the name of the checkpoints. Defaultval_loss
.c. New storage format :
"{epoch}-{step}-{" + monitor + ":.4f}"
.Perform the checkpoint reading specified from the new flag
checkpoint
(only defined inmode=eval
). Three ways are taken into account:a.
--checkpoint last
(default): Reading of the last checkpoint.b.
--checkpoint *.ckpt
: Reading of a specific checkpoint.c.
--checkpoint best
: Try to read the checkpoint with the bestmonitor
, in two steps:torch.load(ckpt)
), extracting themonitor
value from :ckpt['callbacks'][ModelCheckpoint]['current_score']
.If none of the steps is possible, it throws an error.
Note:
With the implementation of 1.c, step 2.c.ii would not be necessary. It was implemented as a transition step between previous projects to this new feature (compatibility).
Closes #142