Improve ModelCheckpoint

[cmpute]

🚀 Feature

Add two optional features:

1. Save the trainer checkpoint just before shutdown: add an optional argument (e.g. save_on_shutdown) in ModelCheckpoint to save the current trainer state before shutdown. Value of save_on_shutdown can only be None or the file path for saving.
2. Maintain a file (e.g. latest.ckpt) linking to the latest saved model (across multiple runs of training): add an optional argument (e.g. create_link_for_latest), the value can only be None or file path for saving.

Motivation

For the first one, if training is interrupted in the middle, no checkpoint is left after last saving, which could be several epochs ago. If I want to continue, I can only resume, at most, with the one saved at last epoch.

For the second one, this is a feature I always implement, maybe it's not essential for everyone. This is useful when I'm doing frequent training, I have to find all the way to the exact model saved last time. So I create a file called latest.ckpt at somewhere easy to reach, linking to the lastest model.

[benob]

I second this feature request. I would be satisfied with a model checkpoint callback which would both save two sets of checkpoints, one for the latest epoch, and one for the top_k according to validation metrics.

[cmpute]

And I also suggest adding another kind of checkpoint logger in time manner, i.e. save checkpoints every X seconds/hours

[daden-ms]

also, allow users to save checkpoints at a specific step, e.g. save checkpoints every 5000 steps,

[riklopfer]

@daden-ms you might be able to use this Trainer flag https://pytorch-lightning.readthedocs.io/en/latest/trainer.html#val-check-interval

[stale]

This issue has been automatically marked as stale because it hasn't had any recent activity. This issue will be closed in 7 days if no further activity occurs. Thank you for your contributions, Pytorch Lightning Team!