Trainer.train argument resume_from_last_checkpoint

[tanmay17061]

🚀 Feature request

Trainer.train accepts resume_from_checkpoint argument, which requires the user to explicitly provide the checkpoint location to continue training from.
resume_from_last_checkpoint can be useful to resume training by picking the latest checkpoint from output_dir of the TrainingArguments passed.

Motivation

1. The checkpoint directory is created by the library, so user needs to navigate to the directory to find the value to provide for resume_from_checkpoint
2. User may just want to resume from the last valid checkpoint since their training got disrupted previously (a common scenario for someone to want to resume training). All they know is the output_dir they provided initially

This motivates to provide a resume_from_last_checkpoint=True to the Trainer.train(...) call, which will pick the latest checkpoint from args.output_dir. FYI get_last_checkpoint function from trainer_utils can be used to do exactly the same.

Your contribution

I can raise a PR if it is a useful feature to have!

[sgugger]

Instead of adding a new argument, I would use the existing resume_from_checkpoint and change its type to bool or str/PathLike. If it's a bool and if it's True, we then use get_last_checkpoint to get the last checkpoint in args.output_dir. Does that sound good to you?

[tanmay17061]

Yes, SGTM. I have raised a PR doing the same. Do let me know if there is any other change required as well!

PS: Can you also review my other PR introducing save_strategy in TrainingArguments? This PR is the last one to round-up the save_strategy, evaluation_strategy and logging_strategy enhancements.

Thanks!

[npoulose]

🚀 Feature request

Trainer.train accepts resume_from_checkpoint argument, which requires the user to explicitly provide the checkpoint location to continue training from. resume_from_last_checkpoint can be useful to resume training by picking the latest checkpoint from output_dir of the TrainingArguments passed.

Motivation

1. The checkpoint directory is created by the library, so user needs to navigate to the directory to find the value to provide for resume_from_checkpoint
2. User may just want to resume from the last valid checkpoint since their training got disrupted previously (a common scenario for someone to want to resume training). All they know is the output_dir they provided initially

This motivates to provide a resume_from_last_checkpoint=True to the Trainer.train(...) call, which will pick the latest checkpoint from args.output_dir. FYI get_last_checkpoint function from trainer_utils can be used to do exactly the same.

Your contribution

I can raise a PR if it is a useful feature to have!
Is it possible to train while adding a new category to the dataset using this resume_from_checkpoint argument?