After completion of Trainer.hyperparameter_search() attribute trainer.state.best_model_checkpoint references the last trained model instead of the best one

[fantauzzi]

System Info

• transformers version: 4.28.1
• Platform: Linux-5.19.0-41-generic-x86_64-with-glibc2.35
• Python version: 3.11.3
• Huggingface_hub version: 0.14.1
• Safetensors version: not installed
• PyTorch version (GPU?): 2.0.0+cu117 (True)
• Tensorflow version (GPU?): not installed (NA)
• Flax version (CPU?/GPU?/TPU?): not installed (NA)
• Jax version: not installed
• JaxLib version: not installed
• Using GPU in script?: yes
• Using distributed or parallel set-up in script?: no

Who can help?

@sgugger

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

https://colab.research.google.com/drive/1Ht14ntTQy96-_zO-iVlwvAgkyY8t6vKc?usp=sharing

Call Trainer.hyperparameter_search(); when it completes, attribute Trainer.state.best_model_checkpoint and other Trainer.state attributes reference the last trained model, in the sequence of models trained by Trainer.hyperparameter_search().

Note: to speed-up reproduction of the issue, I have limited the training dataset size in the provided code, line #49; that's why the evaluation metrics at the end of the hyperparameters search are poor.

Expected behavior

After Trainer.hyperparameter_search() completes, attribute Trainer.state.best_model_checkpoint should contain the filename of the checkpoint with the best model among all the models trained during hyperparameters search, not the last model; that is, the model trained during the run indicated in the BestRun instance returned by hyperparameter_search()

Likewise, other Trainer.state attributes should relate to the same model, e.g:
Trainer.state.best_metric
Trainer.state.epoch
Trainer.state.global_step

[sgugger]

Hyperparameter search does not play well with the best model indeed. That's not something in our roadmap for fixing, but we are happy to look at any PR!

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.