EarlyStopping - Torch Handler

[HripsimeS]

Dear MONAI team,

I saw in MONAI there is an Early Stop Handler which acts as an Ignite handler (monai.handlers.EarlyStopHandler)
Do you have an Early Stopping as a Torch Handler and can you please send me an example?

Thank you in advance!

[wyli]

Hi @HripsimeS, is it possible to keep a copy of the associated code within the discussion threads? I suspect in that way our discussions can potentially benefit more users who arrive here via the search engines.

[Nic-Ma]

Hi @HripsimeS ,

Here are 2 simple examples for the usage of EarlyStopping handler:

1. How can I apply “EarlyStop” logic based on the loss value or validation metrics during model training?
   To use loss value to execute EarlyStop, attach the handler to trainer, added “-” negative value because smaller loss is better:

handler = EarlyStopHandler(
    trainer=trainer,
    patience=100,
    score_function=lambda x: -x.state.output['loss'],
    epoch_level=False,
)

2. To use validation metrics(for example: MeanDice) to execute EarlyStop, just attach the handler to the evaluator:

handler = EarlyStopHandler(
    trainer=trainer,
    patience=1,
    score_function=lambda x: x.state.metrics['val_mean_dice'],
)

Thanks.

[Nic-Ma]

Hi @HripsimeS ,

I am not expert on TensorBoard, I think you can refer to the TensorBoardHandler attached to the trainer:
https://github.com/Project-MONAI/tutorials/blob/master/modules/engines/unet_training_dict.py#L169
And the TensorBoardHandler attached to the evaluator:
https://github.com/Project-MONAI/tutorials/blob/master/modules/engines/unet_training_dict.py#L130
Please check the output_transform arg carefully to get what you want.

Thanks.

[HripsimeS]

Hi @Nic-Ma. In this MONAI tutorial with the simple two line of code is possible to do visualization of results on Tensorboard by having an access to log_dir
https://github.com/Project-MONAI/tutorials/blob/master/3d_segmentation/unet_segmentation_3d_ignite.ipynb

%load_ext tensorboard
%tensorboard --logdir=$log_dir

But unfortunately in my case it tells "No dashboards are active for the current data set. You haven’t written any data to your event files or TensorBoard can’t find your event files"

Maybe "unet_segmentation_3d_ignite" notebook writer can guess what is the issue with tensorboard visualization, as the logic is the same but for some reason it did not work in Early Stopping example by using log_dir. Thanks in advance!
https://github.com/Project-MONAI/tutorials/blob/master/modules/engines/unet_training_dict.py

[Nic-Ma]

Maybe @ericspod knows something?
I think you need to make sure the TensorBoard file exists in your log_dir folder.

Thanks.

[HripsimeS]

@Nic-Ma I found a simple solution. When the Early Stopping script is executed and all event files are saved in the log_dir we mentioned (in my case I saved in "E:\runs"), then just need to write on Command Prompt

tensorboard --logdir=E:\runs

It will show something like "Serving TensorBoard on localhostand", then we just need to copy-past http://localhost:6006/ on new Google browser and we will have nice tensorboard visualization of "train_loss", "train_acc", "val_mean_dice" and "val_acc".

Hope this will be useful for other MONAI users too, who want to do a visualization of events with Tensorboard.

[Nic-Ma]

Thanks for your sharing!