[Feature Request] EarlyStopping logging on rank 0 only

[austinmw]

🚀 Feature

Toggle switch to turn off EarlyStopping logging for processes other than rank 0

Motivation

EarlyStopping logging can be a bit spammy when viewing aggregate logs across all processes. For example, with my custom CloudWatch logger:

xnpww4j62d-algo-1-vr8o9 | 14:17:49 [INFO] Epoch 9: [ Training | 100%  iter# 49/49    19.28 batches/s ] train/loss_step=0.764418, train/loss_epoch=0.773, train/acc=0.68356
xnpww4j62d-algo-1-vr8o9 | 14:17:55 [INFO] Epoch 9: [ Validation | 100%  iter# 10/10     2.34 batches/s ] val/loss_step=1.253475, val/loss_epoch=1.278802, val/acc=0.6107
xnpww4j62d-algo-1-vr8o9 | 14:17:55 [INFO] [rank: 0] Metric val/acc improved by 0.195 >= min_delta = 0.0. New best score: 0.611
xnpww4j62d-algo-1-vr8o9 | 14:17:55 [INFO] [rank: 2] Metric val/acc improved by 0.195 >= min_delta = 0.0. New best score: 0.611
xnpww4j62d-algo-1-vr8o9 | 14:17:55 [INFO] [rank: 1] Metric val/acc improved by 0.195 >= min_delta = 0.0. New best score: 0.611
xnpww4j62d-algo-1-vr8o9 | 14:17:55 [INFO] [rank: 3] Metric val/acc improved by 0.195 >= min_delta = 0.0. New best score: 0.611
xnpww4j62d-algo-1-vr8o9 | 14:17:55 [INFO] [rank: 4] Metric val/acc improved by 0.195 >= min_delta = 0.0. New best score: 0.611
xnpww4j62d-algo-1-vr8o9 | 14:17:55 [INFO] [rank: 5] Metric val/acc improved by 0.195 >= min_delta = 0.0. New best score: 0.611
xnpww4j62d-algo-1-vr8o9 | 14:17:55 [INFO] [rank: 6] Metric val/acc improved by 0.195 >= min_delta = 0.0. New best score: 0.611
xnpww4j62d-algo-1-vr8o9 | 14:17:55 [INFO] [rank: 7] Metric val/acc improved by 0.195 >= min_delta = 0.0. New best score: 0.611
xnpww4j62d-algo-1-vr8o9 | 14:18:20 [INFO] Epoch 14: [ Training | 100%  iter# 49/49    18.94 batches/s ] train/loss_step=0.611876, train/loss_epoch=0.55, train/acc=0.80096
xnpww4j62d-algo-1-vr8o9 | 14:18:26 [INFO] Epoch 14: [ Validation | 100%  iter# 10/10     2.29 batches/s ] val/loss_step=0.748429, val/loss_epoch=0.828285, val/acc=0.726

Pitch

It would be nice if we could turn off printing of this message on processes other than rank 0. I understand that this is actually useful to monitor in some cases, so maybe this toggle could be set to False by default.

Alternatives

Custom EarlyStopping callback?

cc @Borda @carmocca @awaelchli @rohitgr7

[carmocca]

I think we can add this flag. It's useful for metrics logged with sync_dist=True.

The relevant piece of code is here:

https://github.com/PyTorchLightning/pytorch-lightning/blob/dd475183227644a8d22dca3deb18c99fb0a9b2c4/pytorch_lightning/callbacks/early_stopping.py#L256-L261

[austinmw]

Thanks!

[ekagra-ranjan]

Hi @carmocca! I would like to take this up.

[ekagra-ranjan]

Hi @carmocca - I have a doubt regarding this:

It's useful for metrics logged with sync_dist=True

The metrics for callbacks are stored during epoch end:
https://github.com/PyTorchLightning/pytorch-lightning/blob/c1f05021ff0093f720770a6065ab62a70c535add/pytorch_lightning/trainer/connectors/logger_connector/result.py#L584-L587

and sync_dist will always be true for epoch level metric computation as per:
https://github.com/PyTorchLightning/pytorch-lightning/blob/c1f05021ff0093f720770a6065ab62a70c535add/pytorch_lightning/trainer/connectors/logger_connector/result.py#L524-L531

so will our new flag be dependent on the value of sync_dist somehow?

Also, if the epoch level metrics will always be reduced across processes, then won't the logs be the same for all the ranks (as seen in the logs shared by @austinmw) ?

[carmocca]

@ekagra-ranjan For an initial implementation, I suggest you don't try to decide this automatically. Just allow the user to choose manually by passing rank_0_only=True in the constructor.