init_optimizers(self, model)

[clmartinez151]

🐛 Bug

pytorch_lightning/trainer/optimizers.py in init_optimizers(self, model) fails to load monitor value when configure_optimizers() returns a tuple with multiple optimizers in dictionary form, each with its own LR Scheduler and Monitor Value. The bug seems to occur in the elif clause in line 56, where no monitor value is attempted to be extracted, so monitor val is always None in this case.

Please reproduce using SomeModel class

Example class below:

class SomeModel(TrainingBehavior):
    """
    Regular ResNet model wrapper
    """
    def __init__(self, hparams={}, num_outputs=11, input_size=sample_input_size, stacks=5):
        super(SomeModel, self).__init__()
        self.classifier = LegitClassifier()
        self.descriminator = SomeDescriminator()

    .........

    def configure_optimizers(self):
        optimizer1 = optim.Adam(self.classifier.parameters())
        optimizer2 = optim.Adam(self.descriminator.parameters())
        scheduler1 = optim.lr_scheduler.ReduceLROnPlateau(optimizer1, verbose=True, patience=4, factor=0.1)
        scheduler2 = optim.lr_scheduler.ReduceLROnPlateau(optimizer2, verbose=True, patience=4, factor=0.1)
        return (
            {'optimizer': optimizer1, 'lr_scheduler': scheduler1, 'monitor': 'val_loss'},
            {'optimizer': optimizer2, 'lr_scheduler': scheduler2, 'monitor': 'val_loss'},
        )

Expected behavior

You get an exception when trainer attempts to load model.

trainer = pl.Trainer()
trainer.fit(model)

**---------------------------------------------------------------------------
MisconfigurationException Traceback (most recent call last)
in
53 limit_test_batches=0.01)
54
---> 55 trainer.fit(model)
56 trainer.test(model)
57 model_metrics.append(trainer.progress_bar_metrics)

/net/10.57.1.2/vol/homes/martinezniev1/complexcode/env/lib64/python3.6/site-packages/pytorch_lightning/trainer/trainer.py in fit(self, model, train_dataloader, val_dataloaders, datamodule)
492 # ----------------------------
493 self.accelerator_backend = self.accelerator_connector.select_accelerator()
--> 494 self.accelerator_backend.setup(model)
495
496 # ----------------------------

/net/10.57.1.2/vol/homes/martinezniev1/complexcode/env/lib64/python3.6/site-packages/pytorch_lightning/accelerators/dp_accelerator.py in setup(self, model)
53 # CHOOSE OPTIMIZER
54 # allow for lr schedulers as well
---> 55 self.setup_optimizers(model)
56
57 # init torch data parallel

/net/10.57.1.2/vol/homes/martinezniev1/complexcode/env/lib64/python3.6/site-packages/pytorch_lightning/accelerators/accelerator.py in setup_optimizers(self, model)
148 return
149
--> 150 optimizers, lr_schedulers, optimizer_frequencies = self.trainer.init_optimizers(model)
151 self.trainer.optimizers = optimizers
152 self.trainer.lr_schedulers = lr_schedulers

/net/10.57.1.2/vol/homes/martinezniev1/complexcode/env/lib64/python3.6/site-packages/pytorch_lightning/trainer/optimizers.py in init_optimizers(self, model)
78 )
79
---> 80 lr_schedulers = self.configure_schedulers(lr_schedulers, monitor=monitor)
81 _validate_scheduler_optimizer(optimizers, lr_schedulers)
82

/net/10.57.1.2/vol/homes/martinezniev1/complexcode/env/lib64/python3.6/site-packages/pytorch_lightning/trainer/optimizers.py in configure_schedulers(self, schedulers, monitor)
130 if monitor is None:
131 raise MisconfigurationException(
--> 132 'configure_optimizers must include a monitor when a ReduceLROnPlateau scheduler is used.'
133 ' For example: {"optimizer": optimizer, "lr_scheduler": scheduler, "monitor": "metric_to_track"}'
134 )

MisconfigurationException: configure_optimizers must include a monitor when a ReduceLROnPlateau scheduler is used. For example: {"optimizer": optimizer, "lr_scheduler": scheduler, "monitor": "metric_to_track"}**

Environment

• PyTorch Version (e.g., 1.0): 1.7
• OS (e.g., Linux): Linux
• Python version: 3.7
• CUDA/cuDNN version: 10.2

[rohitgr7]

I guess it's not a valid config. you can pass it like:

opt1 = ...
opt2 = ...
sched1 = {'scheduler': ..., monitor: ...}
sched2 = {'scheduler': ..., monitor: ...}
return (
    {'optimizer': opt1, 'lr_scheduler': sched1},
    {'optimizer': opt2, 'lr_scheduler': sched2},
)

[tchaton]

Dear @clmartinez151,

I think @rohitgr7 answered your question, closing this issue for now.

Best,
T.C