Learning rate finder

[SuroshAhmadZobair]

Hi!
I have been struggling lately to implement a learning rate finder in detectron2 for maskrcnn. Fastai lr_finder makes life very easy. i was wondering if any of you (users) have implemented such a lr_finder or of any kind in D2.

Thanks in advance.
cheers!

[jvdgoltz]

Hi,
Because LR is one of the most important parameters to set for training, I also thought to implement a LR-range finder for faster-rcnn.
Unfortunately, not with much success.

I implemented it so that it increases the LR exponentially over one epoch and records the range where the decrease in loss is the highest.
That then sets your min and max bounds for your learning rate.

I implemented it such that I could also use only sub losses, for example box regression loss or classification loss as the monitor for the LR finder. The following plot shows the output of the finder, I am finding the optimal bounds (red) by fitting a 3-part piecewise linear curve (green) on the monitored loss:

The learning rate finder itself works pretty well but the resulting LR bounds are useless. From my notes:

LearningRateFinder returns learning rates that are too low and lead to very slow convergence. Even when using the 
`loss_rpn_cls` that has the highest rate of change at higher learning rates than `total_loss`, the resulting 
learning rates are too slow, judged by the validation AP in tensorboard.

So yeah, if you find a way, I'd be eager to hear!

[mohdsaqibxa]

Hi! I have been struggling lately to implement a learning rate finder in detectron2 for maskrcnn. Fastai lr_finder makes life very easy. i was wondering if any of you (users) have implemented such a lr_finder or of any kind in D2.

Thanks in advance. cheers!

Have you implemented lr_finder in detectron2 for maskrcnn? If yes, please share the details.