AdaLomo optimizer step method

[winglian]

Hi @KaiLv69

thanks for the writeup and implementation for AdaLomo! It looks like it is missing the step method which torch needs to use this in other frameworks. Can you help with this please?

[KaiLv69]

Hi, thanks for your interests :)

AdaLomo (and LOMO) fused the backward and step process into one single process called fused backward to save memory for gradient. So there is no step method in AdaLomo. We have done special processing for this in the trainer. https://github.com/OpenLMLab/collie/blob/db76a99758ddecb3be48ab87e04643f7d7932ac4/collie/controller/trainer.py#L456

[winglian]

Thanks @KaiLv69 . Would you be able to share the LRs you used for the adalomo and adamw experiments in the paper? I think I might have the LR off by a large factor as adalomo converges much slower for me than adamw when using the same LR for both.

[KaiLv69]

1e-3 or 5e-4 might be proper for adalomo.