RMS PROP 8bit support

[NicolasMejiaPetit]

Current only rms prop is supported by trl for DPO training. But bitsandbytes contains 8bit RMSprop that could be used to save on vram, making DPO QLORA on 7b’s fit better without going into shared memory, on 24 gig cards. Could we work on an implementation of 8bit rms prop into trl?

[younesbelkada]

Hi @NickWithBotronics !
Very nice feature requests :) this indeed will be very helpful for the community !
I made huggingface/transformers#29082 which should add the support for bnb RMSProp optimizers in HF Trainer. Once that is merged, you'll just need to pass optim="rmsprop_bnb" / "rmsprop_bnb_8bit" / "rmsprop_bnb_32bit" to TrainingArguments before passing it to SFTTrainer or DPOTrainer

[NicolasMejiaPetit]

I love you and the HF team y'all are so awesome! Thank you! Also on a side note, there is a 4bit adamnw optimizer that claims the same performance as 32bit adamnw I don’t know how well these lower bit optimizers work from first hand use, but they have a awesome paper, and code. “ https://github.com/thu-ml/low-bit-optimizers ” . Maybe it’s worth looking into integrating into the HF trainer.