[BOUNTY] update ReLoRA implementation

[winglian]

⚠️ Please check that this feature request hasn't been suggested before.

• I searched previous Ideas in Discussions didn't find any similar feature requests.
• I searched previous Issues didn't find any similar feature requests.

🔖 Feature description

relora improved their implementation as the original wasn't very good.

Value of the bounty is $100 in runpod GPU credits.

✔️ Solution

see Guitaricet/relora@d8ff7ee#diff-0d94c276364524bc8e6bdd0d43b1ee104fe56a541129a1e588aeffd87bb22840

❓ Alternatives

No response

📝 Additional Context

No response

Acknowledgements

• My issue title is concise, descriptive, and in title casing.
• I have searched the existing issues to make sure this feature has not been requested yet.
• I have provided enough information for the maintainers to understand and evaluate this request.

[SwapnilDreams100]

Hi @winglian, happy to start working on this. Are there any specific changes you are looking for? It seems like the key changes implemented in the new ReLora implementation is the addition of quantization, which is already present in the current implementation.

[ehartford]

This is exciting, I was pretty sad when the original ReLoRA didn't work out. Would love to do some pretraining on a GPU-poor setup!

[dustydecapod]

I'd like to add an additional bounty to this of $400 in credits at runpod. I pretty much /need/ this in order to train some models.

[winglian]

Going to close this. I've added magnitude pruning of the optimizer states in #1245 which is the main change