Add ZipLora

[shauray8]

What does this PR do?

Adds ZipLoRA: Any Subject in Any Style by Effectively Merging LoRAs

Fixes #5925

• Did you read the contributor guideline?
• Did you read our philosophy doc (important for complex PRs)?
• Was this discussed/approved via a GitHub issue or the forum? Please add a link to it if that's the case.
• Did you make sure to update the documentation with your changes? Here are the
  documentation guidelines, and
  here are tips on formatting docstrings.

Who can review?

@sanchit-gandhi @patrickvonplaten

@AmericanPresidentJimmyCarter

[patrickvonplaten]

Should we maybe try to directly add this to PEFT cc @younesbelkada @pacman100 @sayakpaul

[sayakpaul]

Adding to peft sounds good to me. But for training, I think we should encourage the community to use the scripts developed by @mkshing here: https://github.com/mkshing/ziplora-pytorch. They're quite elegant and are fully diffusers compatible.

[shauray8]

The training script by @mkshing is pretty neat actually, I'm not sure if you want this to be an inference-only addition to PEFT or if it is something else.

[younesbelkada]

Yes it would make sense to add ZipLora directly in PEFT and use it in diffusers. You can take a look at a similar PR here: huggingface/peft#1160
How does that to you @shauray8 ?

[shauray8]

Sounds good, I might need a little help moving forward @younesbelkada. Closing this and opening up a PR in PEFT.