Add ZipLora to enhance multi-lora inference

[AmericanPresidentJimmyCarter]

Is your feature request related to a problem? Please describe.
Refer to https://ziplora.github.io/

Describe the solution you'd like.
When multiple LoRAs are passed to the model, you should have the ability to use ziplora to merge them.

Additional context.
arxiv paper: https://arxiv.org/abs/2311.13600

[patrickvonplaten]

@younesbelkada @pacman100 is this something you are looking into?

[sayakpaul]

I think @pacman100 is already looking into it.

[Xynonners]

wouldn't this make more sense in https://github.com/huggingface/peft though?

since it seems to be a training method / pseudo-new LoRA adapter (only for training).

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.