[PEFT] set_adapters with weights argument currently broken

[younesbelkada]

Describe the bug

Currently set_adapters with weights argument is broken in diffusers. In fact set_adapters will correctly call module.scale_layer that currently overrides the scale value with the scaled value. But during inference, that information is being lost because we re-compute the original scale here: https://github.com/huggingface/peft/blob/45565f4357e24177020b12f43373c962497d82a2/src/peft/tuners/lora/layer.py#L163

We plan to fix it on PEFT side huggingface/peft#1028 but requires to upstream the changes on diffusers

Reproduction

Notebook shared in #5395

Logs

No response

System Info

xxx

Who can help?

@younesbelkada @sayakpaul @pacman100