Fixed LoraConfig alpha modification on add_weighted_adapter

[kovalexal]

Hi!

Currently adding a weighted adapter rewrites base lora alpha value like this:

import torch
from diffusers import StableDiffusionPipeline
from peft import get_peft_model
from peft.tuners.lora import LoraConfig

pipe = StableDiffusionPipeline.from_pretrained(
    "runwayml/stable-diffusion-v1-5",
    torch_dtype=torch.float16
).to("cuda")

config = LoraConfig(
    r=8,
    lora_alpha=32,
    target_modules=["to_q"]
)

pipe.unet = get_peft_model(pipe.unet, config)

print(pipe.unet.peft_config)
# {'default': LoraConfig(
# peft_type=<PeftType.LORA: 'LORA'>, base_model_name_or_path=None,
# revision=None, task_type=None, inference_mode=False, r=8, target_modules=['to_q']
# lora_alpha=32, lora_dropout=0.0, fan_in_fan_out=False, bias='none', modules_to_save=None,
# init_lora_weights=True, layers_to_transform=None, layers_pattern=None)}

pipe.unet.add_weighted_adapter(["default"], [0.5], "default_05")
print(pipe.unet.peft_config)
# {'default': LoraConfig(
# peft_type=<PeftType.LORA: 'LORA'>, base_model_name_or_path=None,
# revision=None, task_type=None, inference_mode=False, r=8, target_modules=['to_q'],
# lora_alpha=8, lora_dropout=0.0, fan_in_fan_out=False, bias='none', modules_to_save=None,
# init_lora_weights=True, layers_to_transform=None, layers_pattern=None),
# 'default_05': LoraConfig(
# peft_type=<PeftType.LORA: 'LORA'>, base_model_name_or_path=None,
# revision=None, task_type=None, inference_mode=False, r=8, target_modules=['to_q'],
# lora_alpha=8, lora_dropout=0.0, fan_in_fan_out=False, bias='none', modules_to_save=None,
# init_lora_weights=True, layers_to_transform=None, layers_pattern=None)}

This PR just fixes this small issue.

@pacman100 your review is kindly appreciated.

[HuggingFaceDocBuilderDev]

The documentation is not available anymore as the PR was closed or merged.

[BenjaminBossan]

Thanks for the PR. Would you mind adding a small test that fails with the current code and works with your fix? It could be based on the example you posted. That would be great.

[pacman100]

Thank you @kovalexal for resolving this issue, LGTM! 🤗

[kovalexal]

@BenjaminBossan I've added a small test that will fail if this issue shows up again.

@pacman100, thanks for review, ready to be merged 🤗

[pacman100]

Thank you @kovalexal for iterating!