[LoRA] fix adapter movement when using DoRA.

[sayakpaul]

What does this PR do?

To confirm, run: pytest tests/lora/test_lora_layers_sd.py::StableDiffusionLoRATests::test_integration_move_lora_cpu after replacing "runwayml/stable-diffusion-v1-5" with "Jiali/stable-diffusion-1.5" in the tests/lora/test_lora_layers_sd.py file.

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[BenjaminBossan]

Ah yes, nice catch, LGTM.

[yiyixuxu]

tests/lora/test_lora_layers_sd.py::StableDiffusionLoRATests::test_integration_move_lora_cpu after replacing "runwayml/stable-diffusion-v1-5" with "Jiali/stable-diffusion-1.5" in the tests/lora/test_lora_layers_sd.py file.

is this test supposed to fail at main right now? (after replace the checkpoint)?

[sayakpaul]

Yes, it'd fail with the replacement. However, previously, it wasn't the case. @BenjaminBossan I wonder if something changed on the peft side?

[BenjaminBossan]

Yes, it'd fail with the replacement. However, previously, it wasn't the case. @BenjaminBossan I wonder if something changed on the peft side?

Sorry, I'm missing some context. Under what circumstances is the test failing? Are there logs?

[sayakpaul]

Sorry, what I meant is that the test tests/lora/test_lora_layers_sd.py::StableDiffusionLoRATests::test_integration_move_lora_cpu was passing without this PR. But currently it fails (of course we have to replace the Runway SD v1.5 checkpoint with "Jiali/stable-diffusion-1.5" as Runway has taken it down).

The failure:

        # We will offload the first adapter in CPU and check if the offloading
        # has been performed correctly
>       pipe.set_lora_device(["adapter-1"], "cpu")

tests/lora/test_lora_layers_sd.py:126: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
src/diffusers/loaders/lora_base.py:702: in set_lora_device
    module.lora_magnitude_vector[adapter_name] = module.lora_magnitude_vector[
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = ModuleDict(), key = 'adapter-1'

    @_copy_to_script_wrapper
    def __getitem__(self, key: str) -> Module:
>       return self._modules[key]
E       KeyError: 'adapter-1'

But with peft=0.11.0 the test successfully passes. Hopefully, this is fully clear now?

[BenjaminBossan]

I see, thanks for the extra info. Indeed, this PR broke the test: huggingface/peft#1806. It's been merged for quite some time, not sure why we didn't notice earlier, is it because the SD model was removed? Anyway, the fix in this PR is correct given the change in PEFT.

[yiyixuxu]

thanks!