[WIP] Add sdxl lora support

[StAlKeR7779]

What type of PR is this? (check all applicable)

• Refactor
• Feature
• Bug Fix
• Optimization
• Documentation Update
• Community Node Submission

Have you discussed this change with the InvokeAI team?

• Yes
• No, because:

Have you updated all relevant documentation?

• Yes
• No

Description

Add lora loading for sdxl.
NOT TESTED - I run only 2 loras, please check more(including lycoris if they already exists).

QA Instructions, Screenshots, Recordings

https://civitai.com/models/118536/voxel-xl

Added/updated tests?

• Yes
• No

[psychedelicious]

Can you link to a LoRA that should work? nvm, it's just above the image, got it

[RyanJDick]

I tested this with nodes, and it was failing to correctly recognize the LoRA's base model type (it detected SD1 rather than SDXL) for LoRAs in <invokeai_root>/autoimport/lora/. I'm not very familiar with the model management code, but I think model_probe.py needs to be updated.

After forcing it to SDXL, I tested with voxel-xl and the official sd_xl_offset_example-lora_1.0.safetensors. Both worked, as expected.

[lstein]

Not working for me, but maybe I wired up the nodes wrong? I have never created a LoRA-containing graph before.

Here’s the relevant part of the node diagram:

I wired the model’s unet, clip and clip2 to the corresponding LoRA inputs, and wired the outputs from the LoRA to the positive and negative compels and the text2latents. I’ve tried several SDXL LoRAs, including the one referenced above, and never saw it taking effect even with a weight of 1.0

[lstein]

I’ll go ahead and update the model probe so that the correct base model is recognized during installation.

[lstein]

Tried again this morning with a fresh graph. Still no difference between having the LoRA loaded vs not loaded. The enclosed screenshot shows a graph that uses the "Voxel" LoRA. In the gallery the upper image is with the LoRA applied, and the lower one is without. I don't see an effect even when weight is cranked high. The log shows that the lora is being loaded.

[StAlKeR7779]

Tried again this morning with a fresh graph. Still no difference between having the LoRA loaded vs not loaded. The enclosed screenshot shows a graph that uses the "Voxel" LoRA. In the gallery the upper image is with the LoRA applied, and the lower one is without. I don't see an effect even when weight is cranked high. The log shows that the lora is being loaded.

In lora said that it specially trained for "voxel style" prompt, which i can't see in your prompt.
Also I never use invoke as reference, I always run a1111 lora in parallel and compare outputs.
//oh wait, it's sdxl topic, this one i not tested in a1111

[lstein]

Tried again this morning with a fresh graph. Still no difference between having the LoRA loaded vs not loaded. The enclosed screenshot shows a graph that uses the "Voxel" LoRA. In the gallery the upper image is with the LoRA applied, and the lower one is without. I don't see an effect even when weight is cranked high. The log shows that the lora is being loaded.

In lora said that it specially trained for "voxel style" prompt, which i can't see in your prompt. Also I never use invoke as reference, I always run a1111 lora in parallel and compare outputs. //oh wait, it's sdxl topic, this one i not tested in a1111

I tried with a few different LoRAs and never saw a difference, but I'll try again...

Success with "voxel style":

Also tested with https://civitai.com/models/119405 and https://civitai.com/models/117635?modelVersionId=127510

In some cases it helps to have the trigger term in the Style part of the prompt, and in other cases it doesn't matter.

[lstein]

LGTM. Thanks for the help figuring out how to run the SDXL LoRAs.