HiDream LoRA support

[vladmandic]

Describe the bug

i've tried loras currently available on civitai and most appear to load without any errors using load_lora_weights.
but later, they do not appear in get_list_adapters or get_active_adapters and do not seem to be applied to the model.
if loras were not loaded, i'd expect to see some error?
some return error Invalid LoRA checkpoint, but most do not return any error.

list of currently available loras on civitai: https://civitai.com/search/models?baseModel=HiDream&modelType=LORA&sortBy=models_v9

Reproduction

N/A

Logs

System Info

diffusers==e30d3bf5442fbdbee899e8a5da0b11b621d54f1b

Who can help?

@linoytsaban @sayakpaul

[linoytsaban]

Hey @vladmandic, we haven't yet added support for HiDream LoRAs that are not in diffusers format, we're working on it at the moment and should be supported soon :)

[vladmandic]

which is fine, but why does load_lora_weights returns without any errors for 80% of them?
if lora is not loaded and added to list, that method should not be a silent black hole.

[linoytsaban]

yeah it should throw an error if keys are incompatible, could you please provide code snippet for the LoRA you're trying to load that doesn't load and doesn't error either?

[vladmandic]

@linoytsaban almost every lora from the link i provided.
but heres a specific example: https://civitai.com/api/download/models/1686777?type=Model&format=SafeTensor
regarding code:

pipe.load_lora_weight(filename, adapter_name="test") # no error
print(pipe.get_list_adapters()) # empty list
pipe.set_adapters(adapter_names=["test"], adapter_weights=[1.0]) # fails as lora is not loaded

[sayakpaul]

What model is it? Full or Dev?

Could you please provide two things?

• LoRA file on the Hub (just makes it easier for the folks who use server machines to test things). It helps to view the structure of the LoRA file directly on the Hub.
• A minimal, fully reproducible snippet.

I did the following and it logged:

from diffusers import DiffusionPipeline
from transformers import PreTrainedTokenizerFast, LlamaForCausalLM
import torch 

repo_id = "HiDream-ai/HiDream-I1-Full"

tokenizer_4 = PreTrainedTokenizerFast.from_pretrained("meta-llama/Meta-Llama-3.1-8B-Instruct")
text_encoder_4 = LlamaForCausalLM.from_pretrained(
    "meta-llama/Meta-Llama-3.1-8B-Instruct",
    output_hidden_states=True,
    output_attentions=True,
    torch_dtype=torch.bfloat16,
)

pipeline = DiffusionPipeline.from_pretrained(
    repo_id, 
    tokenizer_4=tokenizer_4,
    text_encoder_4=text_encoder_4,
    torch_dtype=torch.bfloat16
).to("cuda")
pipeline.load_lora_weights("sayakpaul/different-lora-from-civitai", weight_name="486_hidream.safetensors")

Log:

No LoRA keys associated to HiDreamImageTransformer2DModel found with the prefix='transformer'. This is safe to ignore if LoRA state dict didn't originally have any HiDreamImageTransformer2DModel related params. You can also try specifying `prefix=None` to resolve the warning. Otherwise, open an issue if you think it's unexpected: https://github.com/huggingface/diffusers/issues/new

[vladmandic]

i used hidream-i1-full.
regarding upload on hub, i understand your position but do note that civitai is defacto standard for anything regarding image models and. if you cannot access the lora on civitai we have a bigger problem.

your snippet is fine.
btw, what level is that log message? because diffusers can be chatty and i disable everything below warning.

[sayakpaul]

regarding upload on hub, i understand your position but do note that civitai is defacto standard for anything regarding image models and. if you cannot access the lora on civitai we have a bigger problem.

Not to digress, https://huggingface.co/spaces/sayakpaul/civitai-to-hub could help.

btw, what level is that log message? because diffusers can be chatty and i disable everything below warning.

It's logger.warning():

diffusers/src/diffusers/loaders/peft.py

Line 418 in b4be422

logger.warning(

[mukundkhanna123]

Hi, I am facing the same issue. Here's a minimal reproducible example:

import torch
from diffusers import (
    AutoencoderKL,
    FlowMatchEulerDiscreteScheduler,
    HiDreamImagePipeline,
    HiDreamImageTransformer2DModel,
)
from transformers import PreTrainedTokenizerFast, LlamaForCausalLM

llama_repo = "meta-llama/Meta-Llama-3.1-8B-Instruct"
model_id = "HiDream-ai/HiDream-I1-Full"

tokenizer_4 = PreTrainedTokenizerFast.from_pretrained(
    llama_repo,
)
text_encoder_4 = LlamaForCausalLM.from_pretrained(
    llama_repo,
    output_hidden_states=True,
    output_attentions=True,
    torch_dtype=torch.bfloat16,
)

transformer = HiDreamImageTransformer2DModel.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    subfolder="transformer",
)
pipeline = HiDreamImagePipeline.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    tokenizer_4=tokenizer_4,
    text_encoder_4=text_encoder_4,
    transformer=transformer,
)
pipeline.load_lora_weights(
    "sayakpaul/different-lora-from-civitai",
    weight_name="486_hidream.safetensors",
)

print(pipeline.get_list_adapters())

This fails for me with the following error:

ValueError: `state_dict` should be empty at this point but has state_dict.keys()=dict_keys(['double_stream_blocks.0.block.adaLN_modulation.1.lora_A.weight', 'double_stream_blocks.0.block.adaLN_modulation.1.lora_B.weight', ... single_stream_blocks.9.block.ff_i.shared_experts.w3.lora_A.weight', 'single_stream_blocks.9.block.ff_i.shared_experts.w3.lora_B.weight'])

---

Additionally, when I load a LoRA that I trained myself (using diffusers), it doesn't throw an error — but:

• pipeline.get_list_adapters() shows an empty list,
• and the final generated image is identical to the base model output, suggesting that the LoRA was not actually applied during generation.

Would appreciate any help on this!

[sayakpaul]

@mukundkhanna123 that is because the LoRA isn't supported yet but will be supported soon. Additionally, if you use the LoRA mentioned in the snippet, it shouldn't error out. Instead, it should lead to the warning mentioned in #11383 (comment). Did you make any changes to the diffusers codebase?

Additionally, when I load a LoRA that I trained myself (using diffusers), it doesn't throw an error — but:

Can you please open a separate issue for that? Cc: @linoytsaban

[mukundkhanna123]

Okay I used the latest commit bd96a08 and it didn't error out with the lora mentioned in the snippet and just gave the warning. I am able to load the LoRA I trained as well ( trained with diffusers). However there is no change in the output image.

[sayakpaul]

I am able to load the LoRA I trained as well ( trained with diffusers). However, there is no change in the output image.

Could you please create a new issue for that and tag me and @linoytsaban? Linoy has been training a bunch of LoRAs with our script and the resultant LoRAs work. Here is an example: https://huggingface.co/linoyts/hidream-3dicon-lora