LCM sampler support [very fast and low step generation]

[Amin456789]

HI!
can someone please implement this sampler and make a script to convert its model to ggml please? @ursg @leejet

u only need 2-4 steps to generate amazing images! we won't be needing for over 20 steps anymore, it works with cpu too, it has its own model so it has to be converted alongside the sampler itself, please see this here:

https://github.com/luosiallen/latent-consistency-model

it will be amazing with cpp they get along a lot :D

kind regards

[Green-Sky]

that looks very impressive. 2-4 steps would be huge.

[Amin456789]

yes, it will be great if we have it here!

[leejet]

I just added support for the Latent Consistency Models Sampler. You can pull the latest code and try the following steps:

1. Use the script at https://github.com/huggingface/diffusers/blob/main/scripts/convert_diffusers_to_original_stable_diffusion.py to convert the safetensors from https://huggingface.co/SimianLuo/LCM_Dreamshaper_v7 in the diffusers format to the original stable diffusion weights.
2. Then, use the script at https://github.com/leejet/stable-diffusion.cpp/blob/master/models/convert.py to convert the original stable diffusion weights into ggml weights.
3. Now, you can use sd.cpp to run Latent Consistency Models. Please note, it's advisable to set --cfg-scale to 1.0 instead of the default 7.0. For --steps, a range of 2-8 steps is recommended. Use --sampling-method with lcm (although euler_a should work as well; the difference between them is minimal).

Here's an example:

./bin/sd -m ../models/LCM_Dreamshaper_v7_4k-ggml-model-f16.bin -p "Self-portrait oil painting, a beautiful cyborg with golden hair, 8k" --steps 4 --cfg-scale 1.0 --sampling-method lcm

[HoiV]

You are correct - they are safetensors. However using convert.exe straight on the safetensors file generate an empty gguf file (I made sure to sync all modules and rebuild everything and still hit the same issue #92 - converting of safetensors not working):

convert.exe LCM_Dreamshaper_v7_4k.safetensors -t f16
loading model 'LCM_Dreamshaper_v7_4k.safetensors'
model type: checkpoint
Stable Diffusion 1.x - LCM_Dreamshaper_v7_4k.safetensors
preprocessing 0 tensors
using embedded vocab
converting 0 tensors
alphas_cumprod computed

CLIP Model Tensor count: 0
UNET Model Tensor count: 0
VAE Model Tensor count: 0

saving gguf file
model saved 'LCM_Dreamshaper_v7_4k-f16.gguf' correctly.

I then switched to the Python script: [convert_diffusers_to_original_stable_diffusion.py] to get the ckpt file per the instructions on top of the thread, I was able to convert it to gguf format successfully.

[FSSRepo]

@HoiV you can try with my PR #88, I think I already fixed that issue with some changes I made to the converter since I also encountered a safetensor file that prevented me from converting it.

[HoiV]

I have picked up this branch: taesd-impl and synced to this commit 2d760b7.

After rebuilding convert.exe and ran it over the current LCM_Dreamshaper_v7_4k.safetensors file, I no longer get an empty file but the new GGUF file is a lot smaller than the one you shared with us (which worked great by the way - thank you!).

This is the new GGUF when I used the bits from PR #88:
1,720,980,672 LCM_Dreamshaper_v7_4k-f16.gguf

And I still get a lot of the same errors of tensor ... not in model file.

[FSSRepo]

After rebuilding convert.exe and ran it over the current LCM_Dreamshaper_v7_4k.safetensors file, I no longer get an empty file but the new GGUF file is a lot smaller than the one you shared with us (which worked great by the way - thank you!).

This is the new GGUF when I used the bits from PR #88:
1,720,980,672 LCM_Dreamshaper_v7_4k-f16.gguf

And I still get a lot of the same errors of tensor ... not in model file.

This is not a converter error; you're trying to convert a safetensors file that only contains the diffusion model tensors (missing clip and vae tensors), meaning it's incomplete. I also downloaded that mentioned file, thinking it was the complete model. That's the downside of diffuser architecture models—they're delivered in parts instead of a complete file. It's quite confusing.

[HoiV]

Thank you for pointing that out. It looks like that safetensors is only the unet portion. How did you get the full set or how did you manage to create that GGUF file from a complete set given there is no release of a safetensors version that includes everything from HF. Thanks again.

[juniofaathir]

I keep got this error message:
FileNotFoundError: [Errno 2] No such file or directory: '/LCM_Dreamshaper_v7/unet/diffusion_pytorch_model.bin'
Anyone know what happen?

I've downloaded all the 3 folders and the .safetensors files. All of them I put it in /LCM_Dreamshaper_v7 folder with unet, vae, and text_encoder sub-folders

[leejet]

Which script did you use for the conversion?

[juniofaathir]

I used and follow everything from your link above

[leejet]

It looks like you didn't specify the correct model path. /LCM_Dreamshaper_v7 is a directory under the root directory, are you sure it's not ./LCM_Dreamshaper_v7，a directory under your current working directory?

    # Path for safetensors
    unet_path = osp.join(args.model_path, "unet", "diffusion_pytorch_model.safetensors")
    vae_path = osp.join(args.model_path, "vae", "diffusion_pytorch_model.safetensors")
    text_enc_path = osp.join(args.model_path, "text_encoder", "model.safetensors")

    # Load models from safetensors if it exists, if it doesn't pytorch
    if osp.exists(unet_path):
        unet_state_dict = load_file(unet_path, device="cpu")
    else:
        unet_path = osp.join(args.model_path, "unet", "diffusion_pytorch_model.bin")
        unet_state_dict = torch.load(unet_path, map_location="cpu")

[juniofaathir]

Sorry, because I'm still confused, as english is not my main language. But, let me give you an idea of where I put the LCM_Dreamshaper_v7 and all of it's required files:

All above is downloaded from SamianLuo's HF repo. And this is my cli, how I run it >> python3 convert_diffusers_to_original_stable_diffusion.py --model_path /LCM_Dreamshaper_v7 --checkpoint_path /home/mypc --use_safetensors

[leejet]

Replace /LCM_Dreamshaper_v7 with ./LCM_Dreamshaper_v7

python3 convert_diffusers_to_original_stable_diffusion.py --model_path ./LCM_Dreamshaper_v7 --checkpoint_path /home/mypc --use_safetensors

[leejet]

I just added LoRA support, now we can specify the use of LCM-LoRA through the prompt. Any SD1.x model should be adaptable.
https://github.com/leejet/stable-diffusion.cpp#with-lora

Here's a simple example:

./bin/sd -m ../models/v1-5-pruned-emaonly-ggml-model-f16.bin -p "a lovely cat<lora:lcm-lora-sdv1-5:1>" --steps 4 --lora-model-dir ../models -v --cfg-scale 1

[Green-Sky]

Btw, I tried other loras, and they work great too 👍

[leejet]

Support for multiple LoRAs has already been implemented. You can specify multiple LoRAs at the prompt, similar to how it's done in sd-webui.

./bin/sd -m ../models/v1-5-pruned-emaonly-ggml-model-f16.bin -p "a lovely cat<lora:lcm-lora-sdv1-5:1><lora:marblesh:0.6>" --steps 4 --lora-model-dir ../models -v --cfg-scale 1

[Green-Sky]

not quiet, the lora prompt extractor is faulty for me and only extracts the first lora. see 9a9f3da#r132974149

[Green-Sky]

with my fix:
a lovely cat<lora:lcm-lora-sdv1-5:1><lora:LuisapPixelArtLORA_v1:1.5>

[leejet]

Thanks for the feedback, I have synced your fix to master.

[Amin456789]

thanks guys! u rock!
can't wait for SVD and taesd to be released too!

[HoiV]

[like] Hoi Vo reacted to your message:

…

________________________________ From: Steward Garcia ***@***.***> Sent: Tuesday, November 28, 2023 10:11:04 AM To: leejet/stable-diffusion.cpp ***@***.***> Cc: Hoi Vo ***@***.***>; Mention ***@***.***> Subject: Re: [leejet/stable-diffusion.cpp] LCM sampler support [very fast and low step generation] (Discussion #76) @HoiV<https://github.com/HoiV> you can try with my PR #88<#88>, I think I already fixed that issue with some changes I made to the converter since I also encountered a safetensor file that prevented me from converting it. — Reply to this email directly, view it on GitHub<#76 (reply in thread)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AIWGROEXTVIMCGEFYTD2F23YGW2DRAVCNFSM6AAAAAA6MDDXFWVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM3TMOJQGU4TA>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[Amin456789]

i think it will be great if someone upload the converted fp16 gguf of lcm model in huggingface and put the link here for everybody

[FSSRepo]

I will do it

[Amin456789]

thanks a lot!

[FSSRepo]

LCM Dreamshaper GGUF (fp16 for now): https://huggingface.co/Steward/lcm-dreamshaper-v7-gguf

[Amin456789]

thank u so much!

[Amin456789]

@leejet could u please post this model in the read me? this lcm dreamshaper is woking very good and very important, im afraid many people won't see this one

[leejet]

This model doesn't have any differences from using safetensors directly.

[JohnClaw]

Please, upload to HF f32 version of LCM.gguf. And GGUFs for SD-Turbo and SDXL-Turbo are also needed.

[JohnClaw]

This model doesn't have any differences from using safetensors directly.

LCM.safetensors doesn't work with the latest version of SD.cpp

[leejet]

Where did you download 'LCM.safetensors' from? The file 'LCM_Dreamshaper_v7_4k.safetensors' available at 'https://huggingface.co/SimianLuo/LCM_Dreamshaper_v7/blob/main/LCM_Dreamshaper_v7_4k.safetensors' contains only the weights for the UNet model and cannot be used directly.

[JohnClaw]

Where did you download 'LCM.safetensors' from? The file 'LCM_Dreamshaper_v7_4k.safetensors' available at 'https://huggingface.co/SimianLuo/LCM_Dreamshaper_v7/blob/main/LCM_Dreamshaper_v7_4k.safetensors' contains only the weights for the UNet model and cannot be used directly.

Thank you for the answer. Where can i find the suitable LCM.safetensors?

[dmikey]

LCM weights can be found here.

https://huggingface.co/latent-consistency/lcm-lora-sdv1-5
https://huggingface.co/latent-consistency/lcm-lora-sdxl/blob/main/pytorch_lora_weights.safetensors

[JohnClaw]

LCM weights can be found here.

These are loras. I meant standalone all-in-one safetensor or ckpt.

[HoiV]

Hi John, given we can currently consume .ckpt / .safetensors directly, I used this script (https://github.com/huggingface/diffusers/blob/main/scripts/convert_diffusers_to_original_stable_diffusion.py) to reconstruct the .ckpt from the original model from https://huggingface.co/SimianLuo/LCM_Dreamshaper_v7. From there you can go to any quantization you need from sd.cpp.