DiffusionPipeline.from_pretrained SDXL model: Input type and bias type should be the same

[caldwecg]

Describe the bug

I am running into an error when generating images. I tried the below fix and it only let me generate the first image, subsequent images would fail to the error below.

for module in pipe.vae.modules():
if isinstance(module, torch.nn.Conv2d):
module.half()

Reproduction

Here is the code I am using:

`
from diffusers import DiffusionPipeline
import torch
model = "stabilityai/stable-diffusion-xl-base-1.0"
pipe = DiffusionPipeline.from_pretrained(model, torch_dtype = torch.float16,
)
pipe.to("cuda")
pipe.load_lora_weights("/mys3bucket/model/", weight_name = "pytorch_lora_weights.safetensors"
)

prompt = ["photo of (zwx child) in outer space, nasa suit, space station, RAW candid cinema, 16mm, color graded portra 400 film, remarkable color, ultra realistic, textured skin, remarkable detailed pupils, shot with cinematic camera"]
negative_prompt = "lowres"
generator = torch.Generator("cuda")

for seed in range(1):

image = pipe(prompt = prompt[seed], negative_prompt=negative_prompt, generator=generator, num_inference_steps=25)
image = image.images[0]
image.save(f"/mys3bucket/output_images/{seed}.png")

`

Logs

Traceback (most recent call last):
  File "/home/ubuntu/sdxl2.py", line 25, in <module>
    image = pipe(prompt = prompt[seed], negative_prompt=negative_prompt, generator=generator, num_inference_steps=25)
  File "/usr/local/lib/python3.10/dist-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/diffusers/pipelines/stable_diffusion_xl/pipeline_stable_diffusion_xl.py", line 847, in __call__
    image = self.vae.decode(latents / self.vae.config.scaling_factor, return_dict=False)[0]
  File "/usr/local/lib/python3.10/dist-packages/diffusers/utils/accelerate_utils.py", line 46, in wrapper
    return method(self, *args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/diffusers/models/autoencoder_kl.py", line 270, in decode
    decoded = self._decode(z).sample
  File "/usr/local/lib/python3.10/dist-packages/diffusers/models/autoencoder_kl.py", line 256, in _decode
    z = self.post_quant_conv(z)
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/conv.py", line 463, in forward
    return self._conv_forward(input, self.weight, self.bias)
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/conv.py", line 459, in _conv_forward
    return F.conv2d(input, weight, bias, self.stride,
RuntimeError: Input type (c10::Half) and bias type (float) should be the same

System Info

Running on AWS g4dn.xlarge instance

Torch + cuda version: 1.13.1+cu116
Nvidia Version: NVIDIA-SMI 510.108.03 Driver Version: 510.108.03 CUDA Version: 11.6

Who can help?

@william @patrickvonplaten, and @sayakpaul

[sayakpaul]

Could you provide a link to the LoRA safetensors file?

[caldwecg]

https://model-stage-stabledream.s3.us-east-1.amazonaws.com/test/model/pytorch_lora_weights.safetensors?response-content-disposition=inline&X-Amz-Security-Token=IQoJb3JpZ2luX2VjEJH%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FwEaCXVzLWVhc3QtMiJIMEYCIQCqKjon4s9rrl%2BjBjsTCk3AK3zsKkdFmiqin%2Febrzlc%2BAIhAOJYWkRW1tZUGKr169QY5vA89fMSxcoy89kehLIlxVk%2BKvsCCDoQARoMMzEyMzk4NDE0ODYxIgw8f%2BiopsWpaXjTRsIq2ALq9mQBWt%2FHBNuH4LfVzRyRaeoe%2FG%2FidSNazv1u%2B7l7%2B9eczPuqqqd86z4vjKRu5zjYhem%2FU6FgTZIyx9ZruexUgJ1hbc5hINGmaEYm5VtwdnkQkCqt9zsed0wZiMIGUvGiE1dF2%2FT%2FznAVv1TiExgrC2tC0jfNPlJBDjxH0T82vsSZcqOrv1U6o%2Fv4yVLDr2EpPz6fqsxIZF91%2Bc1FwupgXayoYoDEMDUyG%2BXHVRI%2BBdrm2F%2FLrMS4YiomdAzzVoBHE2DvSuU6wJU2dtMVQ0zsRRa7obw9bZkaGzhLzfaGbyFSvO1OI6QBphb3bOFwr2aY8qum%2BRkwSA97dR0T8XDMfx22U4obGlgTm28JPOmsPmxD9nTAvYkPf5QI3HOamBNGritL06bZlmHqLP3rh86qZJgO3ZPoZkzk%2FnT7Yvr%2FwPxlCHs%2B3rG6owVcwF1hcE25aqV%2FufIGkTCnwLamBjqyArkuaSZeP9xG9cdNXYgf71LAkdscHZoBSb3ldCCw%2BC0m5YOEd5XK%2FogVsoG5b1bNFpmqCx%2FL9CwzjjT7AVeOf82HHd5iGcSUTF94eCb8b6kvacZtXuDlyTAhgo%2FzPsolVzt7uKCW8%2Bmd6oQEbBL3qTJbKWEcGf5toYHxev00RTRN88Jgt5nfa%2FhA%2FUhDdFxHDQmz2UcRGarkqIY2rS632QvNeTXIgbZohbHWkdXlhwDR9T8Wn7OyuLu5BNxmHQ0H78DzaHDFePIP06%2B%2Bc4yUqW%2BSrcbmJwbn1FOi6oGURLv2mW9Shq0tcXYALMh82PBUPgLWtcTw%2FnDww7qBP4JTdvlrwz%2B3tCQT%2FxGuts7JIFQzFdUCEqrzxwcV3x7cReUNiRWrfbJ1IR5SM0axKg7nPsCF2A%3D%3D&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20230805T031002Z&X-Amz-SignedHeaders=host&X-Amz-Expires=43200&X-Amz-Credential=ASIAURPDFQQGXFYRRO7A%2F20230805%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Signature=1c63c8024c9530beab12fc7ace0cee287eac9bfb7c82ffe5bc8319da798629b9

[sayakpaul]

Could you host it on the Hugging Face Hub and provide a link to your repository? It's easier for us that way.

[caldwecg]

Sorry not sure what you mean by HF Hub- do you mean post as a topic on the HF forum?

[HuggingAha]

@caldwecg I have the same problem, how did you solve it?

[bharadwaj-kg]

@sayakpaul I am also facing same error when tried to finetune sdxl-base-1.0 model using "train_text_to_image_lora_sdxl.py". Is it a bug?

[sayakpaul]

Need simple and minimal snippet to reproduce the error. Ping @okotaku for that script too.

[bharadwaj-kg]

@okotaku , @sayakpaul . Here is the error I got when I tried to finetune sdxl-base-1.0 using "train_text_to_image_lora_sdxl.py". Below that are the training arguments.

Steps: 0%| | 0/1500 [00:00<?, ?it/s]Traceback (most recent call last):
File "/kaggle/working/diffusers/examples/text_to_image/train_text_to_image_lora_sdxl.py", line 1282, in
main(args)
File "/kaggle/working/diffusers/examples/text_to_image/train_text_to_image_lora_sdxl.py", line 1008, in main
model_input = vae.encode(pixel_values).latent_dist.sample()
File "/opt/conda/lib/python3.10/site-packages/diffusers/utils/accelerate_utils.py", line 46, in wrapper
return method(self, *args, **kwargs)
File "/opt/conda/lib/python3.10/site-packages/diffusers/models/autoencoder_kl.py", line 242, in encode
h = self.encoder(x)
File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/opt/conda/lib/python3.10/site-packages/diffusers/models/vae.py", line 111, in forward
sample = self.conv_in(sample)
File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/conv.py", line 463, in forward
return self._conv_forward(input, self.weight, self.bias)
File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/conv.py", line 459, in _conv_forward
return F.conv2d(input, weight, bias, self.stride,
RuntimeError: Input type (c10::Half) and bias type (float) should be the same

'diffusers/examples/text_to_image/train_text_to_image_lora_sdxl.py', '--pretrained_model_name_or_path=stabilityai/stable-diffusion-xl-base-1.0', '--dataset_name=my_hf_datapath', '--dataloader_num_workers=2', '--resolution=1024', '--center_crop', '--train_batch_size=1', '--gradient_accumulation_steps=4', '--max_train_steps=1500', '--learning_rate=1e-04', '--max_grad_norm=1', '--lr_scheduler=cosine', '--lr_warmup_steps=0', '--output_dir=finetune-model', '--push_to_hub', '--checkpointing_steps=500', '--validation_prompt=myprompt', '--seed=1337']' returned non-zero exit status 1.