Cant Load example from IP Adapters

[AeroDEmi]

System Info

I'm trying to run the following snippet with Diffusers 0.33.1 and Transformers 4.51.3

import  torch
from diffusers import FluxPipeline
from diffusers.utils import load_image

pipe = FluxPipeline.from_pretrained(
"black-forest-labs/FLUX.1-dev", torch_dtype=torch.bfloat16
).to("cuda")

image = load_image("[https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/flux_ip_adapter_input.jpg").resize((1024](https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/flux_ip_adapter_input.jpg%22).resize((1024), 1024))

pipe.load_ip_adapter(
"XLabs-AI/flux-ip-adapter",
weight_name="ip_adapter.safetensors",
image_encoder_pretrained_model_name_or_path="openai/clip-vit-large-patch14"
)
pipe.set_ip_adapter_scale(1.0)

It breaks when loading the ip adapter:
TypeError: CLIPVisionModelWithProjection.init() got an unexpected keyword argument 'dtype'

Who can help?

No response

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

Run the posted code

Expected behavior

No error from the CLIPVisionModelWithProjection.init()

[AeroDEmi]

I had to do this to fix it:

import torch
from diffusers import FluxPipeline, LCMScheduler
from transformers import CLIPVisionModelWithProjection

image_encoder = CLIPVisionModelWithProjection.from_pretrained(
    "openai/clip-vit-large-patch14",
    torch_dtype=torch.bfloat16
)

pipe: FluxPipeline = FluxPipeline.from_pretrained(
    "black-forest-labs/FLUX.1-dev",
    image_encoder=image_encoder,
    torch_dtype=torch.bfloat16,
).to("cuda")

pipe.load_ip_adapter("XLabs-AI/flux-ip-adapter", weight_name="ip_adapter.safetensors")

[vasqu]

This rather seems to be a diffusers issue imo since their pipeline passes an incorrect kwarg on model init.

I've found this PR huggingface/diffusers#11346 - does that fix your issue? Otherwise, I'd advise going to the diffusers repo directly instead.

[ricklancee]

I was running into the same issue and that PR huggingface/diffusers#11346 fixed the issue.

I had to change diffusers in my requirments.txt to git+https://github.com/huggingface/diffusers.git as it's not released yet.