[Bug]: CLIP Interrogation crashes

[steyRNG]

Is there an existing issue for this?

• I have searched the existing issues and checked the recent builds/commits

What happened?

Tried interrogating an Image in img2img with CLIP, but got this crash and in the prompts box.

Since my GPU is neither CUDA nor HIP due to being on Win11, this crash happens. Maybe put CPU as a Fallback Default?

Steps to reproduce the problem

1. Start webui-user.bat
2. Open IMG2IMG
3. Put any Picture in the Box
4. Press "Interrogate CLIP"
5. ???
6. Crash

What should have happened?

CLIP Output in Prompt, no Crash in Console.

Commit where the problem happens

c888249

What platforms do you use to access the UI ?

Windows

What browsers do you use to access the UI ?

Google Chrome

Command Line Arguments

--opt-split-attention-v1 --opt-sub-quad-attention --autolaunch --no-half --disable-nan-check --precision full

List of extensions

Console logs

load checkpoint from G:\Other\AI\stable-diffusion-webui-directml\models\BLIP\model_base_caption_capfilt_large.pth
Error interrogating
Traceback (most recent call last):
  File "G:\Other\AI\stable-diffusion-webui-directml\modules\interrogate.py", line 198, in interrogate
    caption = self.generate_caption(pil_image)
  File "G:\Other\AI\stable-diffusion-webui-directml\modules\interrogate.py", line 183, in generate_caption
    caption = self.blip_model.generate(gpu_image, sample=False, num_beams=shared.opts.interrogate_clip_num_beams, min_length=shared.opts.interrogate_clip_min_length, max_length=shared.opts.interrogate_clip_max_length)
  File "G:\Other\AI\stable-diffusion-webui-directml\repositories\BLIP\models\blip.py", line 156, in generate
    outputs = self.text_decoder.generate(input_ids=input_ids,
  File "G:\Other\AI\stable-diffusion-webui-directml\venv\lib\site-packages\torch\autograd\grad_mode.py", line 27, in decorate_context
    return func(*args, **kwargs)
  File "G:\Other\AI\stable-diffusion-webui-directml\venv\lib\site-packages\transformers\generation\utils.py", line 1518, in generate
    return self.greedy_search(
  File "G:\Other\AI\stable-diffusion-webui-directml\venv\lib\site-packages\transformers\generation\utils.py", line 2267, in greedy_search
    unfinished_sequences = input_ids.new(input_ids.shape[0]).fill_(1)
RuntimeError: new(): expected key in DispatchKeySet(CPU, CUDA, HIP, XLA, MPS, IPU, XPU, HPU, Lazy, Meta) but got: PrivateUse1

Additional information

CPU: 5950x
RAM: 32GB
GPU: 7900 XTX
VRAM: 24GB

[steyRNG]

Workaround btw is adding "--use-cpu interrogate" to the cmdline args for anyone having the same issue

[lshqqytiger]

I added a fall back to cpu.

[Miraihi]

--use-cpu interrogate doesn't work for me. Only solution I found is the "tagger" extension. It allows you to get booru-style tags for the prompt. Not CLIP but at least something, considering that the native DeepBooru doesn't work as well (Produces the same inaccurate set of tags for any image).

[lshqqytiger]

There is a same issue on DirectML repo, it was a problem of pytorch.
PyTorch team fixed it, but fixed version is not released yet.

[Omniphantasm]

Any work around for this that we can do on our end for now?

Kinda sucks not being able to interrogate 😆

[Miraihi]

Any work around for this that we can do on our end for now?

Kinda sucks not being able to interrogate 😆

Just what I've already mentioned above: "tagger" extension