[Bug]: CUFFT_INTERNAL_ERROR on ZLUDA when using inpaint+lama

[BrothaPitagoras]

Checklist

• The issue exists after disabling all extensions
• The issue exists on a clean installation of webui
• The issue is caused by an extension, but I believe it is caused by a bug in the webui
• The issue exists in the current version of the webui
• The issue has not been reported before recently
• The issue has been reported before but has not been fixed yet

What happened?

Inpaint+lama seems to be broken when using controlnet, the exact same issue has been reported but I believe it should've been reported here (link).

Am not able to run preprocessor and generation with this preprocessor is broken and ignores image context.

I also had a similar issue when running Lama on SD.Next but it was the error "CUDNN_STATUS_INTERNAL_ERROR" instead if this is at all relevant.

Steps to reproduce the problem

1. Go to ControlNet Window
2. Choose preprocessor "inpaint_only+lama"
3. Choose your model
4. Draw a mask anywhere on input image for inpainting
5. Either press generate or press the preprocessing button

What should have happened?

Should've generated the image without issue, instead ignores masked area and uses prompt in whole image

What browsers do you use to access the UI ?

Brave

Sysinfo

sysinfo-2024-11-14-00-09.json

Console logs

*** Error running process: C:\Zluda-A1111\stable-diffusion-webui-directml\extensions\sd-webui-controlnet\scripts\controlnet.py
    Traceback (most recent call last):
      File "C:\Zluda-A1111\stable-diffusion-webui-directml\modules\scripts.py", line 784, in process
        script.process(p, *script_args)
      File "C:\Zluda-A1111\stable-diffusion-webui-directml\extensions\sd-webui-controlnet\scripts\controlnet.py", line 1245, in process
        self.controlnet_hack(p)
      File "C:\Zluda-A1111\stable-diffusion-webui-directml\extensions\sd-webui-controlnet\scripts\controlnet.py", line 1230, in controlnet_hack
        self.controlnet_main_entry(p)
      File "C:\Zluda-A1111\stable-diffusion-webui-directml\extensions\sd-webui-controlnet\scripts\controlnet.py", line 995, in controlnet_main_entry
        controls, hr_controls = list(zip(*[preprocess_input_image(img) for img in optional_tqdm(input_images)]))
      File "C:\Zluda-A1111\stable-diffusion-webui-directml\extensions\sd-webui-controlnet\scripts\controlnet.py", line 995, in <listcomp>
        controls, hr_controls = list(zip(*[preprocess_input_image(img) for img in optional_tqdm(input_images)]))
      File "C:\Zluda-A1111\stable-diffusion-webui-directml\extensions\sd-webui-controlnet\scripts\controlnet.py", line 952, in preprocess_input_image
        detected_map, is_image = self.preprocessor[unit.module](
      File "C:\Zluda-A1111\stable-diffusion-webui-directml\extensions\sd-webui-controlnet\scripts\utils.py", line 81, in decorated_func
        return cached_func(*args, **kwargs)
      File "C:\Zluda-A1111\stable-diffusion-webui-directml\extensions\sd-webui-controlnet\scripts\utils.py", line 65, in cached_func
        return func(*args, **kwargs)
      File "C:\Zluda-A1111\stable-diffusion-webui-directml\extensions\sd-webui-controlnet\scripts\global_state.py", line 37, in unified_preprocessor
        return preprocessor_modules[preprocessor_name](*args, **kwargs)
      File "C:\Zluda-A1111\stable-diffusion-webui-directml\extensions\sd-webui-controlnet\scripts\processor.py", line 529, in lama_inpaint
        prd_color = model_lama(img_res)
      File "C:\Zluda-A1111\stable-diffusion-webui-directml\extensions\sd-webui-controlnet\annotator\lama\__init__.py", line 54, in __call__
        result = self.model(image_feed)[0]
      File "C:\Zluda-A1111\stable-diffusion-webui-directml\venv\lib\site-packages\torch\nn\modules\module.py", line 1511, in _wrapped_call_impl
        return self._call_impl(*args, **kwargs)
      File "C:\Zluda-A1111\stable-diffusion-webui-directml\venv\lib\site-packages\torch\nn\modules\module.py", line 1520, in _call_impl
        return forward_call(*args, **kwargs)
      File "C:\Zluda-A1111\stable-diffusion-webui-directml\extensions\sd-webui-controlnet\annotator\lama\saicinpainting\training\modules\ffc.py", line 419, in forward
        return self.model(input)
      File "C:\Zluda-A1111\stable-diffusion-webui-directml\venv\lib\site-packages\torch\nn\modules\module.py", line 1511, in _wrapped_call_impl
        return self._call_impl(*args, **kwargs)
      File "C:\Zluda-A1111\stable-diffusion-webui-directml\venv\lib\site-packages\torch\nn\modules\module.py", line 1520, in _call_impl
        return forward_call(*args, **kwargs)
      File "C:\Zluda-A1111\stable-diffusion-webui-directml\venv\lib\site-packages\torch\nn\modules\container.py", line 217, in forward
        input = module(input)
      File "C:\Zluda-A1111\stable-diffusion-webui-directml\venv\lib\site-packages\torch\nn\modules\module.py", line 1511, in _wrapped_call_impl
        return self._call_impl(*args, **kwargs)
      File "C:\Zluda-A1111\stable-diffusion-webui-directml\venv\lib\site-packages\torch\nn\modules\module.py", line 1520, in _call_impl
        return forward_call(*args, **kwargs)
      File "C:\Zluda-A1111\stable-diffusion-webui-directml\extensions\sd-webui-controlnet\annotator\lama\saicinpainting\training\modules\ffc.py", line 337, in forward
        x_l, x_g = self.conv1((x_l, x_g))
      File "C:\Zluda-A1111\stable-diffusion-webui-directml\venv\lib\site-packages\torch\nn\modules\module.py", line 1511, in _wrapped_call_impl
        return self._call_impl(*args, **kwargs)
      File "C:\Zluda-A1111\stable-diffusion-webui-directml\venv\lib\site-packages\torch\nn\modules\module.py", line 1520, in _call_impl
        return forward_call(*args, **kwargs)
      File "C:\Zluda-A1111\stable-diffusion-webui-directml\extensions\sd-webui-controlnet\annotator\lama\saicinpainting\training\modules\ffc.py", line 304, in forward
        x_l, x_g = self.ffc(x)
      File "C:\Zluda-A1111\stable-diffusion-webui-directml\venv\lib\site-packages\torch\nn\modules\module.py", line 1511, in _wrapped_call_impl
        return self._call_impl(*args, **kwargs)
      File "C:\Zluda-A1111\stable-diffusion-webui-directml\venv\lib\site-packages\torch\nn\modules\module.py", line 1520, in _call_impl
        return forward_call(*args, **kwargs)
      File "C:\Zluda-A1111\stable-diffusion-webui-directml\extensions\sd-webui-controlnet\annotator\lama\saicinpainting\training\modules\ffc.py", line 275, in forward
        out_xg = self.convl2g(x_l) * l2g_gate + self.convg2g(x_g)
      File "C:\Zluda-A1111\stable-diffusion-webui-directml\venv\lib\site-packages\torch\nn\modules\module.py", line 1511, in _wrapped_call_impl
        return self._call_impl(*args, **kwargs)
      File "C:\Zluda-A1111\stable-diffusion-webui-directml\venv\lib\site-packages\torch\nn\modules\module.py", line 1520, in _call_impl
        return forward_call(*args, **kwargs)
      File "C:\Zluda-A1111\stable-diffusion-webui-directml\extensions\sd-webui-controlnet\annotator\lama\saicinpainting\training\modules\ffc.py", line 198, in forward
        output = self.fu(x)
      File "C:\Zluda-A1111\stable-diffusion-webui-directml\venv\lib\site-packages\torch\nn\modules\module.py", line 1511, in _wrapped_call_impl
        return self._call_impl(*args, **kwargs)
      File "C:\Zluda-A1111\stable-diffusion-webui-directml\venv\lib\site-packages\torch\nn\modules\module.py", line 1520, in _call_impl
        return forward_call(*args, **kwargs)
      File "C:\Zluda-A1111\stable-diffusion-webui-directml\extensions\sd-webui-controlnet\annotator\lama\saicinpainting\training\modules\ffc.py", line 86, in forward
        ffted = torch.fft.rfftn(x, dim=fft_dim, norm=self.fft_norm)
    RuntimeError: cuFFT error: CUFFT_INTERNAL_ERROR

Additional information

No response

[BrothaPitagoras]

forgot to say sysinfo does not have controlnet in extensions because i uninstalled A1111 after this issue happened to try SD.Next instead, since something similar happened in SD.Next I decided to open the issue here since I prefer this UI

[ride5k]

would love a fix for this as well!