[Bug]: Inpaint upload fails with "ValueError: images do not match"

[wmjdgla]

Checklist

• The issue exists after disabling all extensions
• The issue exists on a clean installation of webui
• The issue is caused by an extension, but I believe it is caused by a bug in the webui
• The issue exists in the current version of the webui
• The issue has not been reported before recently
• The issue has been reported before but has not been fixed yet

What happened?

"Inpaint upload" feature of img2img fails to generate.
Canvas in UI is blank with the following line below it: "ValueError: images do not match".
Happens whenever "Inpaint area" is set to "Only masked" and dimensions of base image, mask, and "Resize to" field are all different from one another. e.g. base image: 1024x1024, mask: 800x800, "Resize to": 512x512.
Generation succeeds if the dimensions are all the same. e.g. base image: 1024x1024, mask: 1024x1024, "Resize to": 1024x1024.
Generation also succeeds if the dimensions differ but "Inpaint area" is set to "Whole picture".

Steps to reproduce the problem

1. Select img2img > Generation > Inpaint upload
2. Change "Inpaint area" to "Only masked", leave all other settings as default
3. No prompt. For the base image and mask image, upload any image fulfilling the abovementioned conditions. Repro images (base mask) - they're just blank images created in mspaint.
4. Click "Generate"

What should have happened?

Image is generated successfully.

What browsers do you use to access the UI ?

Microsoft Edge

Sysinfo

sysinfo-2023-12-16-13-58.json

Console logs

venv "D:\Dev-ImageGen\stable-diffusion-webui\venv\Scripts\Python.exe"
==============================================================================================================
INCOMPATIBLE PYTHON VERSION

This program is tested with 3.10.6 Python, but you have 3.11.7.
If you encounter an error with "RuntimeError: Couldn't install torch." message,
or any other error regarding unsuccessful package (library) installation,
please downgrade (or upgrade) to the latest version of 3.10 Python
and delete current Python and "venv" folder in WebUI's directory.

You can download 3.10 Python from here: https://www.python.org/downloads/release/python-3106/

Alternatively, use a binary release of WebUI: https://github.com/AUTOMATIC1111/stable-diffusion-webui/releases

Use --skip-python-version-check to suppress this warning.
==============================================================================================================
Python 3.11.7 (tags/v3.11.7:fa7a6f2, Dec  4 2023, 19:24:49) [MSC v.1937 64 bit (AMD64)]
Version: v1.7.0
Commit hash: cf2772fab0af5573da775e7437e6acdca424f26e
Launching Web UI with arguments: --xformers
*** "Disable all extensions" option was set, will not load any extensions ***
Loading weights [879db523c3] from D:\Dev-ImageGen\stable-diffusion-webui\models\Stable-diffusion\dreamshaper_8.safetensors
Running on local URL:  http://127.0.0.1:7860

To create a public link, set `share=True` in `launch()`.
Creating model from config: D:\Dev-ImageGen\stable-diffusion-webui\configs\v1-inference.yaml
Startup time: 14.5s (prepare environment: 4.0s, import torch: 3.2s, import gradio: 1.2s, setup paths: 1.5s, initialize shared: 2.3s, other imports: 0.8s, setup codeformer: 0.1s, scripts list_optimizers: 0.5s, create ui: 0.3s, gradio launch: 0.5s).
Applying attention optimization: xformers... done.
Model loaded in 3.2s (load weights from disk: 0.5s, create model: 0.5s, apply weights to model: 1.1s, calculate empty prompt: 1.0s).
*** Error completing request
*** Arguments: ('task(iiml33hiu8h6u1c)', 4, '', '', [], None, None, None, None, None, <PIL.Image.Image image mode=RGB size=1024x1024 at 0x12D8E9A59D0>, <PIL.Image.Image image mode=RGBA size=800x800 at 0x12D8EB3B350>, 20, 'DPM++ 2M Karras', 4, 0, 1, 1, 1, 7, 1.5, 0.75, 0, 512, 512, 1, 0, 1, 32, 0, '', '', '', [], False, [], '', <gradio.routes.Request object at 0x0000012D8FB534D0>, 0, False, '', 0.8, -1, False, -1, 0, 0, 0, '* `CFG Scale` should be 2 or lower.', True, True, '', '', True, 50, True, 1, 0, False, 4, 0.5, 'Linear', 'None', '<p style="margin-bottom:0.75em">Recommended settings: Sampling Steps: 80-100, Sampler: Euler a, Denoising strength: 0.8</p>', 128, 8, ['left', 'right', 'up', 'down'], 1, 0.05, 128, 4, 0, ['left', 'right', 'up', 'down'], False, False, 'positive', 'comma', 0, False, False, 'start', '', '<p style="margin-bottom:0.75em">Will upscale the image by the selected scale factor; use width and height sliders to set tile size</p>', 64, 0, 2, 1, '', [], 0, '', [], 0, '', [], True, False, False, False, 0, False) {}
    Traceback (most recent call last):
      File "D:\Dev-ImageGen\stable-diffusion-webui\modules\call_queue.py", line 57, in f
        res = list(func(*args, **kwargs))
                   ^^^^^^^^^^^^^^^^^^^^^
      File "D:\Dev-ImageGen\stable-diffusion-webui\modules\call_queue.py", line 36, in f
        res = func(*args, **kwargs)
              ^^^^^^^^^^^^^^^^^^^^^
      File "D:\Dev-ImageGen\stable-diffusion-webui\modules\img2img.py", line 238, in img2img
        processed = process_images(p)
                    ^^^^^^^^^^^^^^^^^
      File "D:\Dev-ImageGen\stable-diffusion-webui\modules\processing.py", line 734, in process_images
        res = process_images_inner(p)
              ^^^^^^^^^^^^^^^^^^^^^^^
      File "D:\Dev-ImageGen\stable-diffusion-webui\modules\processing.py", line 804, in process_images_inner
        p.init(p.all_prompts, p.all_seeds, p.all_subseeds)
      File "D:\Dev-ImageGen\stable-diffusion-webui\modules\processing.py", line 1454, in init
        image_masked.paste(image.convert("RGBA").convert("RGBa"), mask=ImageOps.invert(self.mask_for_overlay.convert('L')))
      File "D:\Dev-ImageGen\stable-diffusion-webui\venv\Lib\site-packages\PIL\Image.py", line 1732, in paste
        self.im.paste(im, box, mask.im)
    ValueError: images do not match

---

Additional information

Pinging @AUTOMATIC1111 as I think I've found the problematic code, and git blame shows you as the author.

Cause

Everything works if in repro step 2, "Inpaint area" was left as default ("Whole picture"). Attaching a debugger and comparing the variables' values during runtime shows the root cause to be in the following lines:

stable-diffusion-webui/modules/processing.py

Lines 1416 to 1454 in cf2772f

	if self.inpaint_full_res:
	self.mask_for_overlay = image_mask
	mask = image_mask.convert('L')
	crop_region = masking.get_crop_region(np.array(mask), self.inpaint_full_res_padding)
	crop_region = masking.expand_crop_region(crop_region, self.width, self.height, mask.width, mask.height)
	x1, y1, x2, y2 = crop_region
	mask = mask.crop(crop_region)
	image_mask = images.resize_image(2, mask, self.width, self.height)
	self.paste_to = (x1, y1, x2-x1, y2-y1)
	else:
	image_mask = images.resize_image(self.resize_mode, image_mask, self.width, self.height)
	np_mask = np.array(image_mask)
	np_mask = np.clip((np_mask.astype(np.float32)) * 2, 0, 255).astype(np.uint8)
	self.mask_for_overlay = Image.fromarray(np_mask)
	self.overlay_images = []
	latent_mask = self.latent_mask if self.latent_mask is not None else image_mask
	add_color_corrections = opts.img2img_color_correction and self.color_corrections is None
	if add_color_corrections:
	self.color_corrections = []
	imgs = []
	for img in self.init_images:
	# Save init image
	if opts.save_init_img:
	self.init_img_hash = hashlib.md5(img.tobytes()).hexdigest()
	images.save_image(img, path=opts.outdir_init_images, basename=None, forced_filename=self.init_img_hash, save_to_dirs=False)
	image = images.flatten(img, opts.img2img_background_color)
	if crop_region is None and self.resize_mode != 3:
	image = images.resize_image(self.resize_mode, image, self.width, self.height)
	if image_mask is not None:
	image_masked = Image.new('RGBa', (image.width, image.height))
	image_masked.paste(image.convert("RGBA").convert("RGBa"), mask=ImageOps.invert(self.mask_for_overlay.convert('L')))

For "Whole picture", execution would enter the else case at L1426. There:

• the image mask would be resized before being assigned to self.mask_for_overlay
• crop_region is not set, as opposed to the earlier if case

When execution continues to the if check at L1449, the, conditions evaluate to true and the image is resized. At this point the dimensions of both image and self.mask_for_overlay match, and image_masked.paste() at L1454 succeeds.

But for "Only masked", execution would enter the if case at L1416. There:

• the image mask is assigned to self.mask_for_overlay before being resized
• crop_region is set

Now when execution continues to the if check at L1449, the, conditions evaluate to false (because crop_region is no longer None) and the image is not resized. At this point the dimensions of image and self.mask_for_overlay don't match, and image_masked.paste() throws ValueError: images do not match.

Solution

I made a quick change to resize the image mask first before assigning it to self.mask_for_overlay, and removed the crop_region is None condition at L1449. This fixed the exception and the output is generated successfully, but it looks rather off, so this is definitely not a proper fix. Also the fact that self.inpaint_full_res is true when "Only masked" is selected and false for "Whole picture" also seems off (from the variable name, I would have thought that it should have been the opposite). The whole thing is rather complex and I don't understand half of what's going on in the lines above, so I'll have to leave it to you to fix it.