add padding_mask_crop to all inpaint pipelines #6360

rootonchair · 2023-12-27T16:28:57Z

What does this PR do?

Add padding_mask_crop to inpaint pipelines: SDXL, ControlNet, ControlNet SDXL

Fixes #6345 (issue)

Before submitting

This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
Did you read the contributor guideline?
Did you read our philosophy doc (important for complex PRs)?
Was this discussed/approved via a GitHub issue or the forum? Please add a link to it if that's the case.
Did you make sure to update the documentation with your changes? Here are the
documentation guidelines, and
here are tips on formatting docstrings.
Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

HuggingFaceDocBuilderDev · 2023-12-27T16:58:27Z

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

yiyixuxu

thank you!

sayakpaul

LGTM! Could I see some results too?

@yiyixuxu shouldn't we add tests too?

yiyixuxu · 2023-12-28T05:25:52Z

I think it is fine not to add tests for these auto1111 features. We are not currently testing all the value combinations for all pipeline arguments

rootonchair · 2023-12-28T09:07:15Z

I will try to get the result of padding_mask_crop with new pipeline. It will help if someone could provide an example code for running ControlNet inpaint

rootonchair · 2023-12-28T09:39:40Z

Here are the result of SDXL

import torch
from diffusers import AutoPipelineForInpainting
from diffusers.utils import load_image
from PIL import Image

model = "stabilityai/stable-diffusion-xl-base-1.0"
blur_factor = 33
seed = 0

img_url = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/sdxl-text2img.png"
mask_url = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/sdxl-inpaint-mask.png"
base = load_image(img_url)
mask = load_image(mask_url)

# create inpaint pipeline
pipe1 = AutoPipelineForInpainting.from_pretrained(model, torch_dtype=torch.float16).to('cuda')

# this is baseline, no mask blur, no inpant_full_res
generator = torch.Generator(device='cuda').manual_seed(seed)    
inpaint = pipe1('boat', image=base, mask_image=mask, strength=0.75,generator=generator).images[0]
inpaint.save(f'out_base.png')

# create blurred nask
mask_blurred = pipe1.mask_processor.blur(mask, blur_factor=blur_factor)
mask_blurred.save(f'mask_blurred.png')

# with mask blur
generator = torch.Generator(device='cuda').manual_seed(seed) 
inpaint = pipe1('boat', image=base, mask_image=mask_blurred, strength=0.75,generator=generator).images[0]
inpaint.save(f'out_mask_blur.png')

# with both mask_blur and inpaint_full_res
generator = torch.Generator(device='cuda').manual_seed(seed) 
inpaint = pipe1('boat', image=base, mask_image=mask_blurred, strength=0.75,generator=generator, padding_mask_crop=32).images[0]
inpaint.save(f'out_mask_blur_full_res.png')

base

mask

out_base

out_base_blur

out_mask_blur_full_res

rootonchair · 2024-01-01T14:14:27Z

Here the result of ControlNet sd1.5 inpaint

from diffusers import StableDiffusionControlNetInpaintPipeline, ControlNetModel, DDIMScheduler
from diffusers.utils import load_image
import numpy as np
import cv2
from PIL import Image
import torch

init_image = load_image(
    "https://huggingface.co/datasets/diffusers/test-arrays/resolve/main/stable_diffusion_inpaint/boy.png"
)
init_image = init_image.resize((512, 512))
init_image.save("input.png")

generator = torch.Generator(device="cpu").manual_seed(1)

mask_image = load_image(
    "https://huggingface.co/datasets/diffusers/test-arrays/resolve/main/stable_diffusion_inpaint/boy_mask.png"
)
mask_image.save("input_mask.png")
mask_image = mask_image.resize((512, 512))


def make_canny_condition(image):
    image = np.array(image)
    image = cv2.Canny(image, 100, 200)
    image = image[:, :, None]
    image = np.concatenate([image, image, image], axis=2)
    image = Image.fromarray(image)
    return image


control_image = make_canny_condition(init_image)
control_image.save("control_image.png")

controlnet = ControlNetModel.from_pretrained(
    "lllyasviel/control_v11p_sd15_canny", torch_dtype=torch.float16
)
pipe = StableDiffusionControlNetInpaintPipeline.from_pretrained(
    "runwayml/stable-diffusion-v1-5", controlnet=controlnet, torch_dtype=torch.float16
)

pipe.scheduler = DDIMScheduler.from_config(pipe.scheduler.config)
pipe.enable_model_cpu_offload()

# generate image
image = pipe(
    "a handsome man with ray-ban sunglasses",
    num_inference_steps=30,
    generator=generator,
    width=512,
    height=512,
    eta=1.0,
    image=init_image,
    mask_image=mask_image,
    control_image=control_image,
    padding_mask_crop=32
).images[0]

image.save("image_out.png")

Input image

Mask image

Control image

Run without padding_mask_crop

Run with padding_mask_crop

rootonchair · 2024-01-01T15:51:00Z

Finally, SDXL control net output

from diffusers import StableDiffusionXLControlNetInpaintPipeline, ControlNetModel, DDIMScheduler
from diffusers.utils import load_image
import cv2
from PIL import Image
import numpy as np
import torch

init_image = load_image(
    "https://huggingface.co/datasets/diffusers/test-arrays/resolve/main/stable_diffusion_inpaint/boy.png"
)
init_image = init_image.resize((1024, 1024))

generator = torch.Generator(device="cpu").manual_seed(1)

mask_image = load_image(
    "https://huggingface.co/datasets/diffusers/test-arrays/resolve/main/stable_diffusion_inpaint/boy_mask.png"
)
mask_image = mask_image.resize((1024, 1024))


def make_canny_condition(image):
    image = np.array(image)
    image = cv2.Canny(image, 100, 200)
    image = image[:, :, None]
    image = np.concatenate([image, image, image], axis=2)
    image = Image.fromarray(image)
    return image


control_image = make_canny_condition(init_image)

controlnet = ControlNetModel.from_pretrained(
    "diffusers/controlnet-canny-sdxl-1.0", torch_dtype=torch.float16
)
pipe = StableDiffusionXLControlNetInpaintPipeline.from_pretrained(
    "stabilityai/stable-diffusion-xl-base-1.0", controlnet=controlnet, torch_dtype=torch.float16
)

pipe.enable_model_cpu_offload()

# generate image
image = pipe(
    "a handsome man with ray-ban sunglasses",
    num_inference_steps=20,
    generator=generator,
    eta=1.0,
    image=init_image,
    mask_image=mask_image,
    control_image=control_image,
).images[0]
image.save("sdxl_controlnet_no_pad.png")


image = pipe(
    "a handsome man with ray-ban sunglasses",
    num_inference_steps=20,
    generator=generator,
    eta=1.0,
    width=1024,
    height=1024,
    image=init_image,
    mask_image=mask_image,
    control_image=control_image,
    padding_mask_crop=32
).images[0]
image.save("sdxl_controlnet_pad.png")

No padding_mask_crop

Add padding_mask_crop

patrickvonplaten · 2024-01-02T15:24:53Z

Can we run make style here?

yiyixuxu · 2024-01-02T17:38:06Z

we still see some of the astronauts in the sdxl example; wonder if it is related to this. #6417

can you run it again with the fix you proposed?

rootonchair · 2024-01-03T02:44:21Z

@yiyixuxu I think it due to the size of padding_mask_crop

padding_mask_crop=32

padding_mask_crop=128

rootonchair · 2024-01-03T02:49:46Z

Can we run make style here?

@patrickvonplaten I did run make style but it doesn't change any files. It seems like the error requires running make fix-copies but it changes many unrelated files

Run python utils/check_copies.py
Traceback (most recent call last):
  File "utils/check_copies.py", line 222, in <module>
    check_copies(args.fix_and_overwrite)
  File "utils/check_copies.py", line 206, in check_copies
    raise Exception(
Exception: Found the following copy inconsistencies:
- src/diffusers/pipelines/controlnet/pipeline_controlnet_inpaint.py: copy does not match pipelines.controlnet.pipeline_controlnet.StableDiffusionControlNetPipeline.prepare_image at line 884
Run `make fix-copies` or `python utils/check_copies.py --fix_and_overwrite` to fix them.

GoGiants1 · 2024-01-03T05:30:08Z

Hi @rootonchair
Some inpaint models have Unet that in_channels==9 (realistic vision 5.1, runwayml/stable-diffusion-inpainting).

in check_inputs
    raise ValueError(
ValueError: The UNet should have 4 input channels for inpainting mask crop, but has 9 input channels.

I got error from StableDiffusionControlNetInpaintPipeline when using realistic vision inpainting model..!

rootonchair · 2024-01-03T07:13:48Z

Hi @rootonchair Some inpaint models have Unet that in_channels==9 (realistic vision 5.1, runwayml/stable-diffusion-inpainting).
in check_inputs
    raise ValueError(
ValueError: The UNet should have 4 input channels for inpainting mask crop, but has 9 input channels.
I got error from StableDiffusionControlNetInpaintPipeline when using realistic vision inpainting model..!

@yiyixuxu should we change check_inputs condition? Running inpaint with 9 channel input seem to not raise any error

Below is the result of running padding_mask_crop with realisticVision

patrickvonplaten · 2024-01-05T10:48:40Z

Can we run make style here?

@patrickvonplaten I did run make style but it doesn't change any files. It seems like the error requires running make fix-copies but it changes many unrelated files

Run python utils/check_copies.py
Traceback (most recent call last):
  File "utils/check_copies.py", line 222, in <module>
    check_copies(args.fix_and_overwrite)
  File "utils/check_copies.py", line 206, in check_copies
    raise Exception(
Exception: Found the following copy inconsistencies:
- src/diffusers/pipelines/controlnet/pipeline_controlnet_inpaint.py: copy does not match pipelines.controlnet.pipeline_controlnet.StableDiffusionControlNetPipeline.prepare_image at line 884
Run `make fix-copies` or `python utils/check_copies.py --fix_and_overwrite` to fix them.

Actually you should indeed run make fix-copies, it's expected that this changes unrelated other files due to our # Copied from mechanism

rootonchair · 2024-01-05T11:45:17Z

@patrickvonplaten I see. I will make an update on that

yiyixuxu · 2024-01-08T18:12:19Z

@yiyixuxu should we change check_inputs condition? Running inpaint with 9 channel input seem to not raise any error

let's change it! you can change for inpaint pipeline too :)

let's also pass output_type to check_input() method and make sure we only support PIL output with padding_mask_crop feature see comments #6072 (comment)

yiyixuxu · 2024-01-08T18:14:34Z

great job!
I'm going to look into the sdxl example a little bit more to see what's going on there. Other than that looks good to merge soon:)

rootonchair · 2024-01-09T15:34:39Z

@yiyixuxu should we change check_inputs condition? Running inpaint with 9 channel input seem to not raise any error

let's change it! you can change for inpaint pipeline too :)

let's also pass output_type to check_input() method and make sure we only support PIL output with padding_mask_crop feature see comments #6072 (comment)

@yiyixuxu all done 😄

yiyixuxu · 2024-01-10T07:53:16Z

src/diffusers/pipelines/controlnet/pipeline_controlnet_inpaint.py

@@ -1264,6 +1298,13 @@ def __call__(
        else:
            batch_size = prompt_embeds.shape[0]

+        if padding_mask_crop is not None:


um just saw your issue #6435
maybe we need to move this code into prepare_control_image()?

see my comment here #6435 (comment)

I don't think it would work. width and height are still None in there. Do you think we should handle None in get_crop_region?

ok!
you can use self.image_processor. get_default_height_width(image) to get it

rootonchair · 2024-01-18T05:35:29Z

@yiyixuxu could you help me review this PR?

yiyixuxu

sorry I'm a little bit slow in reviewing this.
looks good and I left a few comments. Thanks again for working on this!

src/diffusers/pipelines/controlnet/pipeline_controlnet_inpaint.py

yiyixuxu · 2024-01-20T06:38:00Z

src/diffusers/pipelines/controlnet/pipeline_controlnet_inpaint.py

+                    f"The mask image should be a PIL image when inpainting mask crop, but is of type"
+                    f" {type(mask_image)}."
+                )
+            if output_type != "pil":


src/diffusers/pipelines/controlnet/pipeline_controlnet_inpaint.py

src/diffusers/pipelines/controlnet/pipeline_controlnet_inpaint_sd_xl.py

yiyixuxu · 2024-01-20T06:52:33Z

src/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion_inpaint.py

-            if self.unet.config.in_channels != 4:
+            if self.unet.config.in_channels != 4 and self.unet.config.in_channels != 9:
                raise ValueError(
-                    f"The UNet should have 4 input channels for inpainting mask crop, but has"
+                    f"The UNet should have 4 or 9 input channels for inpainting mask crop, but has"
                    f" {self.unet.config.in_channels} input channels."
                )


you can remove this warning

src/diffusers/pipelines/stable_diffusion_xl/pipeline_stable_diffusion_xl_inpaint.py

yiyixuxu · 2024-01-20T06:54:46Z

src/diffusers/pipelines/stable_diffusion_xl/pipeline_stable_diffusion_xl_inpaint.py

@@ -1527,10 +1559,22 @@ def denoising_value_valid(dnv):
        is_strength_max = strength == 1.0

        # 5. Preprocess mask and image
-        init_image = self.image_processor.preprocess(image, height=height, width=width)
+        if padding_mask_crop is not None:
+            crops_coords = self.mask_processor.get_crop_region(mask_image, width, height, pad=padding_mask_crop)


we don't need height, width = self.image_processor.get_default_height_width(image, height, width) here?

We don't need because it already been initialized: https://github.com/huggingface/diffusers/blob/main/src/diffusers/pipelines/stable_diffusion_xl/pipeline_stable_diffusion_xl_inpaint.py#L1445-L1446

Co-authored-by: YiYi Xu <yixu310@gmail.com>

rootonchair · 2024-01-21T15:39:26Z

@yiyixuxu fixed. Thank you for your reviews

yiyixuxu

thanks

* add padding_mask_crop --------- Co-authored-by: Sayak Paul <spsayakpaul@gmail.com> Co-authored-by: YiYi Xu <yixu310@gmail.com>

rootonchair added 3 commits December 27, 2023 23:24

add padding_mask_crop to sdxl inpaint

7e3e66f

add padding_mask_crop to controlnet

aa6e4ba

run make style

fdc7c5a

yiyixuxu approved these changes Dec 27, 2023

View reviewed changes

yiyixuxu requested a review from sayakpaul December 27, 2023 18:43

sayakpaul reviewed Dec 28, 2023

View reviewed changes

Merge branch 'main' into add_padding_mask_crop

bb1596e

crop control image also

001c514

rootonchair mentioned this pull request Jan 1, 2024

Correct how apply_overlay read crop_coords #6417

Merged

6 tasks

rootonchair added 2 commits January 1, 2024 18:18

check control_image input

6d1a3fb

crop control image sdxl inpaint pipeline

2fc751a

sayakpaul requested a review from yiyixuxu January 1, 2024 14:15

run make style

9d14a26

rootonchair mentioned this pull request Jan 3, 2024

Default Width, Height in ControlNet does not initialized #6435

Closed

remove copied from mechanism

1ce9ecd

change check inputs condition

a0bedc8

yiyixuxu reviewed Jan 10, 2024

View reviewed changes

patrickvonplaten requested a review from yiyixuxu January 15, 2024 13:56

handle None case for width height in controlnet

98c5b37

yiyixuxu reviewed Jan 20, 2024

View reviewed changes

rootonchair and others added 3 commits January 21, 2024 22:31

Update src/diffusers/pipelines/controlnet/pipeline_controlnet_inpaint.py

3fa0a3e

Co-authored-by: YiYi Xu <yixu310@gmail.com>

Apply suggestions from code review

3e00f00

Co-authored-by: YiYi Xu <yixu310@gmail.com>

remove warning

93bb80e

yiyixuxu approved these changes Jan 22, 2024

View reviewed changes

yiyixuxu merged commit 8e7bbfb into huggingface:main Jan 22, 2024
14 checks passed

rootonchair deleted the add_padding_mask_crop branch January 22, 2024 09:21

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add padding_mask_crop to all inpaint pipelines #6360

add padding_mask_crop to all inpaint pipelines #6360

rootonchair commented Dec 27, 2023

HuggingFaceDocBuilderDev commented Dec 27, 2023

yiyixuxu left a comment

sayakpaul left a comment

yiyixuxu commented Dec 28, 2023

rootonchair commented Dec 28, 2023

rootonchair commented Dec 28, 2023

rootonchair commented Jan 1, 2024

rootonchair commented Jan 1, 2024

patrickvonplaten commented Jan 2, 2024

yiyixuxu commented Jan 2, 2024 •

edited

Loading

rootonchair commented Jan 3, 2024

rootonchair commented Jan 3, 2024

GoGiants1 commented Jan 3, 2024 •

edited

Loading

rootonchair commented Jan 3, 2024

patrickvonplaten commented Jan 5, 2024

rootonchair commented Jan 5, 2024

yiyixuxu commented Jan 8, 2024 •

edited

Loading

yiyixuxu commented Jan 8, 2024

rootonchair commented Jan 9, 2024

yiyixuxu Jan 10, 2024

rootonchair Jan 11, 2024

yiyixuxu Jan 16, 2024

rootonchair commented Jan 18, 2024

yiyixuxu left a comment

yiyixuxu Jan 20, 2024

yiyixuxu Jan 20, 2024

yiyixuxu Jan 20, 2024

rootonchair Jan 21, 2024 •

edited

Loading

rootonchair commented Jan 21, 2024

yiyixuxu left a comment

add padding_mask_crop to all inpaint pipelines #6360

add padding_mask_crop to all inpaint pipelines #6360

Conversation

rootonchair commented Dec 27, 2023

What does this PR do?

Before submitting

Who can review?

HuggingFaceDocBuilderDev commented Dec 27, 2023

yiyixuxu left a comment

Choose a reason for hiding this comment

sayakpaul left a comment

Choose a reason for hiding this comment

yiyixuxu commented Dec 28, 2023

rootonchair commented Dec 28, 2023

rootonchair commented Dec 28, 2023

rootonchair commented Jan 1, 2024

rootonchair commented Jan 1, 2024

patrickvonplaten commented Jan 2, 2024

yiyixuxu commented Jan 2, 2024 • edited Loading

rootonchair commented Jan 3, 2024

rootonchair commented Jan 3, 2024

GoGiants1 commented Jan 3, 2024 • edited Loading

rootonchair commented Jan 3, 2024

patrickvonplaten commented Jan 5, 2024

rootonchair commented Jan 5, 2024

yiyixuxu commented Jan 8, 2024 • edited Loading

yiyixuxu commented Jan 8, 2024

rootonchair commented Jan 9, 2024

yiyixuxu Jan 10, 2024

Choose a reason for hiding this comment

rootonchair Jan 11, 2024

Choose a reason for hiding this comment

yiyixuxu Jan 16, 2024

Choose a reason for hiding this comment

rootonchair commented Jan 18, 2024

yiyixuxu left a comment

Choose a reason for hiding this comment

yiyixuxu Jan 20, 2024

Choose a reason for hiding this comment

yiyixuxu Jan 20, 2024

Choose a reason for hiding this comment

yiyixuxu Jan 20, 2024

Choose a reason for hiding this comment

rootonchair Jan 21, 2024 • edited Loading

Choose a reason for hiding this comment

rootonchair commented Jan 21, 2024

yiyixuxu left a comment

Choose a reason for hiding this comment

yiyixuxu commented Jan 2, 2024 •

edited

Loading

GoGiants1 commented Jan 3, 2024 •

edited

Loading

yiyixuxu commented Jan 8, 2024 •

edited

Loading

rootonchair Jan 21, 2024 •

edited

Loading