How to use inpaint/outpaint models?

[WainWong]

I noticed that you have added support for Controlnet++, thank you very much for your work!

But I would also like to ask, in the new controlnet union, it is claimed that inpaint is supported, but I don't know how to use this feature through your node, can you please take the time to answer my query?

[WainWong]

I'm currently executing inpaint like this, but the drawing comes out as a solid black image

[liushh39]

have you solved this problem?

[WainWong]

liushh39

no

[hben35096]

Me too @WainWong

Fannovel16/comfyui_controlnet_aux#408

[Kosinkadink]

The ControlNet++ inpaint/outpaint probably needs a special preprocessor for itself. From some light testing I just did, if you provide an unprocessed image in, it results something that looks like the colors are inverted, and if you provide an inverted image, it looks like some channels might be switched around.

I'll look into the diffusers code sometime soon(ish) to see what the input is expected to look like. In the meantime, here are the wonky results if you try to put in a normal image and an inverted image:

[hben35096]

It seems that an RGB of 0,0,0 would be considered a mask area

[WainWong]

You could try this.

It seems that an RGB of 0,0,0 would be considered a mask area

Thanks, yes this works fine; but it looks like it's prone to getting redrawn if there are solid black values in the original image; hopefully there's a more elegant way to accomplish this

[hben35096]

You can also do this

[Kosinkadink]

It's not much of a workaround, as having 'black' be the trigger for masked area is already unusual in itself. Normal inpaint controlnets expect -1 for where they should be masked, which is what the controlnet-aux Inpaint Preprocessor returns. The fact that OG controlnets use -1 instead of 0s for the mask is a blessing in that they sorta work even if you don't provide an explicit noise mask, as -1 would not normally be a value encountered by anything.

I'm pretty sure the original union code assumes that there is a noise mask applied in addition to the black RGB values in the mask itself, as just having 0,0,0 values is obviously not gonna work compared to the OG -1,-1,-1 values. The pipeline used in the diffusers code also applies the mask to the latents themselves, so you'll likely need to set up noise_mask (inpaint related stuff) to use inpaint/outpaint as intended. As for testing things out on my end right now, 1) I am busy with other things and 2) I personally don't use SDXL at all, so it's not something I am personally passionate about getting on ASAP compared to other things. If someone wants to look at the original code to confirm, please do so and let me know. The preprocessor for turning a mask into black pixels is dead simple, so I can add it this weekend, but the noise mask node already exists for inpainting in vanilla ComfyUI. Not sure if I'd want to make a 'combined' preprocessor that would simplify inpainting/outpainting using this controlnet or if it would have other consequences, but I can look into that at some point this weekend too.

My guess for the 0,0,0 values instead of -1,-1,-1 values in union is that since it is trying to be an 'all-in-one' controlnet, it would absolutely mess up all the training if just one of the input types happened to use negative values while everything else was scaled for [0, 1]. Even for other controlnet inputs, if there are negatives present in the cond image, something goes either to inf or Nan and ends up returning a black image, so it's a quirk of the model.

TL;DR: in addition to needing a special preprocess node to making mask make the pixels 0,0,0 in the input image, inpaint/outpaint type seems to require that you follow extra inpaint steps like setting a noise mask on the latents in a way that would match up with the black pixel mask applied to the controlnet inputs. I've included screenshots of the vanilla ComfyUI nodes that would create a noise_mask on the latents.

@jokero3answer Hopefully this clears things up. Might be redundant to say after typing all this, but the black requirement has nothing to do with my code and is just the way promax 'repaint' mode was trained.

[Kosinkadink]

There is no unwillingness to expand here. I've been supporting Advanced-ControlNet for a year now and I don't intend on stopping anytime soon.

From my understanding, the inpaint for union just needs a noise mask applied to the latents, which ComfyUI already supports with native nodes, so it can be tested. If there is more than that needed and there is a side by side comparison in the results to show it, please do let me know and we can work on having it be added in. I tagged this issue with "help needed" for a reason, I haven't had the chance to get into the weeds yet of ProMax-only types (tile and repaint), and I'm not sure if the existing preprocessors in control-aux (control-aux is not my repo btw) do what union/promax expects as input and if more needs to be done other than the noise mask. PRs are welcome too, even if just to share snippets of code to inspire the actual implementation.

[pcgamercc]

Hello @Kosinkadink can I bump in this issue with inpaint/outpaint in this same thread?
is this a file versioning issue?
thanks

Error occurred when executing ACN_ControlNet++LoaderAdvanced:

ControlNet++ model 'controlnet-union-sdxl-1.safetensors' does not support control_type 'inpaint/outpaint'.

  File "C:\Users\GAMING PC\Documents\Ai_save\ComfyUI_windows_portable\ComfyUI\execution.py", line 317, in execute
    output_data, output_ui, has_subgraph = get_output_data(obj, input_data_all, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb)
                                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\GAMING PC\Documents\Ai_save\ComfyUI_windows_portable\ComfyUI\execution.py", line 192, in get_output_data
    return_values = _map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\GAMING PC\Documents\Ai_save\ComfyUI_windows_portable\ComfyUI\execution.py", line 169, in _map_node_over_list
    process_inputs(input_dict, i)
  File "C:\Users\GAMING PC\Documents\Ai_save\ComfyUI_windows_portable\ComfyUI\execution.py", line 158, in process_inputs
    results.append(getattr(obj, func)(**inputs))
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\GAMING PC\Documents\Ai_save\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-Advanced-ControlNet\adv_control\nodes_plusplus.py", line 28, in load_controlnet_plusplus
    controlnet.verify_control_type(name, plus_input)
  File "C:\Users\GAMING PC\Documents\Ai_save\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-Advanced-ControlNet\adv_control\control_plusplus.py", line 248, in verify_control_type
    raise Exception(f"ControlNet++ model '{model_name}' does not support control_type '{pp_input.control_type}'.")

[Kosinkadink]

You'll need the Promax model (union was the initial model, Promax was basically the v2 model that added additional control types).

[pcgamercc]

Thank you, you were bang on right. the promax file version was the issue.