Pass the original image to the vanilla SD network for img2img to also use color

[CesarERamosMedina]

Hello! In addition to adding control (in my case, in addition to edge detection), I'd like to also keep the original colors of the original input image. I was thinking of doing this using SD1.5 img2img (which also allows me to modify the image strength) + Canny ControlNet, but I am too much of a noob to understand the implications of this. Has anyone tried keeping the original colors as well, and not only the extra condition of, for example, edges?

[aoirusann]

I think it's impossible, currently.
Actually, keeping original color means no change. Pictures are just arrays of colors.

Of course, I know you are saying a different thing: you want to keep the 'diffuse' color. (diffuse is a term in CG, means the light reflection ability of a surface)
What we want:
A red region in the original picture should also be a red region in new picture (though the new red is a bit different).

Sadly, as far as I know there is no ControlNet works for diffuse color.
But since ControlNet can accept any image as condition, I think maybe training such a model is possible. (not sure)
😸

---

Update

From Composer, I found a good image representation for color:

They call it 'palette'. Maybe it's a good input condition for training a ControlNet model.

What's more, Mikubill/sd-webui-controlnet#359 (comment) is working on adding Composer support.
Hoping Composer will release its 'A light-weighted Latent-Composer built upon Stable Diffusion 2.1' soon, and it may deal the problem.

[geroldmeisinger]

Controlnet color https://huggingface.co/thibaud/controlnet-sd21#color