[Bug]: img2img loopback pushes images towards magenta aggressively

[sedetweiler]

What happened?

Every loopback I have done has resulted in an unusable image in as few as 35 batch counts as the images shift toward magenta quickly and do not recover. Tested repeatedly on different images with similar results.

Version

0.0.1 (Default)

What browsers are you seeing the problem on?

Chrome

Where are you running the webui?

Windows

Custom settings

CFG 5, Denoise 0.4, Steps 50, k_lms, loopback, No random seed

Relevant log output

No response

Code of Conduct

• I agree to follow this project's Code of Conduct

[TingTingin]

This is related to Sygil-Dev/stable-diffusion#153

[sedetweiler]

Here are a few examples of this in action.

[ddgond]

I think this is a separate issue that has to do with which direction the model tends to take images. Pyrocube seems to have fixed it in their repo according to discussion on Discord.

I'm gonna reopen this conversation real quick because I just implemented a correction for that in my img2img cycling implementation. Got a bit annoyed by the fact that everything becomes blue and magenta sooner or later.
TL;Dr is the following: take the first image, remap to YCbCr. ignore Y channel (that's just grayscale), we care about CbCr colorization. Take initial channel averages
For every image, apply a transfer function to remap the CbCr channel midpoints to the original midpoints before feeding it back in (can't use scaling straight-up because 0 needs to remain 0 and 1 needs to remain 1) - I'm just using a cubic spline function for it
let me grab the code real quick
If anyone here has a better idea for how this sorta thing should be done (I don't know too much about proper colorspace design), please @ me (:

^ Pyrocube

[Blue42hand]

The same kind of issue shows up in Disco Diffusion with some clip models. Since img2img loop-back is in a way simulating clip guided diffusion it may be something similar. In my experience the easiest fix was to put "magenta" as a negative weight in the prompt.

[sedetweiler]

I think this is a separate issue that has to do with which direction the model tends to take images. Pyrocube seems to have fixed it in their repo according to discussion on Discord.

I'm gonna reopen this conversation real quick because I just implemented a correction for that in my img2img cycling implementation. Got a bit annoyed by the fact that everything becomes blue and magenta sooner or later.
TL;Dr is the following: take the first image, remap to YCbCr. ignore Y channel (that's just grayscale), we care about CbCr colorization. Take initial channel averages
For every image, apply a transfer function to remap the CbCr channel midpoints to the original midpoints before feeding it back in (can't use scaling straight-up because 0 needs to remain 0 and 1 needs to remain 1) - I'm just using a cubic spline function for it
let me grab the code real quick
If anyone here has a better idea for how this sorta thing should be done (I don't know too much about proper colorspace design), please @ me (:

^ Pyrocube

This makes a lot of sense and keeps the color set from pushing artificially away from the intended set in the previous image.

[JustMaier]

Here's the specific commit that references the change:
Pyr-000/YetAnotherStableDiffusion@c7bf958

[bbecausereasonss]

How does incorporate pyrocubes change into HLKY? My results have been magenta and very dark even after 1'st iteration.

[JustMaier]

Someone with python know how will need to review the implementation I pointed to and apply a similar change to the img2img function in the webui.py file in this repo. Is this something you've got bandwidth for @hlky? I'd take a stab at it myself but python is relatively foreign to me. If you can give me some pointers though (places in your codebase I'll need to tweak etc) I'll give it a go.

[codedealer]

Whoever here has problem with images being pushed to magenta/blue try this https://github.com/codedealer/stable-diffusion-webui/tree/img2img-fix

1. Make sure you're pulling img2img-fix branch (or just copy the files) the sync procedure is the same as with this repo
2. run conda install scikit-image
3. launch webui.py and run img2img on Crop mode with loopback, no random seed.

This branch has Pyro's color correction implemented but I wasn't able to test it because I can't replicate the bug, not sure what it is so I need you to test it. Maybe see if you can the image below to magenta (I couldn't).

[codedealer]

This should be fixed now on master

[bagolyszkij]

Is this issue already handled somehow?
Its still a problem, pink is not my colour! :)
Thanks

[yiqiangjun]

这是来自QQ邮箱的假期自动回复邮件。   您好，我最近正在休假中，无法亲自回复您的邮件。我将在假期结束后，尽快给您回复。