add code for skipping CFG on early steps

[drhead]

Description

• Implements an option to skip CFG on early sampling steps as recommended in https://arxiv.org/abs/2404.07724
• Somewhat awkward to use as it stands. I'll probably change it to floating point for a percentage of sampling steps rather than discrete steps. done. Would recommend anyone that uses it uses it as a quicksetting.
• Probably shouldn't be used for img2img or inpainting (unless using almost 1.0 denoising strength because the assumptions of the paper will almost certainly not hold for those cases (haven't tested this though).
• Improves sample diversity, image quality, and in some cases converges faster. Also often makes higher CFG settings stable. Also, makes the skipped steps twice as fast since the second sample from CFG isn't being made.
• In some cases, may reduce prompt obedience. Particularly, this will also take away your negative prompt on early steps.
• Paper also recommends something similar to Negative Guidance Minimum Sigma, except which skips all steps in that interval instead of every other step. I'll look into adding an option to make NGMS skip all steps towards this end. done, this PR now fully implements the paper. Between the two options, I find I can often get away with skipping CFG on around 1/3 of total sampling steps.

Screenshots/videos:

25 steps, no skipped CFG steps:

25 steps, skipped CFG on first four steps:

50 steps, no CFG skipping:

50 steps, skipped CFG on first eight steps:

As you can see, the image with CFG skipping converged much faster. There are some noticeable side effects on image brightness (prompt on this image contained (dark,:1.3), so this is somewhat less dark than I would typically expect).

Checklist:

• I have read contributing wiki page
• I have performed a self-review of my own code
• My code follows the style guidelines
• My code passes tests

[v0xie]

Awesome PR!

Skip Early Cond should be written in metadata for reproducibility.

[diegocr]

Suggested change

	if shared.opts.skip_early_cond != 0. and self.step / self.total_steps <= shared.opts.skip_early_cond:
	skip_uncond = True
	x_in = x_in[:-batch_size]
	sigma_in = sigma_in[:-batch_size]

[diegocr]

Suggested change

	if (self.step % 2 or shared.opts.s_min_uncond_all) and s_min_uncond > 0 and sigma[0] < s_min_uncond and not is_edit_model:
	if ((self.step % 2 or shared.opts.s_min_uncond_all) and s_min_uncond > 0 and sigma[0] < s_min_uncond
	and not is_edit_model) or (
	shared.opts.skip_early_cond != 0. and self.step / self.total_steps <= shared.opts.skip_early_cond):

I think you meant to do that, at least i am getting the following error otherwise:

      File "..\modules\sd_samplers_cfg_denoiser.py", line 279, in forward
        denoised = self.combine_denoised(x_out, conds_list, uncond, 1.0)
      File "..\modules\sd_samplers_cfg_denoiser.py", line 75, in combine_denoised
        denoised[i] += (x_out[cond_index] - denoised_uncond[i]) * (weight * cond_scale)
    IndexError: index 0 is out of bounds for dimension 0 with size 0

Great PR btw, at times i get strange results but much faster overall :)

[drhead]

Something I would like more feedback on is whether people think it is better to have the option as discrete steps or having it as a float value. Or whether it is better to have it as something else entirely, like as a value of sigma.

Float from 0 to 1 is most consistent with other UI elements. Exact steps is more controllable but may be annoying to tweak between different schedules and step counts. Having it as a sigma will make behaviors more consistent across different schedules and makes it a lot easier to exclude from inpainting and is additionally the same thing that NGMS uses, but it would cause silly problems for zero SNR models because the slider would have to go up to 4500 (can gradio sliders be in log space?)

I have tried both of the others and have been unhappy with them overall, I think that sigma would probably be what I would lean towards switching to but I can't implement that change right now.

[Luke2642]

This is amazing, it's not often you stumble across a real game changer. It's an absolute must to add skip_early_cond to the quick settings and try it out!

Setting it to values like 0.3 on a 20 step DPM++ gen, it feels like it 'fixes' at least three things:

1. CFG scale - now it changes only intensity, and not composition!
2. Negative embeddings that previously affected composition, no longer do! I'd been using [:negembedding:5] to achieve this previously.
3. Diversity. Poor checkpoints now produce huge variety and diversity!

Thank you so much!