1.1.420 Image-wise ControlNet and StyleAlign (Hertz et al.)

[lllyasviel]

Since sd-webui-controlnet 1.1.420, users will be able to use image-wise controlnets. And, based on image-wise controlnets, the "StyleAlign" is avaliable.

Style Aligned Image Generation via Shared Attention
Amir Hertz* 1 Andrey Voynov* 1 Shlomi Fruchter† 1 Daniel Cohen-Or† 1,2
1 Google Research 2 Tel Aviv University

Previously, all ControlNet units will be applied to all images in your batch. Now, if you use “each ControlNet unit for each image in a batch”, you will have each ControlNet unit for each image in your batch.

For example, if your batch size is 4, then your

ControlNet unit 0 will be applied to image 0, 
ControlNet unit 1 will be applied to image 1, 
ControlNet unit 2 will be applied to image 2, 
ControlNet unit 3 will be applied to image 3 ...

And you can use “[StyleAlign] Align image style in the batch” to align the style of all images in a batch (via shared attention reference).

(If the number of batch size does not match ControlNet unit count, the remainder of the division will be used)

Sanity Check (with CN)

Some example images used in the Sanity Check:

The model is realisticVisionV51_v51VAE: https://civitai.com/models/4201?modelVersionId=130072

Prompt

Positive:

best quality, very detailed, high resolution, 4k, 8k, 35 mm, a handsome man, looking at viewer, street

Negative:

text, watermark, low quality, medium quality, blurry, censored, deformed, mutated, anime, toon, render, 3d, ilustration, moles, dark skin spots

Parameters:

4x Openpose CN (control_v11p_sd15_openpose):

All default parameters.

Note that here we use openpose (rather than openpose_full) without face landmarks to avoid the style influence of face appearance.

Also, if you do not have 4 controlnet units, go to settings->controlnet->ControlNet unit number to have any number of units.

Check "Each ControlNet unit for each image in a batch"

Generate, you will get this

The 4 images are generated by these 4 poses

You can see this is what "Each ControlNet unit for each image in a batch".

Then check "[StyleAlign] Align image style in the batch."

Generate again, you will get the "StyleAlign" for the 4 images in a same batch:

Sanity Check (with dynamic-prompts)

You need to install https://github.com/adieyal/sd-dynamic-prompts to use prompt permutation.

The grammar is {A|B|C|D} like "a toy {train|bicycle|car|boat}. macro photo. 3d game asset"

The model is realisticVisionV51_v51VAE: https://civitai.com/models/4201?modelVersionId=130072

Positive:

a toy {train|bicycle|car|boat}. macro photo. 3d game asset

negative:

low quality, lowres, ugly, bad

parameters:

dynamic-prompts:

then if you click generate and generate without using "StyleAlign", the result is

then check "StyleAlign"

then the style will be aligned

FAQ

Q: Does this influence speed?
A: Yes, if you use "StyleAlign", attention context is longer. Generating will be slower. But in my tests not too slow.

Q: Can I use t2ia-style/reference/ip-adapters together with StyleAlign?
A: Yes. But this behavior is still under experiment and may be changed in any future versions. (Not yet very sure what behavior should be the correct behavior because now attention will have formulations both in-batch and globally. It should already work out of box but not extensively tested.)

Q: XL? SSD? Turbo? LCM?
A: yes, yes, yes, and yes. (w/ webui 1.7.0)

[lllyasviel]

@catboxanon @huchenlei @ljleb @anyone-who-know-how-to-make-it-work
currently everything in the batch options is not related to any script args or infotext. help wanted if anyone know how to make it more consistent to previous codes. if nobody know how to do it, then it will be left as is.

[sdbds]

I think we need set number for batch size and batch count like webui for aligning video animation.

For animation or video,they use different context batch size (render context frame) and FPS.

[lllyasviel]

continue-revolution/sd-webui-animatediff#360

[sdbds]

It's great to see this collaboration, continue-revolution is one of my best friend, he's a very nice guy, both in terms of coding ability and academic ability

[continue-revolution]

Thanks for your invitation. Very glad to join you guys and improve CN for video generation in A1111

[lllyasviel]

@continue-revolution welcome
@huchenlei @ljleb

[huchenlei]

Welcome onboard! @continue-revolution

[matrix4767]

Hey, a notebook was added on the main github page for sd 1.4, like 20 minutes after this controlnet update. And so would probably work for 1.5 too.

[lllyasviel]

realisticVisionV51_v51VAE is sd1.5 model

[matrix4767]

This is present in the update?
google/style-aligned@f3a9669#diff-7a71e188435b6f68af74c3dcc72d8df5758a9c190855cc27ba4c2f3bb23315c5

[catboxanon]

Pinging @comfyanonymous here since they claim StyleAlign is not implemented correctly. Maybe they can provide more useful info.
https://boards.4channel.org/g/thread/97736971#p97740259

[lllyasviel]

This impl is converted from the paper.
there is a source-target adain also mentioned in paper and should be already in reference cn units if users use it.
feel free to ping me if in the future users get alter choices in other software

[lllyasviel]

update 12/6:

now also tested dynamic-prompts permutations (see also the updated sanity check)

[sdbds]

https://huggingface.co/spaces/fffiloni/video2openpose2
I have made a video2pose gradio referencing this huggingface space that can be extended to all controlnet preprocessors via the control_aux pip package.
Do I need to add video processing to the sd-webui-control plugin in the future?
Or should I keep the video independent?
https://github.com/sdbds/vid2pose/blob/main/video2openpose2.py
This is the current situation. I just added dwpose to the original openpose and it is faster with cuda.
@lllyasviel @huchenlei @continue-revolution

[continue-revolution]

I‘m planning to move everything about CN hook inside my AnimateDiff extension here, so I think ultimately we may want CN to be compatible with videos.

[drphero]

And you can use “[StyleAlign] Align image style in the batch” to align the style of all images in a batch (via shared attention reference).

What is the style being aligned to?

[kft334]

From my limited testing it seems that all images beyond the first in a batch are aligned to the first image of the batch. The style-align repo does have style alignment from a reference image now, at least for SDXL, so hopefully that finds it's way here soon too. I'm getting artifacts from the first batch image in some other images in the batch so the problem of separating content from style does still exist in style-align but it does appear to do a better job.

[lllyasviel]

"Style Align" align image styles in a batch, it is not "Style Transfer".

[john-mnz]

@lllyasviel
have you seen this?
it seems to be style transfer
https://github.com/google/style-aligned/blob/main/style_aligned_transfer_sdxl.ipynb

[wardensc2]

I can't make it work with Reference Adain the others Reference Only or Reference Adain + Attn work fine. Anytimes I select Reference Adain with Batch Option and Style Align On I get this error message, does anyone know what error is this ?
Style Align Error.docx

[enternalsaga]

I could not replicate results of Sanity Check (with dynamic-prompts)? what did you add in controlnet? Do I need to enable any controlnet or just the stylealign checkbok is fine?

[xpeng]

just check stylealign checkbox

[xpeng]

is there any problem in negative prompt length larger than 75 when enable StyleAlign?
RuntimeError: step must be nonzero

for example in negative prompt box:

anime, cartoon, graphic, text, painting, crayon, graphite, abstract, glitch, deformed, mutated, ugly, disfigured, tripod, camera, anime, animation, cartoon, 3D, drawing, painting, (censorship, censored, worst quality, low quality, normal quality, lowres, low details, bad photo, bad photography, bad art:1.4), (watermark, signature, text font, username, error, logo, words, letters, digits, autograph, trademark, name:1.2), (blur, blurry), morbid, ugly, asymmetrical, mutated malformed, mutilated, poorly lit, bad shadow, draft, cropped, out of frame, cut off, censored, jpeg artifacts, out of focus, glitch, duplicate, (bad hands, bad anatomy, bad body, bad face, bad teeth, bad arms, bad legs, deformities:1.3) (ugly hands, ugly anatomy, ugly body, ugly face, ugly teeth, ugly arms, ugly legs, deformities:1.3) ugly fingers, bad fingers, (((ugly nipples, bad nipples, deformed nipples))), (((Bad teeth, ugly teeth)))

[somanchiu]

It applies not only to the negative prompt, but also to the positive prompt.

[raonsol]

Hello, Do you have plan to implement this method with additional way, such as the way like IP-Adapter?

StyleAlign can do so-called "Style Transfer" method, because in their example it says StyleAlign can be used in generating images with a style from "reference image". The example also shows the method that uses both reference image and depth map to generate same style images with different text prompt.

I think this can be implemented not only by aligning other images with first image during the batch, but some kind of separated type in controlnet unit.
Here is the example that @yvrjsharma implemented to do something like this.

[LukcyOne]

I just want to achieve style alignment by referencing images, perhaps style aligned transfer is a good method.
I have tried the "sref" function of NijiJourney, which can reference the style of images. But I can't find an open-source implementation that can match it with simple operations.

[xpeng]

sd1.5 based model is ok.
but failed on openpose controlnet applying for sdxl based models. come with different errors on different openpose controlnet models.

[qinzhenzhao]

Great.It's marvelous.
But how can I use API request to generate images by BatchOptions and styleAlign?I mean what is the parameter format?

[huchenlei]

Unfortunately, there is currently no way to use style align via API now, as batch options cannot be controlled by API. You can file an issue and we will consider implement/rework it if enough people think it is useful.