first version of controlnet (canny) for stablediffusion 2.1

[thibaudart]

I just released Controlnet (canny) for SD 2.1:

https://twitter.com/thibaudz/status/1632877005866663937

https://huggingface.co/thibaud/controlnet-canny-sd21

[blacklig]

Does anyone know why when using control nets for SD 2.1 I get always rather "blurry" and like "low quality jpeg" outputs? My prompts are good I think, also using some most famous negative embeddings, but sometimes not. No matter what I do, I cannot get crisp output of SD 2.1 (using models trained even on 832x832). If I disable CN for that particullar settings, outputs are super clear, but with CN they tend to lose lot of that crispnes. Dont really know why. Has someone idea what might I be missing? Or is that just like that that those models destroy quality? Interestingly on same settings with 1.5 I dont have those issues and I get crisp results (obviously different models for 1.5).

[thibaudart]

You can try to reduce the weights of the controlnet model or use less models in the same time.
have you tried perpetual diffusion? (My finetune based on sd2.1 but trained with 1024). It works well with the CN2.1 (I m the one who trained them)

[blacklig]

I did not tried perpetual diffusion, so that is your model trained on 1024? Cool! I have it in mind though but did not tried it yet, tried some other models trained on 832 though. Thing is, even if I get some other model trained even on 728, withou control net results are great. When I turn on control nets it gets blurry as I described. More weight I put on it, more blurry and kinda like stretched low quality jpeg it gets. But why is this happening? Is this some crucial difference against 1.5? Because if I switch to 1.5 use same amount of control nets (I am using 3) and same weights, I get still nice and clear pictures. When I do this on 2.1 with yours CNs, then it fells apart and quality is very bad.. If I reduce weight, then yes, quality improves, but also result drifts away from what I wanted :) I dont have weights too much though.. only around 0.3-0.4 and only using during like first 30-40% of steps..

[thibaudart]

SD1.5 and SD2.1 have different behaviors.
And SDXL will have another one too. I think CN on XL will be the solution (but for now they're not compatible - Stability are working on them).

[blacklig]

Yeah.. that is what I had to admit too and decided just to wait for SDXL. Will you be working on CNs for SDXL?

[nityanandmathur]

@thibaudart Could you please let me know, how did you train the controlnet model for SD 2.1? The official repo uses OpenAI's LDM. How did you migrate it to Stability AI's SD 2.1?