ControlNet-XS

[geroldmeisinger]

https://vislearn.github.io/ControlNet-XS

With increasing computing capabilities, current model architectures appear to follow the trend of simply upscaling all components without validating the necessity for doing so. In this project we investigate the size and architectural design of ControlNet [Zhang et al., 2023] for controlling the image generation process with stable diffusion-based models. We show that a new architecture with as little as 1% of the parameters of the base model achieves state-of-the art results. Hence we call it ControlNet-XS. We provide the code for controlling StableDiffusion-XL [Dustin Podell et al., 2023] (Model B, 48M Parameters) and StableDiffusion 2.1 [Robin Rombach et al. 2022] (Model B, 14M Parameters), all under openrail license.

[Vesnica]

What are the pros and cons of this model compared to controlnet-lllite?

[geroldmeisinger]

🤷

code and models are available now!