Hugging Face diffusers

[AK391]

would be awesome to support Hugging Face diffusers with the web ui which has many optimizations compared to the original release and supports things like training with dreambooth and textual inversion. Recently also added 🌾 JAX/Flax integration for super fast Stable Diffusion on TPUs

repo: https://github.com/huggingface/diffusers

[FTC55]

I completely agree with you, TPU support in particular would be a game changer

[patrickvonplaten]

Hey maintainer of diffusers here 👋,

I'm currently trying to see if we could implement diffusers at a low level for UNet + VAE support, e.g. try to load the low-level models (not the schedulers as diffusers doesn't provide all the schedulers this repo already offers) via diffusers. IMO this could have the following advantages:

• offload maintenance to diffusers so that this repo could fully focus on building the newest & hottest features
• TPU, ONNX, optimization support from diffusers (https://huggingface.co/docs/diffusers/optimization/fp16) instead of having to hijack original stable diffusion: https://github.com/AUTOMATIC1111/stable-diffusion-webui/blob/master/modules/sd_hijack.py (think this might be hard to maintain)
• Have full & simple compatibility with both all SD checkpoints and diffusers checkpoints via converting on the fly here:

  stable-diffusion-webui/modules/sd_models.py

  Line 131 in e644b5a

  def load_model_weights(model, checkpoint_info):

If this sounds interesting to the community, I'd be happy to try to focus on this next week (and the week after) with the main goals to not break any existing features. We could also make a new diffusers version for this.

Very curious about hearing feedback!

[dfl]

I became interested in this because I want to use the PNDM sampler... there are certain nuances that are different than PLMS. Plus I'm sure there are lots of other awesome diffuser models that would be interesting to play with.

[thibaudart]

@patrickvonplaten any news about this? :)

[patrickvonplaten]

Still very interested! What would be the preferred plan of action from the maintainers of @AUTOMATIC1111 here? Happy to somehow give users the possibility to combine the web-ui with diffusers here

[ClashSAN]

@patrickvonplaten
I'm not sure if this idea got through to users - #6585. But a good amount of experiments/extensions could be made from using diffusers library.

Try asking AUTOMATIC1111 directly in the development server. https://github.com/AUTOMATIC1111/stable-diffusion-webui/wiki/Contributing

[markovivl]

@patrickvonplaten Hello! Was there any progress with the integration?

[x02Sylvie]

It would also make it much easier to experimentally implement AITemplate (https://github.com/facebookincubator/AITemplate/tree/main/examples/05_stable_diffusion) optimizations in automatic webui

[kopyl]

@patrickvonplaten Hey :)

Do you know how to use this repo's schedulers with Diffusers?

I can't get the same results of generations in Diffusers as I get in WebUI. Maybe the schedulers can be the culprit. Or maybe you know why?

I see that Diffusers generate much worse images than WebUI. It feels like Diffusers don't get the prompt at all while WebUI does...

[patrickvonplaten]

Feel free to open an issue in diffusers: https://github.com/huggingface/diffusers/issues/new?assignees=&labels=bug&template=bug-report.yml

We have made sure to have the exact same results as the original implementation, but we might not have all the features that A1111 has. Would be nice to see a reproducible example of when generations are worse in diffusers compared to A1111

[kopyl]

@patrickvonplaten Thanks a lot, I will work on it to show you.

[bbecausereasonss]

Diffusors is epic, I hope there's a way to bring it into Auto's.

[catboxanon]

Looks like support for this is only being given to a fork. vladmandic/automatic#593