Consider Adding Support for DeepSpeed-MII

[jdries3]

Is your feature request related to a problem? Please describe.
Recently Microsoft announced DeepSpeed-MII (DS-Public, https://github.com/microsoft/DeepSpeed) that adds support for a number of speed improvements for Deep Learning Training and Inference. If this can be tested/evaluated with InvokeAI's current architecture and proven an improvement, it might be worth the extra dependencies to incorporate.

Describe the solution you'd like
https://github.com/microsoft/DeepSpeed-MII/blob/main/examples/local/txt2img-example.py Shows some of the changes to use DeepSpeed-MII with the base Stable Diffusion release. Perhaps start there and validate whether any further work is warranted before launching into deeper implementation.

Additional context
There is an open issue (CompVis/stable-diffusion#180) to incorporate DeepSpeed-MII in upstream. There's been no movement on this since it was filed in September. But if this is incorporated upstream, this might end up being duplicative work.

[lstein]

Thanks for the pointer. This looks very intriguing. Unfortunately it only works with the Hugging Face diffusers implementation of Stable Diffusion and won't work with our architecture. That being said, we're discussing porting to diffusers and this is yet another reason to do so.

Flagging @keturn, who is interested in doing a diffusers port.

[n00mkrad]

Should be mentioned that DeepSpeed effectively doesn't work on Windows.

[0xdevalias]

Probably some relevant 'prior art' over here too:

• https://github.com/AUTOMATIC1111/stable-diffusion-webui/search?q=deepspeed+is%3Aissue&type=issues
  • [Feature request] Dreambooth deepspeed AUTOMATIC1111/stable-diffusion-webui#1734
  • [Feature Request]: Dreambooth on 8GB VRam GPU (holy grail) AUTOMATIC1111/stable-diffusion-webui#3586
    • Dreambooth AUTOMATIC1111/stable-diffusion-webui#2002
  • Running AUTOMATIC1111 / stable-diffusion-webui with Dreambooth fine-tuned models AUTOMATIC1111/stable-diffusion-webui#1429
• https://github.com/huggingface/diffusers/tree/main/examples/dreambooth#training-on-a-8-gb-gpu
  • Uses deepspeed

[LifeIsStrange]

Note that Huggingface Accelerate exposé deepspeed cf CompVis/stable-diffusion#180 (comment)

[keturn]

I haven't looked in to the details, but I think the current state of things is that Accelerate has some integration for DeepSpeed, but not DeepSpeed Mii. DeepSpeed Mii being the thing that's more useful for inference.