Community Integration: Making AIGC cheaper, faster, and more efficient.

[binmakeswell]

Is your feature request related to a problem? Please describe.
AIGC has recently risen to be one of the hottest topics in AI. Unfortunately, large hardware requirements and training costs are still a severe impediment to the rapid growth of the AIGC industry. The Stable Diffusion v1 version of the model requires 150,000 A100 GPU Hours for a single training session.

Describe the solution you'd like
We are happy to share a fantastic solution where the costs of training AIGC models such as stable diffusion can be 7 times cheaper!

Colossal-AI release a complete open-source Stable Diffusion pretraining and fine-tuning solution with the pretraining cost reduced by 6.5 times, and the hardware cost of fine-tuning by 7 times. An RTX 2070/3050 PC is good enough to accomplish the fine-tuning task flow, allowing AIGC models such as Stable Diffusion to be available to a wider community.

Open-source code：https://github.com/hpcaitech/ColossalAI/tree/main/examples/images/diffusion

Additional context
More details can be found on the blog. We believe the democratization of AIGC models is also very helpful for Hugging Face members. We would appreciate it if we could build the integration with you to benefit both of our users and we are willing to provide help you need in this cooperation for free.

Thank you very much.

Best regards,
Yongbin Li, HPC-AI Tech

[patrickvonplaten]

Hey @binmakeswell,

This sounds great - I think we'd be very interested in having a better integration here! Also cc @patil-suraj @thomwolf

[patil-suraj]

This is very cool @binmakeswell ! Would be very happy to integrate diffusers in colossalAI examples!

[binmakeswell]

Thanks for your enthusiastic response and we look forward to working together!

How do you expect it to be carried out?
Take our collaboration with Lightning for example, we communicated in slack for details, provided documentation and eventually did a joint release.

[0xdevalias]

Would love to see a simple integration between diffusers and ColossalAI!

Here's a collection of related issues I found/opened on various StableDiffusion / Dreambooth related projects about this:

• [FEATURE]: Integrate StableDiffusion Dreambooth with ColossalAI hpcaitech/ColossalAI#1935
• [Feature Request]: colossalai integration AUTOMATIC1111/stable-diffusion-webui#4606
• Integrate with ColossalAI for memory/speed efficiency and improvements d8ahazard/sd_dreambooth_extension#174
• Integrate StableDiffusion Dreambooth with ColossalAI TheLastBen/fast-stable-diffusion#493
• Integrate StableDiffusion Dreambooth with ColossalAI ShivamShrirao/diffusers#129

It would also be really interesting to hear/see benchmarks for how ColossalAI is compares and contrasts with the various other speed/memory/etc optimisations out there (eg. xformers, AITemplate, deepspeed, deepspeed-MII, bitsandbytes, kernl, Triton, etc); whether it is a replacement for them, or whether it can be used alongside them for even greater gains. A few related issues off the top of my head:

• Explore potential speed benefits from implementing kernl (Up to 12X faster GPU inference) #1094
• Up to 2x speedup on GPUs using memory efficient attention #532 (comment)
  • Mentions AITemplate, DeepSpeed, DeepSpeed-MII, Triton
• Up to 2x speedup on GPUs using memory efficient attention #532 (comment)
  • More about DeepSpeed-MII
• Up to 2x speedup on GPUs using memory efficient attention #532 (comment)
  • AITemplate + xformers

[binmakeswell]

Thanks for your enthusiastic response and we look forward to working together!

How do you expect it to be carried out? Take our collaboration with Lightning for example, we communicated in slack for details, provided documentation and eventually did a joint release.

Hi, @patrickvonplaten and @patil-suraj , thank you for the enthusiastic response.
Do you have any requests or suggestions? Hopefully, we can discuss the details soon and complete this integration. Thanks.

[patrickvonplaten]

Hey @binmakeswell,

Should we jump on a call to discuss? :-) I think we've reached out via email

[binmakeswell]

Hey @binmakeswell,

Should we jump on a call to discuss? :-) I think we've reached out via email

Sure. @patrickvonplaten I just emailed(ybl@hpcaitech.com) you our available hours. Looking forward to our meeting and cooperation :-)

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[0xdevalias]

Hopefully not stale.

[binmakeswell]

We are working in process for this integration : )

[haofanwang]

Any update @binmakeswell ? It would be very great to fasten training process.

I found train_dreambooth.py and train_dreambooth_colossalai.py that are based on diffusers.

[Fazziekey]

Any update @binmakeswell ? It would be very great to fasten training process.

I found train_dreambooth.py and train_dreambooth_colossalai.py that are based on diffusers.

Thank you for your interest and contribution to colossalai, we have merged into the hugging face diffuser library.
https://github.com/huggingface/diffusers/tree/main/examples/research_projects/colossalai

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[Kuri-su]

Is there any progress?

[patrickvonplaten]

It's merged to diffusers :-) See:
https://github.com/huggingface/diffusers/tree/main/examples/research_projects/colossalai

[Sniper199999]

It's merged to diffusers :-) See: https://github.com/huggingface/diffusers/tree/main/examples/research_projects/colossalai

so does that mean that integration with Automatic1111 is around the corner?

[0xdevalias]

@Sniper199999 Last I paid attention, automatic1111 isn't based on diffusers at all, so unless that has changed since I last looked, short answer is "no, probably not"

[patrickvonplaten]

Actually we'd really like to integrate diffusers into A1111, but it seems quite complicated. Note sure if anybody has a clear plan of action. Also related: AUTOMATIC1111/stable-diffusion-webui#2577

Should we try to create a module plug-in?