The cost of consistency decoder

[Luciennnnnnn]

Describe the bug

I replace original VAE decoder of a stable diffusion model with Consistency Decoder, then CUDA out of memory occurs. My question is that How large of Consistency Decoder is compared to original VAE decoder.

• diffusers version: 0.23.0
• Platform: Linux-5.15.0-60-generic-x86_64-with-glibc2.35
• Python version: 3.10.11
• PyTorch version (GPU?): 2.0.0+cu118 (True)
• Huggingface_hub version: 0.17.3
• Transformers version: 4.34.0
• Accelerate version: 0.23.0
• xFormers version: 0.0.18
• Using GPU in script?:
• Using distributed or parallel set-up in script?:

Reproduction

Decode a large latent

Logs

No response

System Info

..

Who can help?

No response

[eeyrw]

According my observation the consistency decoder uses an UNet2D module in the VAE and does diffusion. That is where huge VRAM cost and latency from. According my test it costs ~16G VRAM when decoding the latent with target size 1024*1024.

[patrickvonplaten]

Hey @Luciennnnnnn,

I recommend to read a bit through the original paper: https://github.com/openai/consistencydecoder

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.