CogVideo & CogVideoX

中文阅读

日本語で読む

Experience the CogVideoX-5B model online at 🤗 Huggingface Space or 🤖 ModelScope Space

📚 View the paper and user guide

👋 Join our WeChat and Discord

📍 Visit QingYing and API Platform to experience larger-scale commercial video generation models.

Update and News

• 🔥🔥 News: 2024/8/29: By adding pipe.enable_sequential_cpu_offload() and pipe.vae.enable_slicing() to the inference code of CogVideoX-5B, VRAM usage can be reduced to 5GB. Please check the updated cli_demo.
• 🔥 News: 2024/8/27: The CogVideoX-2B model's open-source license has been changed to the Apache 2.0 License.
• 🔥 News: 2024/8/27: We have open-sourced a larger model in the CogVideoX series, CogVideoX-5B. We have significantly optimized the model's inference performance, greatly lowering the inference threshold. You can run CogVideoX-2B on older GPUs like the GTX 1080TI, and run the CogVideoX-5B model on mid-range GPUs like the RTX 3060. Please ensure you update and install the dependencies according to the requirements, and refer to the cli_demo for inference code.
• 🔥 News: 2024/8/20: VEnhancer now supports enhancing videos generated by CogVideoX, achieving higher resolution and higher quality video rendering. We welcome you to try it out by following the tutorial.
• 🔥 News: 2024/8/15: The SwissArmyTransformer dependency in CogVideoX has been upgraded to 0.4.12. Fine-tuning no longer requires installing SwissArmyTransformer from source. Additionally, the Tied VAE technique has been applied in the implementation within the diffusers library. Please install diffusers and accelerate libraries from source. Inference for CogVideoX now requires only 12GB of VRAM. The inference code needs to be modified. Please check cli_demo.
• 🔥 News: 2024/8/12: The CogVideoX paper has been uploaded to arxiv. Feel free to check out the paper.
• 🔥 News: 2024/8/7: CogVideoX has been integrated into diffusers version 0.30.0. Inference can now be performed on a single 3090 GPU. For more details, please refer to the code.
• 🔥 News: 2024/8/6: We have also open-sourced 3D Causal VAE used in CogVideoX-2B, which can reconstruct the video almost losslessly.
• 🔥 News: 2024/8/6: We have open-sourced CogVideoX-2B，the first model in the CogVideoX series of video generation models.
• 🌱 Source: 2022/5/19: We have open-sourced CogVideo (now you can see in CogVideo branch)，the first open-sourced pretrained text-to-video model, and you can check ICLR'23 CogVideo Paper for technical details.

More powerful models with larger parameter sizes are on the way~ Stay tuned!

Table of Contents

Jump to a specific section:

• Quick Start
  • SAT
  • Diffusers
• CogVideoX-2B Video Works
• Introduction to the CogVideoX Model
• Full Project Structure
  • Inference
  • SAT
  • Tools
• Introduction to CogVideo(ICLR'23) Model
• Citations
• Open Source Project Plan
• Model License

Quick Start

Prompt Optimization

Before running the model, please refer to this guide to see how we use large models like GLM-4 (or other comparable products, such as GPT-4) to optimize the model. This is crucial because the model is trained with long prompts, and a good prompt directly impacts the quality of the video generation.

SAT

Please make sure your Python version is between 3.10 and 3.12, inclusive of both 3.10 and 3.12.

Follow instructions in sat_demo: Contains the inference code and fine-tuning code of SAT weights. It is recommended to improve based on the CogVideoX model structure. Innovative researchers use this code to better perform rapid stacking and development.

Diffusers

Please make sure your Python version is between 3.10 and 3.12, inclusive of both 3.10 and 3.12.

pip install -r requirements.txt

Then follow diffusers_demo: A more detailed explanation of the inference code, mentioning the significance of common parameters.

Gallery

CogVideoX-5B

5b_1.mp4	5b_2.mp4	5b_3.mp4	5b_4.mp4
5b_5.mp4	5b_6.mp4	5b_7.mp4	5b_8.mp4

CogVideoX-2B

1.mp4

2.mp4

3.mp4

4.mp4

To view the corresponding prompt words for the gallery, please click here

Model Introduction

CogVideoX is an open-source version of the video generation model originating from QingYing. The table below displays the list of video generation models we currently offer, along with their foundational information.

Model Name	CogVideoX-2B	CogVideoX-5B
Model Description	Entry-level model, balancing compatibility. Low cost for running and secondary development.	Larger model with higher video generation quality and better visual effects.
Inference Precision	FP16* (Recommended), BF16, FP32, FP8*, INT8, no support for INT4	BF16 (Recommended), FP16, FP32, FP8*, INT8, no support for INT4
Single GPU VRAM Consumption	SAT FP16: 18GB diffusers FP16: starting from 4GB* diffusers INT8(torchao): starting from 3.6GB*	SAT BF16: 26GB diffusers BF16: starting from 5GB* diffusers INT8(torchao): starting from 4.4GB*
Multi-GPU Inference VRAM Consumption	FP16: 10GB* using diffusers	BF16: 15GB* using diffusers
Inference Speed (Step = 50, FP/BF16)	Single A100: ~90 seconds Single H100: ~45 seconds	Single A100: ~180 seconds Single H100: ~90 seconds
Fine-tuning Precision	FP16	BF16
Fine-tuning VRAM Consumption (per GPU)	47 GB (bs=1, LORA) 61 GB (bs=2, LORA) 62GB (bs=1, SFT)	63 GB (bs=1, LORA) 80 GB (bs=2, LORA) 75GB (bs=1, SFT)
Prompt Language	English*
Prompt Length Limit	226 Tokens
Video Length	6 Seconds
Frame Rate	8 Frames per Second
Video Resolution	720 x 480, no support for other resolutions (including fine-tuning)
Positional Encoding	3d_sincos_pos_embed	3d_rope_pos_embed
Download Page (Diffusers)	🤗 HuggingFace 🤖 ModelScope 🟣 WiseModel	🤗 HuggingFace 🤖 ModelScope 🟣 WiseModel
Download Page (SAT)	SAT

Data Explanation

• When testing using the diffusers library, all optimizations provided by the diffusers library were enabled. This solution has not been tested for actual VRAM/memory usage on devices other than NVIDIA A100 / H100. Generally, this solution can be adapted to all devices with NVIDIA Ampere architecture and above. If the optimizations are disabled, VRAM usage will increase significantly, with peak VRAM usage being about 3 times higher than the table shows. However, speed will increase by 3-4 times. You can selectively disable some optimizations, including:

pipe.enable_model_cpu_offload()
pipe.enable_sequential_cpu_offload()
pipe.vae.enable_slicing()
pipe.vae.enable_tiling()

• When performing multi-GPU inference, the enable_model_cpu_offload() optimization needs to be disabled.
• Using INT8 models will reduce inference speed. This is to ensure that GPUs with lower VRAM can perform inference normally while maintaining minimal video quality loss, though inference speed will decrease significantly.
• The 2B model is trained with FP16 precision, and the 5B model is trained with BF16 precision. We recommend using the precision the model was trained with for inference.
• PytorchAO and Optimum-quanto can be used to quantize the text encoder, Transformer, and VAE modules to reduce CogVideoX's memory requirements. This makes it possible to run the model on a free T4 Colab or GPUs with smaller VRAM! It is also worth noting that TorchAO quantization is fully compatible with torch.compile, which can significantly improve inference speed. FP8 precision must be used on devices with NVIDIA H100 or above, which requires installing the torch, torchao, diffusers, and accelerate Python packages from source. CUDA 12.4 is recommended.
• The inference speed test also used the above VRAM optimization scheme. Without VRAM optimization, inference speed increases by about 10%. Only the diffusers version of the model supports quantization.
• The model only supports English input; other languages can be translated into English during refinement by a large model.

Friendly Links

We highly welcome contributions from the community and actively contribute to the open-source community. The following works have already been adapted for CogVideoX, and we invite everyone to use them:

• Xorbits Inference: A powerful and comprehensive distributed inference framework, allowing you to easily deploy your own models or the latest cutting-edge open-source models with just one click.
• VideoSys: VideoSys provides a user-friendly, high-performance infrastructure for video generation, with full pipeline support and continuous integration of the latest models and techniques.

Project Structure

This open-source repository will guide developers to quickly get started with the basic usage and fine-tuning examples of the CogVideoX open-source model.

Inference

• dcli_demo: A more detailed inference code explanation, including the significance of common parameters. All of this is covered here.
• cli_demo_quantization: Quantized model inference code that can run on devices with lower memory. You can also modify this code to support running CogVideoX models in FP8 precision.
• diffusers_vae_demo: Code for running VAE inference separately.
• space demo: The same GUI code as used in the Huggingface Space, with frame interpolation and super-resolution tools integrated.
• convert_demo: How to convert user input into long-form input suitable for CogVideoX. Since CogVideoX is trained on long texts, we need to transform the input text distribution to match the training data using an LLM. The script defaults to using GLM-4, but it can be replaced with GPT, Gemini, or any other large language model.
• gradio_web_demo: A simple Gradio web application demonstrating how to use the CogVideoX-2B / 5B model to generate videos. Similar to our Huggingface Space, you can use this script to run a simple web application for video generation.

cd inference
# For Linux and Windows users
python gradio_web_demo.py

# For macOS with Apple Silicon users, Intel not supported, this maybe 20x slower than RTX 4090
PYTORCH_ENABLE_MPS_FALLBACK=1 python gradio_web_demo.py

sat

• sat_demo: Contains the inference code and fine-tuning code of SAT weights. It is recommended to improve based on the CogVideoX model structure. Innovative researchers use this code to better perform rapid stacking and development.

Tools

This folder contains some tools for model conversion / caption generation, etc.

• convert_weight_sat2hf: Convert SAT model weights to Huggingface model weights.
• caption_demo: Caption tool, a model that understands videos and outputs them in text.
• AutoDL Mirror: A one-click deployment of Huggingface Space mirror provided by community members.

CogVideo(ICLR'23)

The official repo for the paper: CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformers is on the CogVideo branch

CogVideo is able to generate relatively high-frame-rate videos. A 4-second clip of 32 frames is shown below.

cogvideo.mp4

The demo for CogVideo is at https://models.aminer.cn/cogvideo, where you can get hands-on practice on text-to-video generation. The original input is in Chinese.

Citation

🌟 If you find our work helpful, please leave us a star and cite our paper.

@article{yang2024cogvideox,
  title={CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer},
  author={Yang, Zhuoyi and Teng, Jiayan and Zheng, Wendi and Ding, Ming and Huang, Shiyu and Xu, Jiazheng and Yang, Yuanming and Hong, Wenyi and Zhang, Xiaohan and Feng, Guanyu and others},
  journal={arXiv preprint arXiv:2408.06072},
  year={2024}
}
@article{hong2022cogvideo,
  title={CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformers},
  author={Hong, Wenyi and Ding, Ming and Zheng, Wendi and Liu, Xinghan and Tang, Jie},
  journal={arXiv preprint arXiv:2205.15868},
  year={2022}
}

Open Source Project Plan

• CogVideoX Model Open Source
  • CogVideoX Model Inference Example (CLI / Web Demo)
  • CogVideoX Online Experience Example (Huggingface Space)
  • CogVideoX Open Source Model API Interface Example (Huggingface)
  • CogVideoX Model Fine-Tuning Example (SAT)
  • CogVideoX Model Fine-Tuning Example (Huggingface Diffusers)
  • CogVideoX-5B Open Source (Adapted to CogVideoX-2B Suite)
  • CogVideoX Technical Report Released
  • CogVideoX Technical Explanation Video
• CogVideoX Peripheral Tools
  • Basic Video Super-Resolution / Frame Interpolation Suite
  • Inference Framework Adaptation
  • ComfyUI Full Ecosystem Tools

We welcome your contributions! You can click here for more information.

License Agreement

The code in this repository is released under the Apache 2.0 License.

The CogVideoX-2B model (including its corresponding Transformers module and VAE module) is released under the Apache 2.0 License.

The CogVideoX-5B model (Transformers module) is released under the CogVideoX LICENSE.