2s 512×512	2s 512×512	2s 512×512
A serene night scene in a forested area. [...] The video is a time-lapse, capturing the transition from day to night, with the lake and forest serving as a constant backdrop.	A soaring drone footage captures the majestic beauty of a coastal cliff, [...] The water gently laps at the rock base and the greenery that clings to the top of the cliff.	The majestic beauty of a waterfall cascading down a cliff into a serene lake. [...] The camera angle provides a bird's eye view of the waterfall.
A bustling city street at night, filled with the glow of car headlights and the ambient light of streetlights. [...]	The vibrant beauty of a sunflower field. The sunflowers are arranged in neat rows, creating a sense of order and symmetry. [...]	A serene underwater scene featuring a sea turtle swimming through a coral reef. The turtle, with its greenish-brown shell [...]

Videos are downsampled to .gif for display. Click for original videos. Prompts are trimmed for display, see here for full prompts.

• ✅ Following the findings in OpenSora, we also adopt the VAE from Stable Diffusion for video latent encoding.
• ✅ We pick the STDiT model as our video diffusion transformer following the best practice in OpenSora.
• ✅ Support T5 text conditioning.

• Evaluation pipeline.
• Complete the data processing pipeline (including dense optical flow, aesthetics scores, text-image similarity, etc.).

• STDit:

Convert to MS checkpoint:

python tools/convert_pt2ms.py --src /path/to/OpenSora-STDiT-v2-stage3/model.safetensors --target models/opensora_v1.1_stage3.ckpt

• T5 and VAE models are identical to OpenSora 1.0 and can be downloaded from the links below.

Please prepare the model checkpoints of T5, VAE, and STDiT and put them under models/ folder as follows.

• T5: Download the DeepFloyd/t5-v1_1-xxl folder and put it under models/

  Convert to ms checkpoint:

  python tools/convert_t5.py --src models/t5-v1_1-xxl/pytorch_model-00001-of-00002.bin  models/t5-v1_1-xxl/pytorch_model-00002-of-00002.bin --target models/t5-v1_1-xxl/model.ckpt
  
  

• VAE: Download the safetensor checkpoint from here

  Convert to ms checkpoint:

  python tools/convert_vae.py --src /path/to/sd-vae-ft-ema/diffusion_pytorch_model.safetensors --target models/sd-vae-ft-ema.ckpt
  

• STDiT: Download OpenSora-v1-16x256x256.pth / OpenSora-v1-HQ-16x256x256.pth / OpenSora-v1-HQ-16x512x512.pth from here

  Convert to ms checkpoint:

  python tools/convert_pt2ms.py --src /path/to/OpenSora-v1-16x256x256.pth --target models/OpenSora-v1-16x256x256.ckpt
  

  Training orders: 16x256x256 $\to$ 16x256x256 HQ $\to$ 16x512x512 HQ.

  These model weights are partially initialized from PixArt-α. The number of parameters is 724M. More information about training can be found in HPC-AI Tech's report. More about the dataset can be found in datasets.md from HPC-AI Tech. HQ means high quality.

• PixArt-α: Download the pth checkpoint from here (for training only)

  Convert to ms checkpoint:

  python tools/convert_pt2ms.py --src /path/to/PixArt-XL-2-512x512.pth --target models/PixArt-XL-2-512x512.ckpt
  

You can run text-to-video inference via the script scripts/inference.py as follows.

# Sample 16x256x256 videos
python scripts/inference.py --config configs/opensora/inference/stdit_256x256x16.yaml --ckpt_path models/OpenSora-v1-HQ-16x256x256.ckpt --prompt_path /path/to/prompt.txt

# Sample 16x512x512 videos
python scripts/inference.py --config configs/opensora/inference/stdit_512x512x16.yaml --ckpt_path models/OpenSora-v1-HQ-16x512x512.ckpt --prompt_path /path/to/prompt.txt

# Sample 64x512x512 videos
python scripts/inference.py --config configs/opensora/inference/stdit_512x512x64.yaml --ckpt_path /path/to/your/opensora-v1.ckpt --prompt_path /path/to/prompt.txt

For parallel inference, please use mpirun or msrun, and append --use_parallel=True to the inference script referring to scripts/run/run_infer_t2v_parallel.sh

We also provide a three-stage sampling script run_sole_3stages.sh to reduce memory limitation, which decomposes the whole pipeline into text embedding, text-to-video latent sampling, and vae decoding.

For more usage on the inference script, please run python scripts/inference.py -h

We evaluate the inference performance of text-to-video generation by measuring the average sampling time per step and the total sampling time of a video.

All experiments are tested on ascend 910* with mindspore 2.3.1 graph mode.

The text-video pair data should be organized as follows, for example.

.
├── video_caption.csv
├── video_folder
│   ├── part01
│   │   ├── vid001.mp4
│   │   ├── vid002.mp4
│   │   └── ...
│   └── part02
│       ├── vid001.mp4
│       ├── vid002.mp4
│       └── ...

The video_folder contains all the video files. The csv file video_caption.csv records the relative video path and its text caption in each line, as follows.

video,caption
video_folder/part01/vid001.mp4,a cartoon character is walking through
video_folder/part01/vid002.mp4,a red and white ball with an angry look on its face

For acceleration, we pre-compute the t5 embedding before training stdit.

python scripts/infer_t5.py \
    --csv_path /path/to/video_caption.csv \
    --output_path /path/to/text_embed_folder \
    --model_max_length 300     # 300 for OpenSora v1.2, 200 for OpenSora v1.1, 120 for OpenSora 1.0

OpenSora v1 uses text embedding sequence length of 120 (by default). If you want to generate text embeddings for OpenSora v1.1, please change model_max_length to 200.

After running, the text embeddings saved as npz file for each caption will be in output_path. Please change csv_path to your video-caption annotation file accordingly.

If the storage budget is sufficient, you may also cache the video embedding by

python scripts/infer_vae.py \
    --csv_path /path/to/video_caption.csv  \
    --video_folder /path/to/video_folder  \
    --output_path /path/to/video_embed_folder  \
    --vae_checkpoint models/sd-vae-ft-ema.ckpt \
    --image_size 512 \

for parallel running, please refer to scripts/run/run_infer_vae_parallel.sh

For more usage, please check python scripts/infer_vae.py -h

After running, the vae latents saved as npz file for each video will be in output_path.

Finally, the training data should be like follows.

.
├── video_caption.csv
├── video_folder
│   ├── part01
│   │   ├── vid001.mp4
│   │   ├── vid002.mp4
│   │   └── ...
│   └── part02
│       ├── vid001.mp4
│       ├── vid002.mp4
│       └── ...
├── text_embed_folder
│   ├── part01
│   │   ├── vid001.npz
│   │   ├── vid002.npz
│   │   └── ...
│   └── part02
│       ├── vid001.npz
│       ├── vid002.npz
│       └── ...
├── video_embed_folder  # optional
│   ├── part01
│   │   ├── vid001.npz
│   │   ├── vid002.npz
│   │   └── ...
│   └── part02
│       ├── vid001.npz
│       ├── vid002.npz
│       └── ...

Each npz file contains data for the following keys:

• latent_mean mean of vae latent distribution
• latent_std: std of vae latent distribution
• fps: video fps
• ori_size: original size (h, w) of the video

After caching VAE, you can use them for STDiT training by parsing --vae_latent_folder=/path/to/video_embed_folder to the training script python train.py.

If there are multiple folders named in latent_{h}x{w} format under the --vae_latent_folder folder (which is parsed to train.py), one of resolutions will selected randomly during training. For example:

video_embed_folder
   ├── latent_576x1024
   │   ├── vid001.npz
   │   ├── vid002.npz
   │   └── ...
   └── latent_1024x576
       ├── vid001.npz
       ├── vid002.npz
       └── ...

Once you prepare the data in a csv file, you may run the following commands to launch training on a single card.

# standalone training for stage 1
python scripts/train.py --config configs/opensora-v1-1/train/train_stage1.yaml \
    --csv_path /path/to/video_caption.csv \
    --video_folder /path/to/video_folder \
    --text_embed_folder /path/to/text_embed_folder \
    --vae_latent_folder /path/to/video_embed_folder

text_embed_folder and vae_latent_folder are optional and used to speed up the training. You can find more in T5 text embeddings and VAE Video Embeddings

For parallel training, use msrun and along with --use_parallel=True:

# distributed training for stage 1
msrun --master_port=8200 --worker_num=8 --local_worker_num=8 --log_dir=$output_dir  \
    python scripts/train.py --config configs/opensora-v1-1/train/train_stage1.yaml \
    --csv_path /path/to/video_caption.csv \
    --video_folder /path/to/video_folder \
    --text_embed_folder /path/to/text_embed_folder \
    --vae_latent_folder /path/to/video_embed_folder \
    --use_parallel True

OpenSora v1.1 supports training with multiple resolutions, aspect ratios, and a variable number of frames. This can be enabled in one of two ways:

1. Provide variable sized VAE embeddings with the --vae_latent_folder option.
2. Use bucket_config for training with videos in their original format. More on the bucket configuration can be found in Multi-resolution Training with Buckets.

Detailed running command can be referred in scripts/run/run_train_os_v1.1_stage2.sh

Once the training data including the T5 text embeddings is prepared, you can run the following commands to launch training.

# standalone training, 16x256x256
python scripts/train.py --config configs/opensora/train/stdit_256x256x16_ms.yaml \
    --csv_path /path/to/video_caption.csv \
    --video_folder /path/to/video_folder \
    --text_embed_folder /path/to/text_embed_folder \

To use the cached video embedding, please replace --video_folder with --video_embed_folder and pass the path to the video embedding folder.

For parallel training, please use msrun and pass --use_parallel=True

# 8 NPUs, 64x512x512
msrun --master_port=8200 --worker_num=8 --local_worker_num=8 --log_dir=$output_dir  \
    python scripts/train.py --config configs/opensora/train/stdit_512x512x64_ms.yaml \
    --csv_path /path/to/video_caption.csv \
    --video_folder /path/to/video_folder \
    --text_embed_folder /path/to/text_embed_folder \
    --use_parallel True \

We evaluate the training performance of Open-Sora v1.1 on a subset of the MixKit dataset.

All experiments are tested on ascend 910* with mindspore 2.3.1 graph mode.

vae cache: whether vae embedding is pre-computed and cached before training.

Note that T5 text embedding is pre-computed before training.

Here are some generation results after fine-tuning STDiT2 on a mixkit subset.

All experiments are tested on ascend 910* with mindspore 2.3.1 graph mode.

Here are some generation results after fine-tuning STDiT on a subset of WebVid dataset.

For quality evaluation, please refer to the original HPC-AI Tech evaluation doc for video generation quality evaluation.

A 3D-VAE pipeline consisting of a spatial VAE followed by a temporal VAE is trained in OpenSora v1.1. For more details, refer to VAE Documentation.

• Download pretained VAE-2D checkpoint from PixArt-alpha/pixart_sigma_sdxlvae_T5_diffusers if you aim to train VAE-3D from spatial VAE initialization.

  Convert to ms checkpoint:

  python tools/convert_vae1.2.py --src /path/to/pixart_sigma_sdxlvae_T5_diffusers/vae/diffusion_pytorch_model.safetensors --target models/sdxl_vae.ckpt --from_vae2d
  

• Downalod pretrained VAE-3D checkpoint from hpcai-tech/OpenSora-VAE-v1.2 if you aim to train VAEA-3D from the VAE-3D model pre-trained with 3 stages.

  Convert to ms checkpoint:

  python tools/convert_vae1.2.py --src /path/OpenSora-VAE-v1.2/models.safetensors --target models/OpenSora-VAE-v1.2/sdxl_vae.ckpt
  

• Download lpips mindspore checkpoint from here and put it under 'models/'

Before VAE-3D training, we need to prepare a csv annotation file for the training videos. The csv file list the path to each video related to the root video_folder. An example is

video
dance/vid001.mp4
dance/vid002.mp4
...

Taking UCF-101 for example, please download the UCF-101 dataset and extract it to datasets/UCF-101 folder. You can generate the csv annotation by running python tools/annotate_vae_ucf101.py. It will result in two csv files, datasets/ucf101_train.csv and datasets/ucf101_test.csv, for training and testing respectively.

# stage 1 training, 8 NPUs
msrun --worker_num=8 --local_work_num=8 \
python scripts/train_vae.py --config configs/vae/train/stage1.yaml --use_parallel=True --csv_path datasets/ucf101_train.csv --video_folder datasets/UCF-101

# stage 2 training, 8 NPUs
msrun --worker_num=8 --local_work_num=8 \
python scripts/train_vae.py --config configs/vae/train/stage2.yaml --use_parallel=True --csv_path datasets/ucf101_train.csv --video_folder datasets/UCF-101

# stage 3 training, 8 NPUs
msrun --worker_num=8 --local_work_num=8 \
python scripts/train_vae.py --config configs/vae/train/stage3.yaml --use_parallel=True --csv_path datasets/ucf101_train.csv --video_folder datasets/UCF-101

You can change the csv_path and video_folder to train on your own data.

To evaluate the VAE performance, you need to run VAE inference first to generate the videos, then calculate scores on the generated videos:

# video generation and evaluation
python scripts/inference_vae.py --ckpt_path /path/to/you_vae_ckpt --image_size 256 --num_frames=17 --csv_path datasets/ucf101_test.csv --video_folder datasets/UCF-101

You can change the csv_path and video_folder to evaluate on your own data.

Here, we report the training performance and evaluation results on the UCF-101 dataset.

All experiments are tested on ascend 910* with mindspore 2.3.1 graph mode.

Note that we train with mixed video ang image strategy i.e. --mixed_strategy=mixed_video_image for stage 3 instead of random number of frames (mixed_video_random). Random frame training will be supported in the future.

Thanks go to the support from MindSpore team and the open-source contributions from the OpenSora project.

If you wish to contribute to this project, you can refer to the Contribution Guideline.

• ColossalAI: A powerful large model parallel acceleration and optimization system.
• DiT: Scalable Diffusion Models with Transformers.
• OpenDiT: An acceleration for DiT training. We adopt valuable acceleration strategies for training progress from OpenDiT.
• PixArt: An open-source DiT-based text-to-image model.
• Latte: An attempt to efficiently train DiT for video.
• StabilityAI VAE: A powerful image VAE model.
• CLIP: A powerful text-image embedding model.
• T5: A powerful text encoder.
• LLaVA: A powerful image captioning model based on Mistral-7B and Yi-34B.

We are grateful for their exceptional work and generous contribution to open source.