This repository contains the official pytorch implementation of Ground-A-Video.
[ICLR 2024] Ground-A-Video: Zero-shot Grounded Video Editing using Text-to-image Diffusion Models
Hyeonho Jeong, Jong Chul Ye

Ground A Video is the first groundings-driven video editing framework, specially designed for Multi-Attribute Video Editing.
Ground A Video is the first framework to intergrate spatially-continuous and spatially-discrete conditions.
Ground A Video does not neglect edits, confuse edits, but does preserve non-target regions.
[TL;DR] Stable Diffusion 3D + ControlNet 3D + GLIGEN 3D + Optical Flow Smoothing = Ground-A-Video

• [11/11/2023] The paper is currently under review process.
• [01/15/2024] Code released!

Input Video	Video Groundings	Depth Map	Optical Flow	Output Video
"A man is walking a dog on the road."	man, dog, road	by ZoeDepth	by RAFT-large	"Iron Man is walking a sheep on the lake."
"A rabbit is eating a watermelon on the table."	rabbit, watermelon, table	by ZoeDepth	by RAFT-large	"A squirrel is eating an orange on the grass, under the aurora."

git clone https://github.com/Ground-A-Video/Ground-A-Video.git
cd Ground-A-Video

conda create -n groundvideo python=3.8
conda activate groundvideo
pip install -r requirements.txt

Important: Ensure that you download the model weights before executing the scripts

git lfs install
git clone https://huggingface.co/gligen/gligen-inpainting-text-box
git clone https://huggingface.co/ground-a-video/unet3d_ckpts
git clone https://huggingface.co/lllyasviel/control_v11f1p_sd15_depth

These commands will place the pretrained GLIGEN weights at:

• Ground-A-Video/gligen-inpainting-text-box/diffusion_pytorch_model.bin
• Ground-A-Video/unet3d_ckpts/diffusion_pytorch_model.bin
• Ground-A-Video/unet3d_ckpts/config.json
• Ground-A-Video/control_v11f1p_sd15_depth/diffusion_pytorch_model.bin
• Ground-A-Video/control_v11f1p_sd15_depth/config.json

Alternatively, you can manually download the weights using the web interface from the following links:

• GLIGEN
• Ground A Video
• ControlNet

The input video frames should be stored in video_images , organized by each video's name.
Pre-computed groundings, including bounding box coordinates and corresponding text annotations, for each video are available in configuration files located at video_configs/{video_name}.yaml

Ground-A-Video is designed to be a training-free framework. To run the inference script, use the following command:

python main.py --config video_configs/rabbit_watermelon.yaml --folder outputs/rabbit_watermelon

• --config: Specifies the path to the configuration file. Modify the config files under video_configs as needed
• --folder: Designates the directory where output videos will be saved
• --clip_length: Sets the number of input video frames. Default is 8.
• --denoising_steps: Defines the number of denoising steps. Default is 50.
• --ddim_inv_steps: Determines the number of steps for per-frame DDIM inversion and Null-text Optimization. Default is 20.
• --guidance_scale: Sets the CFG scale. Default is 12.5.
• --flow_smooth_threshold: Threshold for optical flow guided smoothing. Default is 0.2.
• --controlnet_conditioning_scale: Sets the conditioning scale for ControlNet. Default is 1.0.
• --nti: Whether to perfrom Null-text Optimization after DDIM Inversion. Default is False.
  (If your CUDA Version is 11.4, then you can set is as True. If your CUDA Version is 12.2 or higher, set it as False: The codes are implemented using fp16 dtypes but in 12.2 higher CUDA version, gradient backpropagation from nti incurs errors)

Input Videos	Output Videos

If you like our work, please cite our paper.

@article{jeong2023ground,
  title={Ground-A-Video: Zero-shot Grounded Video Editing using Text-to-image Diffusion Models},
  author={Jeong, Hyeonho and Ye, Jong Chul},
  journal={arXiv preprint arXiv:2310.01107},
  year={2023}
}

• Ground-A-Video builds upon huge open-source projects:
  diffusers, Stable Diffusion, GLIGEN, ControlNet, GLIP, RAFT.
  Thank you for open-sourcing!