Trajectory Attention For Fine-grained Video Motion Control

Zeqi Xiao¹ Wenqi Ouyang¹ Yifan Zhou¹ Shuai Yang² Lei Yang³ Jianlou Si³ Xingang Pan^1
1S-Lab, Nanyang Technological University,
²Wangxuan Institute of Computer Technology, Peking University,
³Sensetime Research

demo.mp4

🏠 About

Trajectory attention injects partial motion information by making content along trajectories consistent. It facilitates various tasks such as camera motion control on images and videos, and first-frame-guided video editing. Yellow boxes indicate reference contents. Green boxes indicate input frames. Blue boxes indicate output frames.

Our method allows for conditioning on trajectories from various sources -- such as camera motion derived from a single image, as shown in this figure. We inject these conditions into the model through trajectory attention, enabling explicit and fine-grained control over the motion in the generated video.

Installation

1. Create a Conda Environment

This codebase is tested with the versions of PyTorch 1.13.1+cu117.

conda create -n trajattn python==3.10
conda activate trajattn
pip install -r requirements.txt

2. Download model weights Download model weights from huggingface.

3. Clone Relevant Repositories and Download Checkpoints

# Clone the Depth-Anything-V2 repository
git clone https://github.com/DepthAnything/Depth-Anything-V2
# Download the Depth-Anything-V2-Large checkpoint
wget https://huggingface.co/depth-anything/Depth-Anything-V2-Large/resolve/main/depth_anything_v2_vitl.pth?download=true
# Overwrite the run.py
cp depth_anything/run.py Depth-Anything-V2/run.py

Save the checkpoints to the checkpoints/ directory. You can also modify the checkpoint path in the running scripts if needed.

Runnig

To control camera motion on images, execute the following script

sh image_control.sh

TODO

• Release models and weight;
• Release pipelines for single image camera motion control;
• Release pipelines for video camera motion control;
• Release pipelines for video editing;
• Release training pipeline

🔗 Citation

If you find our work helpful, please cite:

@misc{xiao2024trajectoryattentionfinegrainedvideo,
      title={Trajectory Attention for Fine-grained Video Motion Control}, 
      author={Zeqi Xiao and Wenqi Ouyang and Yifan Zhou and Shuai Yang and Lei Yang and Jianlou Si and Xingang Pan},
      year={2024},
      eprint={2411.19324},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2411.19324}, 
}

👏 Acknowledgements

• SVD: Our model is tuned from SVD.
• MiraData: We use the data collected by MiraData.
• Depth-Anything-V2: We estimate depth map by Depth-Anything-V2.
• Unimatch: We estimate optical flow map by Unimatch.
• Cotracker: We estimate point trajectories by Cotracker.
• NVS_Solver: Our camera rendering code is based on NVS_Solver.