MIMO - Official PyTorch Implementation

Project page | Paper | Video | Online Demo

MIMO: Controllable Character Video Synthesis with Spatial Decomposed Modeling
Yifang Men, Yuan Yao, Miaomiao Cui, Liefeng Bo

MIMO is a generalizable model for controllable video synthesis, which can not only synthesize realistic character videos with controllable attributes (i.e., character, motion and scene) provided by very simple user inputs, but also simultaneously achieve advanced scalability to arbitrary characters, generality to novel 3D motions, and applicability to interactive real-world scenes in a unified framework.

Demo

Animating character image with driving 3D pose from motion dataset

github_teaser_motion.mp4

Driven by in-the-wild video with spatial 3D motion and interactive scene

github_teaser_wildvid.mp4

More results can be found in project page.

Updates

(2024-11-26) The online demo (v1.0) is available on ModelScope now! Try out . The 2.0 version to support custom driving videos will be coming soon.

(2024-09-25) The project page, demo video and technical report are released. The full paper version with more details is in process.

Code release plan

In response to the numerous inquiries we have received about the source code or a demo, we appreciate all the interest in our project. We are actively working on an online demo for everyone to try out, and it will be available recently. Additionally, we plan to release the source code and pretrained model upon paper acceptance. We will keep you updated on our progress here and thank you for your patience. Something may be late but never absent～

Citation

If you find this code useful for your research, please use the following BibTeX entry.

@inproceedings{men2024mimo,
  title={MIMO: Controllable Character Video Synthesis with Spatial Decomposed Modeling},
  author={Men, Yifang and Yao, Yuan and Cui, Miaomiao and Liefeng Bo},
  journal={arXiv preprint arXiv:2409.16160},
  website={https://menyifang.github.io/projects/MIMO/index.html},
  year={2024}}