Skip to content

Commit

Permalink
[doc] add news section for tracking change logs
Browse files Browse the repository at this point in the history
  • Loading branch information
Isaac Gu committed Jul 5, 2024
1 parent 33d019b commit 0399e29
Showing 1 changed file with 8 additions and 2 deletions.
10 changes: 8 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,8 +29,14 @@ MimicMotion: High-Quality Human Motion Video Generation with Confidence-aware Po

In recent years, generative artificial intelligence has achieved significant advancements in the field of image generation, spawning a variety of applications. However, video generation still faces considerable challenges in various aspects such as controllability, video length, and richness of details, which hinder the application and popularization of this technology. In this work, we propose a controllable video generation framework, dubbed *MimicMotion*, which can generate high-quality videos of arbitrary length with any motion guidance. Comparing with previous methods, our approach has several highlights. Firstly, with confidence-aware pose guidance, temporal smoothness can be achieved so model robustness can be enhanced with large-scale training data. Secondly, regional loss amplification based on pose confidence significantly eases the distortion of image significantly. Lastly, for generating long smooth videos, a progressive latent fusion strategy is proposed. By this means, videos of arbitrary length can be generated with acceptable resource consumption. With extensive experiments and user studies, MimicMotion demonstrates significant improvements over previous approaches in multiple aspects.

## News

* `[2024-07-01]`: Project page, code, technical report and a basic model checkpoint are released. A better checkpoint supporting higher quality video generation will be released very soon. Stay tuned!

## Quickstart

For the initial released version of the model checkpoint, it supports generating videos with a maximum of 16 frames at a 576x1024 resolution. If you encounter insufficient memory issues, you can appropriately reduce the number of frames.

### Environment setup

Recommend python 3+ with torch 2.x are validated with an Nvidia V100 GPU. Follow the command below to install all the dependencies of python:
Expand Down Expand Up @@ -78,12 +84,12 @@ models/
### Model inference
We provide the inference script.
A sample configuration for testing is provided as `test.yaml`. You can also easily modify the various configurations according to your needs.
```
python inference.py --inference_config configs/test.yaml
```
## Citation
```bib
@article{mimicmotion2024,
Expand Down

0 comments on commit 0399e29

Please sign in to comment.