[ACMMM23] GraMMaR: Ground-aware Motion Model for 3D Human Motion Reconstruction

This is the official repository of the paper [ACMMM23] GraMMaR: Ground-aware Motion Model for 3D Human Motion Reconstruction.

The code will be released soon.

Sihan Ma, Qiong Cao, Jing Zhang, and Dacheng Tao

Introduction | Statement | Video Demo

3D motion in the camera view is misleading. A representative optimization method HuMoR [34 ] produces correct poses under camera view but physically implausible poses in world view when faced with ambiguity (Row1) and noise (Row2). In contrast, our method provides a ground-aware motion, thereby ensuring physical plausibility across all views. To achieve it, we explicitly present the human-ground interaction and devise a CVAE-based model to jointly learn the distribution of transitions in interaction and pose. Body torso direction and contacts for HuMoR and ours are highlighted. GT in Row1 is reconstructed from multi-view images.

Introduction

Demystifying complex human-ground interactions is essential for accurate and realistic 3D human motion reconstruction from RGB videos, as it ensures consistency between the humans and the ground plane.

Prior methods have modeled human-ground interactions either implicitly or in a sparse manner, often resulting in unrealistic and incorrect motions when faced with noise and uncertainty. In contrast, our approach explicitly represents these interactions in a dense and continuous manner. To this end, we propose a novel Ground-aware Motion Model for 3D Human Motion Reconstruction, named GraMMaR, which jointly learns the distribution of transitions in both pose and interaction between every joint and ground plane at each time step of a motion sequence. It is trained to explicitly promote consistency between the motion and distance change towards the ground.

After training, we establish a joint optimization strategy that utilizes GraMMaR as a dual-prior, regularizing the optimization towards the space of plausible ground-aware motions. This leads to realistic and coherent motion reconstruction, irrespective of the assumed or learned ground plane.

Through extensive evaluation on the AMASS and AIST++ datasets, our model demonstrates good generalization and discriminating abilities in challenging cases including complex and ambiguous human-ground interactions.

Statement

If you are interested in our work, please consider citing the following:

@inproceedings{ma2023grammar,
  title={GraMMaR: Ground-aware Motion Model for 3D Human Motion Reconstruction},
  author={Ma, Sihan and Cao, Qiong and Yi, Hongwei and Zhang, Jing and Tao, Dacheng},
  booktitle={Proceedings of the 31st ACM International Conference on Multimedia},
  pages={2817--2828},
  year={2023}
}

For further questions, please contact Sihan Ma at sima7436@uni.sydney.edu.au.

This work is intended for studying the human motion reconstruction from monocular RGB videos, helping develop more realistic motion sequences that interact with the ground in a physically plausible way. Since our techniques can generate realistic and diverse 3D human motion sequences from videos, there is a risk that such techniques could be potentially misused for fake video generation. For instance, motion sequences from videos could be used to construct highly realistic 3D body motions while they never happened. Thus, we should use such technology responsibly and carefully. We hope to raise the public’s awareness about the safe use of such technology.