ZeDO: Back to Optimization: Diffusion-based Zero-Shot 3D Human Pose Estimation (Accepted by WACV 2024)

This is the official implementation of this paper:

Zhongyu Jiang, Zhuoran Zhou, Lei Li, Wenhao Chai, Cheng-Yen Yang, and Jenq-Neng Hwang. Back to Optimization: Diffusion-based Zero-Shot 3D Human Pose Estimation WACV 2024.

And its follow-up paper:

Zhuoran Zhou, Zhongyu Jiang, Wenhao Chai, Cheng-Yen Yang, Lei Li and Jenq-Neng Hwang. Efficient Domain Adaptation via Generative Prior for 3D Infant Pose Estimation WACVW 2024.

3D human pose estimation

Results on 3DPW

Under PA-MPJPE and MPJPE

Methods	CE	Opt	PA-MPJPE ↓	MPJPE ↓
SPIN			59.2	96.9
VIBE			51.9	82.9
PARE			46.4	74.7
HybrIK			45.0	74.1
HybrIK	✓		50.9	82.0
PoseAug	✓		58.5	94.1
AdaptPose	✓		46.5	81.2
PoseDA	✓		55.3	87.7
ZeDO (J=17)	✓	✓	40.3	69.7
ZeDO (J=14)	✓	✓	43.1	76.6
ZeDO (J=17, Additional Training Data)	✓	✓	38.4	68.3
ZeDO (J=17, S=50, Additional Training Data)	✓	✓	30.6	54.7

Evaluate model

Envrionment Setup

• pytorch >= 1.10

conda create -n ZeDO python==3.9
conda activate ZeDO
pip install -r requirements.txt

Data and Model Preparation

Evaluation dataset, clusters and checkpoint: Google Drive

${POSE_ROOT}
|-- configs
|-- lib
|-- run
|-- checkpoint
    |-- concatebb
        |-- checkpoint_1500.pth
|-- data
    |-- h36m
        |-- h36m_test.pkl
        |-- h36m_sh_dt_ft.pkl
    |-- 3dpw
        |-- pw3d_test.npz
    |-- 3dhp
        |-- mpii3d_test.pkl
|-- clusters
    |-- 3dhp_cluster1.pkl
    |-- h36m_cluster1.pkl
    |-- 3dhp_cluster50.pkl
    |-- h36m_cluster50.pkl

Evaluation script

python -m run.opt_main --config configs/subvp/concat_pose_optimization_<dataset>.py --ckpt_dir ./checkpoint/concatebb --ckpt_name checkpoint_1500.pth --hypo 1 <--gt> 

In the wild inference

Please modify the custom dataset lib/dataset/custom.py to load your data, following the instructions in the Python file and filling out the read_data() function. Use the script for inference:

python -m run.inference --config configs/subvp/concat_pose_optimization_wild.py --ckpt_dir ./checkpoint/concatebb --ckpt_name checkpoint_1500.pth --hypo 1 <--eval>

--eval is for evaluation if you provide the ground truth 3D keypoints.

The results will be saved in results.npy by default. You can change the way saving the data yourself.

Make sure the 2D keypoints are following the keypoint definitions of Human3.6M. Otherwise, you may need to train the pose generation model on your keypoint definition.

3D infant pose estimation

Data and Model Preparation

We use Mini-RGBD and SyRIP in experiments, please follow the offical instructons to download the datasets under a data directory like this:

${POSE_ROOT}
|-- configs
|-- lib
|-- run
|-- checkpoint
|-- data
    |-- mini-rgbd
    |-- syrip
|-- clusters
    |-- 3dhp_cluster1.pkl
    |-- h36m_cluster1.pkl
    |-- 3dhp_cluster50.pkl
    |-- h36m_cluster50.pkl

Then run syrip_process and mini_process to extrace data npy files. Make sure that the extracted data are placed under corresponding data folders.

Evaluation script

For reproducing our results, we provide our models: Google-drive which includes all bakcbones of three training strategies. (ZeDO-i means training from scratch)

For MINI-RGBD dataset

python -m run.opt_main_infant  --config configs/subvp/concat_pose_optimization_mini.py --ckpt_dir <dir_path> --ckpt_name <ckpt_name>  --gt --hypo 1

For SyRIP dataset

python -m run.opt_main_infant  --config configs/subvp/concat_pose_optimization_syrip.py --ckpt_dir <dir_path> --ckpt_name <ckpt_name>  --gt -hypo 1

Citation

If you find this code useful in your project, please consider citing:

@inproceedings{Jiang2024ZeDO,
  title={Back to Optimization: Diffusion-based Zero-Shot 3D Human Pose Estimation},
  author={Jiang, Zhongyu and Zhou, Zhuoran and Li, Lei and Chai, Wenhao and Yang, Cheng-Yen and Hwang, Jenq-Neng},
  booktitle={Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision},
  year={2024}
}

@article{zhou2023efficient,
  title={Efficient Domain Adaptation via Generative Prior for 3D Infant Pose Estimation},
  author={Zhou, Zhuoran and Jiang, Zhongyu and Chai, Wenhao and Yang, Cheng-Yen and Li, Lei and Hwang, Jenq-Neng},
  journal={arXiv preprint arXiv:2311.12043},
  year={2023}
}

Acknowledgement

This repo is built on the excellent work score_sde by Yang Song and GFPose by Hai Ci.