Code repository for the paper: Reconstructing Hand-Held Objects in 3D
Jane Wu, Georgios Pavlakos, Georgia Gkioxari, Jitendra Malik
Installation and preparation follow the MAE repo, just like MCC. Please also install PyTorch3D for 3D related funcionality and the following libraries:
pip install omegaconf trimesh
Please see DATASET.md for information on data preparation.
To run MCC-HO inference on any input image and estimated 3D hand, please use, e.g.,
python demo.py \
--image demo/boardgame_v_W_qdSiPKSdQ_frame000019.jpg \
--hand demo/boardgame_v_W_qdSiPKSdQ_frame000019_hand.obj \
--seg demo/boardgame_v_W_qdSiPKSdQ_frame000019_mask.png \
--cam demo/camera_intrinsics_mow.json \
--checkpoint [path to model checkpoint]
One may use a checkpoint from the training step below or download our pretrained model (trained on DexYCB, MOW, and HOI4D) [here].
One may set the --score_thresholds argument to specify the score thresholds (More points are shown with a lower threshold, but the predictions might be noisier).
The script will generate an html file showing an interactive visualizaion of the MCC-HO output with plotly.
To train an MCC-HO model, please run
OUTPUT=model_outputs
python main_mccho.py \
--mccho_path [path to MCC-HO preprocessed data] \
--dataset_cache [path to dataset cache] \
--job_dir $OUTPUT \
--output_dir $OUTPUT/log \
--shuffle_train
- Optional: MCC-HO (excluding the segmentation output layers) may be initialized using MCC pre-trained. A pretrained MCC model is available [here].
This implementation builds on the MCC codebase, which in turn is based on MAE.
If you find this code useful for your research, please consider citing the following paper:
@article{wu2024reconstructing,
title={Reconstructing Hand-Held Objects in 3D},
author={Wu, Jane and Pavlakos, Georgios and Gkioxari, Georgia and Malik, Jitendra},
journal={arXiv preprint arXiv:2404.06507,
year={2024},
}