EgoExo Forge: All the Data + Utilities Needed for Egocentric and Exocentric Human Data

Uses Rerun to visualize, Gradio for an interactive UI, and Pixi for a easy installation

Demo

Hosted Demos can be found on huggingface spaces

Installation

Using Pixi

Make sure you have the Pixi package manager installed

git clone https://github.com/rerun-io/egoexo-forge.git
cd egoexo-forge
pixi run app

All commands can be listed using pixi task list

Usage

Gradio App

pixi run app

CLI (TODO)

You can see all tasks by running pixi task list

Dataset Information

Assembly101

A procedural activity dataset with 4321 multi-view videos of people assembling and disassembling 101 take-apart toy vehicles, featuring rich variations in action ordering, mistakes, and corrections. Note: Assembly101 is very large and can take some time to fully load each episode.

HoCap

A dataset for 3D reconstruction and pose tracking of hands and objects in videos, featuring humans interacting with objects for various tasks including pick-and-place actions and handovers.

EgoDex

The largest and most diverse dataset of dexterous human manipulation with 829 hours of egocentric video and paired 3D hand tracking, covering 194 different tabletop tasks with everyday household objects.

UmeTrack

Coming soon...

Supported Datasets

Currently Supported

• Assembly101 ✅
• HO-Cap ✅
• EgoDex ✅

Planned Support

• UmeTrack 🚧

Acknowledgements

Assembly101

@article{sener2022assembly101,
    title = {Assembly101: A Large-Scale Multi-View Video Dataset for Understanding Procedural Activities},
    author = {F. Sener and D. Chatterjee and D. Shelepov and K. He and D. Singhania and R. Wang and A. Yao},
    journal = {CVPR 2022},
}

HO-Cap

@misc{wang2024hocapcapturedataset3d,
      title={HO-Cap: A Capture System and Dataset for 3D Reconstruction and Pose Tracking of Hand-Object Interaction},
      author={Jikai Wang and Qifan Zhang and Yu-Wei Chao and Bowen Wen and Xiaohu Guo and Yu Xiang},
      year={2024},
      eprint={2406.06843},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2406.06843},
}

EgoDex

@misc{hoque2025egodexlearningdexterousmanipulation,
      title={EgoDex: Learning Dexterous Manipulation from Large-Scale Egocentric Video},
      author={Ryan Hoque and Peide Huang and David J. Yoon and Mouli Sivapurapu and Jian Zhang},
      year={2025},
      eprint={2505.11709},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}