README.md

Retarget Robot Motion from Human Hand Video

Generate the robot joint pose trajectory from our pre-recorded video

cd example/vector_retargeting
python3 detect_from_video.py \
  --robot-name allegro \
  --video-path data/human_hand_video.mp4 \
  --retargeting-type dexpilot \
  --hand-type right \
  --output-path data/allegro_joints.pkl 

This command will output the joint trajectory as a pickle file at the output_path.

The pickle file is a python dictionary with two keys: meta_data and data. meta_data, a dictionary, includes details about the robot, while data, a list, contains the robotic joint positions for each frame. For additional options, refer to the help information. Note that the time cost here includes both the hand pose detection from video, and the hand pose retargeting in single process mode.

python3 detect_from_video.py --help

Utilize the pickle file to produce a video of the robot

python3 render_robot_hand.py \
  --pickle-path data/allegro_joints.pkl \
  --output-video-path data/allegro.mp4 \
  --headless

This command uses the data saved from the previous step to create a rendered video.

Capture a Video Using Your Webcam

The following instructions assume that your computer has a webcam connected.

python3 capture_webcam.py --video-path data/my_human_hand_video.mp4

This command enables you to use your webcam to record a video saved in MP4 format. To stop recording, press Esc on your keyboard.

Real-time Visualization of Hand Retargeting via Webcam

pip install loguru
python3 show_realtime_retargeting.py \
  --robot-name allegro \
  --retargeting-type dexpilot \
  --hand-type right 

This process integrates the tasks described above. It involves capturing your hand movements through the webcam and instantaneously displaying the retargeting outcomes in the SAPIEN viewer. Special thanks to @xbkaishui for contributing the initial pull request.