Skip to content

Latest commit

 

History

History
116 lines (87 loc) · 3.84 KB

README.md

File metadata and controls

116 lines (87 loc) · 3.84 KB
Raw

Perception Stitching: Zero-Shot Perception Encoder Transfer for Visuomotor Robot Policies

Pingcheng Jian, Easop Lee, Zachary Bell, Michael M. Zavlanos, Boyuan Chen
Duke University

Project Website | Video | Paper

Overview

This repo contains the implementation for the paper Perception Stitching: Zero-Shot Perception Encoder Transfer for Visuomotor Robot Policies.

pes_teaser

Citation

If you find our paper or codebase helpful, please consider citing:

@article{jian2024perception,
title={Perception Stitching: Zero-Shot Perception Encoder Transfer for Visuomotor Robot Policies},
author={Pingcheng Jian and Easop Lee and Zachary I. Bell and Michael M. Zavlanos and Boyuan Chen},
journal={Transactions on Machine Learning Research},
issn={2835-8856},
year={2024},
url={https://openreview.net/forum?id=tYxRyNT0TC},
note={}
}

Content

installation

  • The development tools of this project can be installed with conda:
$ conda env create -f pes_env.yml .

training

  1. Example of training with Perception Stitching method
python pes_train.py --device cuda:0 --save_model --seed 101 --process1 --effect blur
  1. Example of training with Perception Stitching (with L1 and L2 regularization) method
python pes_l1l2_train.py --device cuda:0 --save_model --seed 101 --process2 --effect fisheye
  1. Example of training with Perception Stitching (without disentanglement regularization) method
python pes_without_disent_train.py --device cuda:0 --save_model --seed 101 --process1 --effect zoomin
  1. Example of training with Cannistraci et al. 2024 (linear sum) baseline
python linear_sum_train.py --device cuda:0 --save_model --seed 101 --process2 --effect mask
  1. Example of training with Cannistraci et al. 2024 (non-linear sum) baseline
python nonlinear_sum_train.py --device cuda:0 --save_model --seed 101 --process1 --effect noise

testing

  1. Example of testing the policy trained with Perception Stitching method
python pes_test.py --device cuda:0 --vision1 robot0_eye_in_hand --vision2 agentview --effect blur
  1. Example of testing the policy trained with Perception Stitching (with L1 and L2 regularization) method
python pes_l1l2_test.py --device cuda:0 --vision1 robot0_eye_in_hand --vision2 agentview --effect fisheye
  1. Example of testing the policy trained with Perception Stitching (without disentanglement regularization) method
python pes_without_disent_test.py --device cuda:0 --vision1 robot0_eye_in_hand --vision2 agentview --effect zoomin
  1. Example of testing the policy trained with Cannistraci et al. 2024 (linear sum) baseline
python linear_sum_test.py --device cuda:0 --vision1 robot0_eye_in_hand --vision2 agentview --effect mask
  1. Example of testing the policy trained with Cannistraci et al. 2024 (non-linear sum) baseline
python nonlinear_sum_test.py --device cuda:0 --vision1 robot0_eye_in_hand --vision2 agentview --effect noise

Experiment setup in the simulation ps_teaser

Experiment setup in the real world ps_teaser

License

This repository is released under the Apache License 2.0. See LICENSE for additional details.

Acknowledgement

This project refers to the github repositories robomimic, robosuite, and relreps.