Merge branch 'main' of https://github.com/Dingry/BunnyVisionPro into …

…main

.gitignore

@@ -14,3 +14,7 @@ imgui.ini
 /log
 /site/
 /bunny_teleop.egg-info/
+
+# Examples
+/examples/retargeting/data/
+!examples/retargeting/data/offline_avp_stream.pkl

README.md

@@ -12,7 +12,7 @@
     ·
     <a href="https://www.researchgate.net/profile/Chengzhe-Jia"><strong>Chengzhe Jia</strong></a>
     <br>
-    <a href="https://scholar.google.com/citations?user=OQQzJb4AAAAJ&hl=en"><strong>Shiqi Yang</strong></a>
+    <a href="https://aaronyang1223.github.io/"><strong>Shiqi Yang</strong></a>
     ·
     <a href="https://rchalyang.github.io/"><strong>Ruihan Yang</strong></a>
     .
@@ -40,7 +40,7 @@ instructions.
 
 - [x] Tutorial for initialization
 - [x] Tutorial for tips and troubleshooting
-- [ ] Tutorial for retargeting
+- [x] Tutorial for retargeting
 - [ ] Robot collision avoidance
 - [ ] Robot arm singularity avoidance
 - [ ] Collision-free retargeting

bunny_teleop/__init__.py

@@ -1 +1 @@
-__version__ = "0.0.1"
+__version__ = "0.0.2"

docs/advanced/visualization.md

@@ -17,5 +17,5 @@ progress, transitioning to two smaller 3D gizmos indicating the end effector pos
 After initialization, the robot will begin moving inside the web visualizer. The two small gizmo shows the end effector
 pose from human.
 
-![](../assets/videos/output_file.webp)
+![](../assets/videos/gizmo.webp)
 

docs/example/retargeting.md

@@ -0,0 +1,56 @@
+# Hand Retargeting
+
+This example demonstrates how to use data exclusively from Apple Vision Pro to manipulate bimanual robot hands, omitting
+the robot arm component. We capture some offline Vision Pro data and employ it to create retargeting visualizations as
+shown below.
+
+![bimanual_retargeting](../assets/videos/teaser.webp)
+
+## Installation
+
+We provide an offline data file, `data/offline_avp_stream.pkl`, which contains
+recorded Vision Pro streams. This data is used to generate trajectories for robot hand joints. To process the data, you
+will need specific robot hand
+models.
+
+```shell
+cd examples/retargeting
+
+# Download robot hand models
+git clone https://github.com/dexsuite/dex-urdf
+
+# Install additional Python dependencies needed for this example
+pip install sapien dex_retargeting tyro opencv-python
+```
+
+## Retargeting Human Hand Data
+
+Use the following commands to convert recorded human hand data into robot hand joint
+trajectories:
+
+```shell
+python retargeting.py --robot-name allegro --data-path data/offline_avp_stream.pkl --output-path data/allegro.pkl
+
+# You can also do that for other robot hands.
+python retargeting.py --robot-name ability --data-path data/offline_avp_stream.pkl --output-path data/ability.pkl
+python retargeting.py --robot-name svh --data-path data/offline_avp_stream.pkl --output-path data/svh.pkl 
+```
+
+## Visualize Retargeting Results
+
+We also provide scripts to convert the retargeting data into videos for visualization, such
+as `data/allegro.mp4`. Check out the video to see the results!
+
+```shell
+python render_retargeting.py --pickle-path data/allegro.pkl --output_video_path data/allegro.mp4 
+
+# You can also do that for other robot hands.
+python render_retargeting.py --pickle-path data/ability.pkl --output_video_path data/ability.mp4 
+python render_retargeting.py --pickle-path data/svh.pkl --output_video_path data/svh.mp4 
+```
+
+!!! tip "Record Your Own Vision Pro Data"
+
+    If you have a Vision Pro, you can record your own data by opening the Tracking Stream App as described in the [Basic Usage Section](https://dingry.github.io/BunnyVisionPro/getstarted/usage/#3-launch-tracking-streamer-app).
+    Then, run the command below to save your data: `python save_offline_avp_stream.py --avp_ip=YOUR_VISION_PRO_IP`
+    This will generate a file `data/your_offline_avp_stream.pkl` in the provided format.

docs/getstarted/usage.md

@@ -10,7 +10,7 @@ set up on different machines, for simplicity, this tutorial assumes they are on
 Execute the following command to run the teleoperation server inside a Docker container:
 
 ```shell
-docker run -it --net host --name bunny_teleop_server -v yzqin/bunny_teleop_server bash
+docker run -it --net host --name bunny_teleop_server yzqin/bunny_teleop_server bash
 ```
 
 ### 2. **Network Configuration**

mkdocs.yml

@@ -60,11 +60,6 @@ plugins:
   # - minify:
   #     minify_html: true
 
-# # Hooks
-# hooks:
-#   - material/overrides/hooks/shortcodes.py
-#   - material/overrides/hooks/translations.py
-
 # Additional configuration
 extra:
   status:
@@ -146,5 +141,5 @@ nav:
       - system/overview.md
       - advanced/initialization.md
       - advanced/visualization.md
-#  - Reference:
-#    - reference/index.md
+  - Example:
+      - example/retargeting.md

setup.py

@@ -17,6 +17,7 @@
 core_requirements = [
     "pyzmq",
     "numpy",
+    "tornado",
 ]
 
 dev_requirements = [
@@ -26,7 +27,12 @@
     "mypy",
 ]
 
-example_requirements = [""]
+example_requirements = [
+    "dex_retargeting",
+    "tyro",
+    "sapien==3.0.0b0",
+    "opencv-python",
+]
 
 classifiers = [
     "Development Status :: 3 - Alpha",